Add number of shards by node

[arob1n]

Hello,

In the elasticsearch prometheus, there is a metric named elasticsearch_indices_shards_docs (Count of documents on this shard) which we use to get the number of shards by node.

Existing metrics :

• opensearch_cluster_shards_number (for the whole cluster)
• opensearch_index_shards_number (for number of shard by index)

These 2 metrics are useful, but not for what we need : the number of shards by node

(It was present with another plugin for opensearch -discontinued- https://github.com/aparo/opensearch-prometheus-exporter )

We do need a metric that would provide the numbers of shards per node. Do you plan to add such a metric in a near future or if not, would you consider a pull request to add one ?

[lukas-vlcek]

@arob1n Thanks for reaching out. First, PRs are always welcome :-)

However, let me first have a look at this because if there are some metrics that were available in @aparo 's clone are are not present in this plugin then this might be for a reason... so let me have a look at this first and I will get beck with some explanation. Then we will know more before we start adding this metric.

BTW, can you see this metric being exposed in any existing OpenSearch API? (which OpenSearch version are you using?)

[arob1n]

Currently we are using OS 2.6.0

We can get this information with :

GET _cluster/stats/nodes/my-node-1
GET _cluster/stats/nodes/my-node-2
GET _cluster/stats/nodes/my-node-3
...

Where the info is in : indices.shards.total

However, we need to get node names with GET _cat/nodes

[evheniyt]

This metric could be very useful, as you could monitor how many shards are present on the node and alert if there are no shards assigned (the node is excluded from the allocation for some reason) @lukas-vlcek could you/team review the PR above?

[lukas-vlcek]

@evheniyt Thanks for pinging me. Yes, this metric is definitely useful. I reviewed the PR and left some ideas/proposals. Let's take it from there and we will see. I am happy to help with the implementation.

[lukas-vlcek]

For reference, there are more "max shards" limits in the cluster, see: https://opster.com/guides/opensearch/opensearch-basics/opensearch-max-shards-per-node-exceeded/ (at index, node or cluster levels)

It would be very useful to expose all of these and enable alerting.