search

opensearch-project / OpenSearch

🔎 Open source distributed and RESTful search engine.
https://opensearch.org/docs/latest/opensearch/index/
Apache License 2.0
9.73k stars 1.8k forks source link

[Discuss] Metrics Plugin for Cross Cluster Search #560

Open soosinha opened 3 years ago

soosinha commented 3 years ago

Is your feature request related to a problem? Please describe. Currently there is no way to track the cross cluster request/connection metrics. The connection from local to remote cluster is tracked in the local cluster using the remote cluster alias provided in the settings. But there are no metrics tracked along with the connection. We would like APIs to obtain the following information:

Describe the solution you'd like A plugin to provide the above APIs would solve the problem. Source cluster: The total connections can be tracked when the cluster settings are added to create a new connection to the remote cluster. The outgoing requests can be tracked using an interceptor which will aggregate the count using the remote cluster alias as the key. The remote cluster alias will be unique at the local cluster level. Destination cluster: Tracking the metrics at the destination cluster is tricky as there is no setting that is updated here. So it is difficult to track the total number of connections. Also, even if we set the connection alias at the destination cluster, it may conflict with other aliases. For example, cluster A can create remote1 alias for connection cluster B. But cluster A cannot use remote1 again for connection to cluster C. Although cluster C can create a connection remote1 to cluster B. So tracking the number of requests at the destination cluster needs some unique identifier.

Describe alternatives you've considered One way to track metrics at the destination cluster is to use unique connection id for each combination of clusters and add this setting at both the clusters. This would require additional changes in the core cross cluster search feature to support connection ids. Looking for more ideas to solve this problem.

Additional context Add any other context or screenshots about the feature request here.

swartz-k commented 3 years ago

Maybe Prometheus Metrics and Traces will be helpful？

minalsha commented 2 years ago

@elfisher could you please determine how this should be prioritized?

elfisher commented 2 years ago

@soosinha thanks for sharing this proposal! We'll track this issue, and if you if you are interested in contributing this let us know.

On the proposal itself, adding more metrics to understand cross cluster search stats would be useful. I'm not sure this needs to be a separate plugin versus extending the existing stats/cat APIs. Why are you thinking a separate plugin? Thanks!

soosinha commented 2 years ago

@elfisher It makes sense to have a stats API in core OpenSearch itself as cross cluster search is a core feature. There is no need of a separate plugin.