Open kkewwei opened 2 months ago
@rwali-aws, there is not much difference whether is opensearch or elasticsearch.
In opensearch, RetentionLeaseBackgroundSyncAction still use MANAGEMENT threadpool, https://github.com/opensearch-project/OpenSearch/blob/6b8b3efe01a62c221f308a2e3b019d75a7f5ad8a/server/src/main/java/org/opensearch/index/seqno/RetentionLeaseBackgroundSyncAction.java#L112 TransportNodesStatsAction still also use MANAGEMENT threadpool, this case will continue to happen, in which we will still can't get cluster metrics. https://github.com/opensearch-project/OpenSearch/blob/6b8b3efe01a62c221f308a2e3b019d75a7f5ad8a/server/src/main/java/org/opensearch/index/seqno/RetentionLeaseBackgroundSyncAction.java#L112
@rwali-aws Did you mean to close this issue after your comment above, looks like this is still untriaged?
Is your feature request related to a problem? Please describe
In our product, GET _nodes/stats is blocked, as we know node will use MANAGEMENT thread pool to get metrics. https://github.com/opensearch-project/OpenSearch/blob/6267e94b00d62c3fba80abf02ea584c85ea3aad0/server/src/main/java/org/opensearch/action/admin/cluster/node/stats/TransportNodesStatsAction.java#L80
We find because there are too many tasks pending in MANAGEMENT thread pool, so the NodesStats request is blocked. The usage of the management thread pool is as follow:
Why is the threadpool so busy? the reason is that the 5 active threads are used to write disk:
The MANAGEMENT threadpool will be blocked, as long as one disk is busy, which is common in produce. The api is relatively important:
timeout
is added to the api in high version, it doesn't solve the race.Describe the solution you'd like
If we should create a new threadpool to do with those tasks related to written io.
Related component
Cluster Manager
Describe alternatives you've considered
No response
Additional context
Version: es7.10.2