[Feature Request] we should separate write-related io request from MANAGEMENT thread pool

[kkewwei]

Is your feature request related to a problem? Please describe

In our product, GET _nodes/stats is blocked, as we know node will use MANAGEMENT thread pool to get metrics. https://github.com/opensearch-project/OpenSearch/blob/6267e94b00d62c3fba80abf02ea584c85ea3aad0/server/src/main/java/org/opensearch/action/admin/cluster/node/stats/TransportNodesStatsAction.java#L80

We find because there are too many tasks pending in MANAGEMENT thread pool, so the NodesStats request is blocked. The usage of the management thread pool is as follow:

 "management" : {
          "threads" : 5,
          "queue" : 793,
          "active" : 5,
          "rejected" : 0,
          "largest" : 5,
          "completed" : 888436
}

Why is the threadpool so busy? the reason is that the 5 active threads are used to write disk:

"elasticsearch[node0][management][T#7]" #421 daemon prio=5 os_prio=0 cpu=13232740.84ms elapsed=432098.95s tid=0x00007fb9d8109eb0 nid=0x8bd runnable  [0x00007fbf2fefd000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.FileDispatcherImpl.force0(java.base@15.0.1/Native Method)
        at sun.nio.ch.FileDispatcherImpl.force(java.base@15.0.1/FileDispatcherImpl.java:82)
        at sun.nio.ch.FileChannelImpl.force(java.base@15.0.1/FileChannelImpl.java:465)
        at org.apache.lucene.util.IOUtils.fsync(IOUtils.java:471)
        at org.apache.lucene.store.FSDirectory.fsync(FSDirectory.java:331)
        at org.apache.lucene.store.FSDirectory.sync(FSDirectory.java:286)
        at org.elasticsearch.gateway.MetadataStateFormat.writeStateToFirstLocation(MetadataStateFormat.java:124)
        at org.elasticsearch.gateway.MetadataStateFormat.write(MetadataStateFormat.java:243)
        at org.elasticsearch.gateway.MetadataStateFormat.writeAndCleanup(MetadataStateFormat.java:185)
        at org.elasticsearch.index.seqno.ReplicationTracker.persistRetentionLeases(ReplicationTracker.java:494)
        - locked <0x000000102c49f540> (a java.lang.Object)
        at org.elasticsearch.index.shard.IndexShard.persistRetentionLeases(IndexShard.java:2269)
        at org.elasticsearch.index.seqno.RetentionLeaseBackgroundSyncAction.lambda$shardOperationOnReplica$1(RetentionLeaseBackgroundSyncAction.java:172)
        at org.elasticsearch.index.seqno.RetentionLeaseBackgroundSyncAction$$Lambda$5907/0x0000000801f55ef0.get(Unknown Source)
        at org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:325)
        at org.elasticsearch.index.seqno.RetentionLeaseBackgroundSyncAction.shardOperationOnReplica(RetentionLeaseBackgroundSyncAction.java:168)
        at org.elasticsearch.index.seqno.RetentionLeaseBackgroundSyncAction.shardOperationOnReplica(RetentionLeaseBackgroundSyncAction.java:64)
        at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.onResponse(TransportReplicationAction.java:577)
        at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.onResponse(TransportReplicationAction.java:551)
        at org.elasticsearch.index.shard.IndexShard.lambda$innerAcquireReplicaOperationPermit$30(IndexShard.java:3035)
        at org.elasticsearch.index.shard.IndexShard$$Lambda$5904/0x0000000801f55858.accept(Unknown Source)
        at org.elasticsearch.action.ActionListener$3.onResponse(ActionListener.java:113)
        at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:285)
        at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:237)
        at org.elasticsearch.index.shard.IndexShard.lambda$acquireReplicaOperationPermit$28(IndexShard.java:2974)
        at org.elasticsearch.index.shard.IndexShard$$Lambda$5903/0x0000000801f55620.accept(Unknown Source)
        at org.elasticsearch.index.shard.IndexShard.innerAcquireReplicaOperationPermit(IndexShard.java:3074)
        at org.elasticsearch.index.shard.IndexShard.acquireReplicaOperationPermit(IndexShard.java:2973)
        at org.elasticsearch.action.support.replication.TransportReplicationAction.acquireReplicaOperationPermit(TransportReplicationAction.java:924)
        at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.doRun(TransportReplicationAction.java:653)

The MANAGEMENT threadpool will be blocked, as long as one disk is busy, which is common in produce. The api is relatively important:

1. ._cat/nodes
2. _nodes/stats
3. _cat/thread_pool
4. _cat/tasks ...... too many import metrics are depended on the MANAGEMENT threadpool. even the timeout is added to the api in high version, it doesn't solve the race.

Describe the solution you'd like

If we should create a new threadpool to do with those tasks related to written io.

Related component

Cluster Manager

Describe alternatives you've considered

No response

Additional context

Version: es7.10.2

[rwali-aws]

[Triage - attendees 1 2 3 4 5 6]

Thanks for filing this issue. From the stacktrace shared, it looks like related to ElasticSearch. Please try OpenSearch version and share the debug steps. Feel free to reopen once you add more details.

[kkewwei]

@rwali-aws, there is not much difference whether is opensearch or elasticsearch.

In opensearch, RetentionLeaseBackgroundSyncAction still use MANAGEMENT threadpool, https://github.com/opensearch-project/OpenSearch/blob/6b8b3efe01a62c221f308a2e3b019d75a7f5ad8a/server/src/main/java/org/opensearch/index/seqno/RetentionLeaseBackgroundSyncAction.java#L112 TransportNodesStatsAction still also use MANAGEMENT threadpool, this case will continue to happen, in which we will still can't get cluster metrics. https://github.com/opensearch-project/OpenSearch/blob/6b8b3efe01a62c221f308a2e3b019d75a7f5ad8a/server/src/main/java/org/opensearch/index/seqno/RetentionLeaseBackgroundSyncAction.java#L112

[peternied]

@rwali-aws Did you mean to close this issue after your comment above, looks like this is still untriaged?

[dblock]

[Catch All Triage w/ 1, 2, 3]