Is your feature request related to a problem? Please describe
We find the took in search response is smaller than the external statistics timeout in product, partial reason is that some threads of worker thread pool are relatively busy, the search requests are pending on the worker queue. Yet there is not available api, to get the statistics of worker queue, we have to add metrics and build the source code, If fact, There is a serious queuing in rarely case:
97, 64, 68, 108, 133, 61, 62, 123, 70, 66, 118, 110, 65, 52, 95, each number represents one of the worker threads's pending tasks, It is very difficult to troubleshoot when it happens.
Just as described in #639 and #13662 and #13741.
Describe the solution you'd like
If we could monitor the queue, add api to get those metrics, and print log in server log? I'm pleasure to implement it.
dblock
commented
1 month ago
Is your feature request related to a problem? Please describe
We find the took in search response is smaller than the external statistics timeout in product, partial reason is that some threads of worker thread pool are relatively busy, the search requests are pending on the worker queue. Yet there is not available api, to get the statistics of worker queue, we have to add metrics and build the source code, If fact, There is a serious queuing in rarely case: 97, 64, 68, 108, 133, 61, 62, 123, 70, 66, 118, 110, 65, 52, 95, each number represents one of the worker threads's pending tasks, It is very difficult to troubleshoot when it happens.
Just as described in #639 and #13662 and #13741.
Describe the solution you'd like
If we could monitor the queue, add api to get those metrics, and print log in server log? I'm pleasure to implement it.
Related component
Cluster Manager
Describe alternatives you've considered
No response
Additional context
No response