Make ThreadPool size dynamic

[gbbafna]

Is your feature request related to a problem? Please describe.

The ThreadPool sizing is only configurable through yml and needs node reboot everytime we need a change .

This is related to https://github.com/opensearch-project/OpenSearch/issues/7832 .

Describe the solution you'd like

A way to increase or decrease threadpool dynamically . Decreasing might be tricky in cases when threads are already occupied .

A clear and concise description of what you want to happen.

Describe alternatives you've considered

Status quo. Do node reboots after yml update on every node.

[andrross]

There are various dynamic thread pools, e.g. ScalingExecutorBuilder. I think what you're asking for here is to be able to dynamically change the limits of the thread pools, right? Would "Make thread pool limits dynamically configurable" be a better title?

[andrross]

Naively, I might argue that optimal thread pool sizes are based on the hardware and therefore wouldn't need to be dynamically configurable. #7832 mentions replacing a fixed thread pool with a scaling thread pool, but doesn't explicitly state why it needs to be dynamically configurable. Can you provide more detail on the use case here?

[anasalkouz]

May be we need evaluate this for specific ThreadPool and specific use-case

[mch2]

+1 to evaluating this by use case given most are computed by processor count. Other than merge thread pool is there another where a dynamic update would be valuable?