Throughput varies over time without Liberty executor tuning

[punktilious]

Describe the bug Performance tests show that in high load (request rate) scenarios, throughput is seen to vary a lot over time. The root cause isn't yet known, but using a fixed executor configuration addresses the issue. In server.xml:

    <executor coreThreads="40" maxThreads="40"/>   

Environment Which version of IBM FHIR Server: 4.10.1

To Reproduce Steps to reproduce the behavior:

1. Load synthea test data
2. Run fhir-bucket "interop" workload with a moderate thread-count
3. Observe the throughput rate vary over time
4. Add the executor configuration to the server.xml and restart the server
5. Run fhir-bucket "interop" workload with a moderate thread-count
6. Observe the throughput is improved and steady over time

Expected behavior Liberty should automatically tune the thread pool size based on load, so it's unclear why the fixed executor configuration makes such a difference.

Additional context This behavior had been observed before in much older versions of Liberty Profile. The executor thread pool tuning has since been improved and it was thought the fixed executor configuration was no longer required (the Liberty docs recommend to avoid such tuning if possible). More investigation is warranted, but fixed executor configuration above is a good work-around.

[lmsurpre]

1. set a reasonable default
2. document how to override this, probably in our Performance Guide

optionally: do further investigation / open an issue for liberty team to look into why the auto-tuning isn't working as expected for us