Specify multiple search clients for easier benchmarking

Description

Finding the maximum search throughput for an OpenSearch cluster is an important benchmarking scenario in vector search. Currently the only way to figure out the maximum search throughput is to perform multiple benchmark runs with different search client settings (e.g. search_clients = 3, search_clients = 5, ...). Waiting for a run to conclude, changing the config, and rerunning OSB with the associated startup time is tedious. Ideally the maximum throughput could be found automatically.

This PR allows users to provide a clients_list in a operation which runs the operation with each client setting.

For instance, a user might specify "clients_list": [1, 5] in their parameters. Then OSB will schedule search tasks with 1, 5, 10, and 12 clients. The final benchmark results will look something like the following:

|                                                 Min Throughput | search-only_5_clients |     1637.47 |  ops/s |
|                                                Mean Throughput | search-only_5_clients |     1637.47 |  ops/s |
|                                              Median Throughput | search-only_5_clients |     1637.47 |  ops/s |
|                                                 Max Throughput | search-only_5_clients |     1637.47 |  ops/s |
|                                        50th percentile latency | search-only_5_clients |     1.34731 |     ms |
|                                        90th percentile latency | search-only_5_clients |     1.85056 |     ms |
|                                        99th percentile latency | search-only_5_clients |     6.70892 |     ms |
|                                       100th percentile latency | search-only_5_clients |     6.79992 |     ms |
|                                   50th percentile service time | search-only_5_clients |     1.34731 |     ms |
|                                   90th percentile service time | search-only_5_clients |     1.85056 |     ms |
|                                   99th percentile service time | search-only_5_clients |     6.70892 |     ms |
|                                  100th percentile service time | search-only_5_clients |     6.79992 |     ms |
|                                                     error rate | search-only_5_clients |           0 |      % |
|                                  Num clients to max throughput |                       |           5 |        |
|                                                  Mean recall@k | search-only_1_clients |        0.92 |        |
|                                                  Mean recall@1 | search-only_1_clients |        0.99 |        |
|                                                  Mean recall@k | search-only_5_clients |        0.92 |        |
|                                                  Mean recall@1 | search-only_5_clients |        0.99 |        |

Issues Resolved

Closes #613

Testing

[X] New functionality includes testing

Unit tested loader.py changes + verified with multiple OSB runs that result publisher output looks good.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

opensearch-project / opensearch-benchmark