[FEATURE] Add availability and scalability features to all clients

[Bukhtawar]

Is your feature request related to a problem? The clients need to be smarter espl when the system is overloaded and traffic pattern needs to be adjusted to minimize downtimes giving sufficient time for the system to recover. The clients need to ensure that they maximum server uptime without sacrificing too much on performance

What solution would you like?

1. Handling too many client connections
2. Preventing retry storm
3. Managing/safe-guarding critical system resources
4. Adaptive timeouts and backoffs
5. Penalizing bad connections/erroneous targets
6. Load balancing across targets
7. Automatic pagination on larger response
8. Asynchronous searches on slower requests
9. Query prioritisation
10. Client-side metrics on data sources like AWS Cloudwatch

What alternatives have you considered? A clear and concise description of any alternative solutions or features you've considered.

Do you have any additional context? Add any other context or screenshots about the feature request here.

[brijos]

Hi @Bukhtawar (and others),

Are there examples of clients which do it better than others?

[dblock]

I've renamed this to "Add availability and scalability features to all clients". YMMV client-by-client, for example Java client does support retries "out-of-the-box". This is a good meta issue with a list of high level features.

[wbeckler]

A couple more scalability features would be:

• Adaptive worker pool sizing (increase or decrease number of workers to find the sweet spot)
• Adaptive bulk operation sizing (adjust number of documents to include in a bulk index operation toward a sweet spot)