[ENHANCEMENT] Implement SniffingConnectionPool

[russcam]

Is your feature request related to a problem? Please describe. The client can work against an Elasticsearch cluster behind a single endpoint e.g. a single node cluster, a cluster behind a proxy/load balancer. It is also useful however to be able to seed a client with a collection of endpoints, where each endpoint connects to an Elasticsearch node. In doing so, it allows for future features such as retries/failover, executing requests on a node matching a predicate e.g. data nodes only, etc. Since Elasticsearch clusters can change in size, the client should be able to reseed itsefl with the nodes (endpoints) available within the cluster. This is commonly referred to across the other Elasticsearch clients as sniffing.

Describe the solution you'd like A SniffingConnectionPool that implements ConnectionPool that can

1. be seeded with an initial collection of Url
2. returns the next Connection from the pool when next() is called. In the initial implementation, this can simply iterate over connections, but in future, can mark connections dead/alive, weight them by aliveness, etc. to determine the next connection to return
3. can periodically reseed the connections by sniffing the cluster

Additional context

An example implementation of SniffingConnectionPool from the .NET client and the behaviour

[srleyva]

I’ve been implementing this locally over the elasticsearch vended primitives. Still a little newer to rust but mind if I take a shot at this issue?

[russcam]

@srleyva please do! Please also reach out if you would like input on implementation considerations.

[srleyva]

Similar to the .Net client, I’ve implemented a static node list connection pool as a building block. I’ve noticed a common pattern of building on top of this and refreshing the connections. Lmk if this is sound! 😀