Support adaptive replica selection and preference

[CAFxX]

Description

Currently if the preference is set during a search operation it disables adaptive replica selection for that query. This is understandable, as the two mechanisms are different and conflicting approaches for replica selection.

In some cases it would be ideal though if there was a hybrid schema that allowed to reap the benefit of both, and specifically improving caching hit ratios (by using a custom string preference to steer similar requests to the same nodes) while avoiding hotspots (by using adaptive replica selection to spread the load only in case it becomes excessive on any individual node).

The ideal solution would steer requests for the same preference string to the minimum number of replicas required to prevent any individual node from becoming overloaded.

So e.g.

• if all nodes are not overloaded, it would behave as today when a custom preference string is specified
• as one or more nodes become overloaded, queries that would hit those overloaded nodes are increasingly spread to an increasing number of alternate replicas of the shards on the overloaded nodes
• in the pathological example of a single custom preference string being used on 100% of requests it should therefore behave as if today adaptive replica selection was enabled and no preference was set.

[elasticsearchmachine]

Pinging @elastic/es-distributed (Team:Distributed)