Avoid retrying bad search requests on replicas

[jtibshirani]

If a shard fails during a search phase, then we retry the shard request on its replicas. This makes sense for many types of errors, but for bad_request errors we know that the replicas will fail as well. Could we avoid retrying on replicas in this case? Some examples of bad request errors we see in search: invalid query syntax, too many docvalue_fields requested, attempting to run a query or agg on a field whose type isn't supported.

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[dnhatn]

+1

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)