Jsonnet: Support deploying queriers in multiple zones

[stevesg]

Is your feature request related to a problem? Please describe.

Using querier_allow_multiple_replicas_on_same_node can be useful for increasing node utilization, but introduces the risk that in the case that all queriers fit on a single node. If that node becomes unresponsive, the read path is down until pods are moved to different nodes.

Describe the solution you'd like

• Jsonnet should have an option for deploying queriers into multiple Deployments (e.g. zone-a, zone-b), though pods in each zone won't be configured any differently.
• Each Deployment should have a separate anti-affinity rule, such that queriers from different zones cannot be placed on the same node, but queriers in the same zone can (same rules as for e.g. ingester multi-zone)
• This should apply to querier and ruler-querier

This will allow better node utilization, whilst preserving availability of queriers.

Arguably the same logic should apply to query-frontend and query-scheduler.

[pracucci]

I understand the use case, but I think using multi-zone is an overcomplicated solution, which will also make autoscaling more complicated.

I think what you want is just a preferred (not required) anti-affinity rule based on the node topology key, so that if the pod can be scheduled on a node where no other querier is running, then it will, otherwise it will be scheduled on a node where another querier is already running.