Envoy multi-zone deployment

[alungu]

Is your feature request related to a problem? Please describe. The current ingress implementation has no guarantee that the ingress and the Kafka broker would be deployed in the same data center. While this might be good for availability, it is a cost vector in any public cloud infrastructure, since any inter-DC traffic is paid traffic, while the intra-DC traffic is, usually, free.

Describe the solution you'd like to see We'd like to have the option to deploy a DC aware ingress, with one deployment for each broker config group. The ingress deployment should honor the NodeAffinity and the NodeSelector configurations of the group, to ensure that both the ingress and the broker pods (from the same group) would be scheduled in the same data center.

The schema below describes the deployment using a data-center aware ingress (envoy) deployment. This is already implemented in our fork: https://github.com/adobe/kafka-operator/pull/15 We are running with this config for awhile now, but the implementation was updated recently to reflect the new ingress deployment updates (ingress config map is honoring the Cluster status instead of its spec).

Some notes regarding the implementation:

• The change is backward compatible
• The change is implemented for Envoy only (it's an envoy specific configuration - dedicatedEnvoyPerBrokerGroup); there is not support for Istio ingress in the existing implementation
• The deployment requires multiple single-DC Load Balanacers (see schema below). To minimize the impact on the Kafka operator, we created this LBs externally and added a new flag in the operator - useExistingLoadBalancer). This option could be used independently from dedicatedEnvoyPerBrokerGroup, if the administrator wants a self-managed LoadBalancer.
• A series of NodePorts were created externally (from a chart). We should analyze if the operator should create the NodePorts automatically if useExistingLoadBalancer is enabled
• a new hostnameOverride broker config was added. In the current implementation, since it was assumed that there is a single LoadBalancer, all the brokers use the same hostname for the external listener. Now, each broker or broker config group can use a different address (and a different LB)

[amuraru]

@alungu Adi I suggest we share our implementation here as a PR and discuss it.