Provide capability to add additional external nodes to cluster

[soudaburger]

What would you like to be added?

I basically would like to be able to deploy a second set of kafka pods in a separate cluster in a separate region to allow them to replicate data between my regions. This would ensure that even in the event of a failover DR event, I don't actually have to modify any deployments of services using kafka, as both regions effectively become active.

Example:

1. Deploy redpanda helm chart with stateful set to us-east1 as a starting cluster. Let's assume 3 nodes.
2. Validate functionality of cluster and everything is healthy.
3. Deploy redpanda helm chart with additional statefulset to us-west1 as an "expansion". Let's also assume 3 nodes.

This would provide me with 6 Kafka nodes, all able to take traffic and replicate accordingly. I could, in theory, allow my services to write to either region and support running a hot/hot architecture, which is desired.

I assume that I would need to have the following set up in order to make this functional:

1. Networking connectivity between both clusters.
2. Common CA infrastructure between the two, or otherwise have certificate trust shared/propagated.
3. Proper configuration of DNS/hosts to support external hostnames advertised by kafka.

Why is this needed?

This would allow me to run a hot/hot cross-regional deployment on separate clusters to avoid certain catastrophic failure scenarios.

JIRA Link: K8S-37

[soudaburger]

This would also help in the situation of: #481

[alejandroEsc]

There is some discussion about this being done, however this requires additional tooling on top of what we have and will probably have in this chart. Please stay tuned.