How does a kafka cluster handle ephemeral, containerized brokers

[dbykat]

If using the ConfluentInc docker images deployed on a docker cluster, are Kafka and Zookeeper capable of handling the ephemeral, elastic nature of containers? Docker runs under the assumption that a container (e.g. a kafka broker) can run anywhere on the cluster. To this end, if the containerized broker dies on one linux worker node, it can be brought up on an entirely different linux worker node. Note: our docker cluster is utilizing local storage on each worker node of the docker cluster, so if a broker spins up on a new worker, the storage volume will be empty requiring replication of partitions.

Can the Kafka cluster be expected to handle replication and leader elections under these exceptional conditions? what about zookeeper?

What I'm hoping for? In the scenario that we have Min ISR configured to 2, producer acks set to all, and our cluster only has 1 live broker left, I’d want new brokers to be spun up allowing us to continue using the cluster (even if that means all leaders are on the same broker temporarily). For this to work, when the new kafka containers join the cluster, we'd need the partitions replicated to the new brokers, the new brokers to join the ISRs, and for the cluster to return to health automatically.

• Is this possible for kafka?
• or must we use the auto rebalancer tool when new containers join the cluster, meaning manual intervention?
• Does it matter if they have the same broker_id, or should they have new broker_ids?
• is zookeeper also tolerant of this elastic node scenario?

Ideally, for both Kafka and Zookeeper, we'd want full replication and leader elections when the new containers join the cluster, and we'd want this to happen with no manual intervention.

I'm going to test this behavior, but hoping someone could set my expectations for what to expect.

Thank you!

[chicagobuss]

To accomplish what you're asking people generally use a container orchestration platform that offers persistent volume support or resource pinning, like dc/os or kubernetes.

I recommend reading more on the subject here: https://kubernetes.io/docs/concepts/storage/persistent-volumes/

If you do not have an environment with shared storage, then you need to do even more manual orchestration / work like you're describing.

Let me know if that helps.

[dbykat]

Thanks @chicagobuss . We are using Docker Enterprise/DataCenter orchestration, and we do utilize persistent volumes as you said. Here's the rub, however... containers only persist to the local host that they're running (i believe kubernetes offers some statefulset functionality for the volume to follow to other hosts, but we're not using kubernetes).

If the container dies on host 1, and is spun up on host 2, host 2's persistent volume will be empty as no container has existed on host 2 before. Thus, replication must be triggered. It's up to the Kafka application to recognize that a new broker just joined the cluster, and begin replicating to that new broker. Basically asking how Kafka handles a new broker joining the cluster? If there were not enough ISRs to meet the "minimum ISR" configuration for the cluster, would partitions replicate to the new broker automatically until it ultimately joins the ISR list? As i mentioned, i certainly plan to test this scenario, but curious if anyone is using these confluentinc docker images, and whether their clusters are self-healing in this way, and if so... what are the best practices... (for example, use static broker_ids, etc...)

[ewencp]

@dbykat Kafka handles that just fine. You can run Kafka under a system that doesn't have state follow the process, e.g. under Marathon in Mesos or even just a traditional orchestration tool. However, you do have to be more careful as you need to all the brokers for a partition don't all go offline at once (which obviously you generally want, but at least if you can track the location of the state you can recover from).

Generally you will want to reuse the broker IDs if possible since otherwise you'll need to reassign partitions when brokers die or are restarted.

[GMartinez-Sisti]

Also please take a look at this issue https://github.com/confluentinc/cp-docker-images/issues/608. Even with PVC's attached you'll lose your data.