Support for clustering Kafka via new Docker swarm mode?

spotify / docker-kafka

Kafka (and Zookeeper) in Docker

Apache License 2.0

1.4k stars 640 forks source link

Support for clustering Kafka via new Docker swarm mode? #45

Open bluepuma77 opened 8 years ago

bluepuma77 commented 8 years ago

Are there plans to support clustering Kafka via Docker swarm mode, which is new in Docker v1.12.0? With the new swarm you can define services and automatically scale them up or down. Plus docker handles load balancing automatically between instances of the service (default round robin).

docker service create --replicas 1 --name kafka spotify/kafka
docker service scale kafka=3

Intro: https://docs.docker.com/engine/swarm/ Concepts: https://docs.docker.com/engine/swarm/key-concepts/

baseman commented 8 years ago

You might be looking for something like this...

https://github.com/wurstmeister/kafka-docker

You can use this projects against your swarm using docker compose. Note: I'm about to try this myself.

brandonlamb commented 8 years ago

Got here from google on this.

Docker Swarm != Docker Engine Swarm.

Docker Engine Swarm is not currently compatible with docker compose (it will just start all containers on the single host).

My team is trying to figure out how to handle the new Docker Engine Swarm and kafka

ygizhitsa commented 7 years ago

with the new engine 1.12 kafka might not need to use zookeeper . topics and partitions also might fit beautifully with stacks and bundles (experimental mode)

mostolog commented 7 years ago

@ygizhitsa Could you provide a link or more info about that?

shay1bz commented 5 years ago

Hi, any 2019 update on this issue?