Support for deployment with Docker Swarm

[andtan91]

Would be a nice feature to have support and documentation for deploying a cluster with Docker Swarm.

[aahmed-se]

for testing or production ?

[andtan91]

What exactly do you mean for testing? For myself, I'd use the swarm setup for both testing and production environments.

[alexanderursu99]

Noticed there are some compose files under /tests/compose that I think arrange a Pulsar cluster, but use test images. Can this be adapted to use production images?

[alexanderursu99]

I've managed to get a cluster working with Docker Swarm by adapting the instructions from the bare metal cluster deployment outlined here in the official documentation.

There are 4 stacks defined with compose files, and are brought up in the natural order:

1. pulsar-zookeeper.yml
2. pulsar-metadata.yml
3. pulsar-bookies.yml
4. pulsar-brokers.yml

The source code can be found in the upload zip: pulsar-docker-swarm.zip

I'd appreciate any feedback, questions, or comments.

[alexanderursu99]

I'm hoping this can be developed further and soon be put up as official documentation. A few goals of mine with this setup in the future are:

1. Add health checks
2. Automate with Ansible
3. Blue/green deployment

Anything to help make deployment more convenient and production ready.

[gohmc]

pulsar-metadata.yml:

Instead of hardcode each node; use swarm service name (note changes to pulsar-brokers.yml below):

--web-service-url http://broker:8080 --broker-service-url pulsar://broker:6650

pulsar-bookies.yml:

Instead of repeating bookie, use replicas instead

bookie: image: apachepulsar/pulsar command: > /bin/bash -c "bin/apply-config-from-env.py conf/bookkeeper.conf && bin/pulsar bookie" networks:

• pulsar environment: zkServers: zk1:2181,zk2:2181,zk3:2181 deploy: replicas: 3 placement: max_replicas_per_node: 1 constraints: [node.role == manager]

pulsar-brokers.yml:

Should expose 6650/8080 for ingestion

broker: image: apachepulsar/pulsar command: > /bin/bash -c "bin/apply-config-from-env.py conf/broker.conf && bin/pulsar broker" networks:

• pulsar **ports:
• 6650:6650
• 8080:8080 environment: zookeeperServers: zk1:2181,zk2:2181,zk3:2181 configurationStoreServers: zk1:2181,zk2:2181,zk3:2181 clusterName: pulsar-cluster-1 brokerServicePort: "6650" webServicePort: "8080" deploy: replicas: 3 placement: max_replicas_per_node: 1** constraints: [node.role == manager]

[tisonkun]

Closed as stale. Please open a new issue if it's still relevant in maintained versions.

We have the docker image so this is possible. But two years later there's little interest on implementing this feature among the community. If an optional improvement doesn't have a concrete proposal and no contributor works on it, it's valueless to keep it open.

[dadafan]

通过调整官方文档中概述的裸机集群部分的说明，我已经设计方法让集群与 Docker Swarm 一起工作。

使用compose 文件确定了4个堆垛，并按自然顺序排列：

1. pulsar-zookeeper.yml
2. pulsar-metadata.yml
3. pulsar-bookies.yml
4. pulsar-brokers.yml

源代码可以在上传的zip中找到：pulsar-docker-swarm.zip

如果有任何反应、问题或意见，我将不胜感激。

I used version 2.11.0 of pulsar and encountered an issue at this time. When deploying meta.yml, I was unable to successfully create it.