Docker Swarm support

[sarath-mec]

Hi,

I was able to bring this up in two node cluster using Swarm 1.13. Just wanted to share how it was done for all

First create a swarm by

docker swarm init --advertise-addr 10.1.0.4

And then use the command provided to join the other hosts

git clone https://github.com/sarath-mec/smks-docker-hdp.git

We used a combination of a) Docker attachable Overlay Network for swarm b) docker stack deploy for psql and ambari-server c) docker-compose for running individual nodes

Used this approach as privileged for nodes is not supported for docker swarm services But we can create an overlay attachable network and run the containers into the swarm network

All containers are discoverable by hostname by using this approach

---

$ HDP Swarm using stack deploy and docker-compose using external overlay network

both manager and worker hosts to pull images

docker pull sarath4mec/node docker pull sarath4mec/ambari-server docker pull sarath4mec/postgres

create volume directories in both nodes

sudo su cd /home/sarath_mec rm -rd hdfs rm -rd postgresql mkdir hdfs mkdir postgresql mkdir hdfs/dn0 mkdir hdfs/dn1 mkdir postgresql/data chmod 777 hdfs chmod 777 hdfs/dn0 chmod 777 hdfs/dn1 chmod 777 postgresql/data cd smks-docker-hdp

create network

docker network create -d overlay --subnet 10.0.9.0/24 --attachable hdfs

stack deploy postgres and ambari in both manager and worker host

docker stack deploy --compose-file examples/compose/swarm-psql-ambari-container.yml hdfs

inspect docker network from both nodes

docker network inspect hdfs

see the network and virtual ip assigned from manager node

docker service inspect --format='{{json .Endpoint.VirtualIPs}}' hdfs_postgres docker service inspect --format='{{json .Endpoint.VirtualIPs}}' hdfs_ambari-server

stack deploy individually psql and ambari

docker stack deploy --compose-file examples/compose/swarm-psql-container.yml hdfs docker stack deploy --compose-file examples/compose/swarm-ambari-container.yml hdfs

docker-compose and bring up dn0.dev in swarm manager host

docker-compose -f examples/compose/swarm-dn0-container.yml up -d

docker-compose and bring up dn0.dev in swarm worker worker host

docker-compose -f examples/compose/swarm-dn1-container.yml up -d

docker-compose and bring up dn0.dev in swarm manager host

docker-compose -f examples/compose/swarm-dn0-container-resume.yml up -d

docker-compose and bring up dn0.dev in swarm worker worker host

docker-compose -f examples/compose/swarm-dn1-container-resume.yml up -d

Clear everything

docker stack rm hdfs

manager host

docker stop compose_dn0_1 docker rm compose_dn0_1

worker host

docker stop compose_dn1_1 docker rm compose_dn1_1

docker network rm hdfs

Thanks, Sarath

[devopsacid]

Hi Sarah, does your installation works now? I have problem with blueprint deployment as components are in PENDING HOST ASSIGNMENT state.