This is example configuration which launches Elasticsearch Cluster on Docker Swarm Cluster. The entire setup distinguish following services:
In order to run it I recommend to have at least 4 VM's provisioned. You can try AWS or even virtualbox running on your local machine. The easiest way is to setup servers using docker-machine. In this example I am using my AWS account (you can specify amazonec2-access-key and amazonec2_security_key in command line or use it ~/.aws/credentials and defined default profile)
Elasticsearch is distributed document oriented database, in our scenario it requires to keep a data in safe place. That's why I used volume features and use constraint to run data node exactly on the origin node where it was initially deployed.
Use replica crea
I moved the entire Elasticsearch configuration into sepreate YML files, because the only a few parameters are taken from environment variables. The same has been done with tunning JVM.
Execute the following command to provision servers:
docker-machine create --driver amazonec2 --amazonec2-region eu-central-1 --amazonec2-instance-type t2.medium --amazonec2-security-group swarm-sg --amazonec2-open-port 9200/tcp --amazonec2-open-port 9300/tcp --amazonec2-open-port 2377/tcp --amazonec2-open-port 7946/tcp --amazonec2-open-port 4789/udp --amazonec2-open-port 7946/udp node-1The given command should be executed 4 times at least, make sure you change hostname every time you are executing the command,the last argument in the command. Please note that created AWS security group is only for demonstration purposes, you never should expose crucial ports to the internet.
Before running a stack, minor changes have to be adopted on each of newaly created Vm's,
vm.max_map_count=262144 or just simple execute
docker-machine ssh node-1 sudo sysctl -w vm.max_map_count=262144for each of the node. Better to add that add it paramentnly:
docker-machine ssh swarm-4 sudo "sed -i '$ a vm\.max_map_count=262144' /etc/sysctl.conf"/etc/systemd/system/docker.service.d/10-machine.conf and modify default ulimit because Elasticsearch cluster is requesting memory locking (see bootstrap.memory_lock=true), thus following parameter to ExecStart --default-ulimit memlock=-1. You can simple execute that command on the each of the node: docker-machine ssh node-1 sudo "sed -i '/ExecStart=\/usr\/bin\/dockerd/ s/$/--default-ulimit memlock=-1/' /etc/systemd/system/docker.service.d/10-machine.conf"Please note that location of that file might be different depending of your Linux distribution. That one works with Ubuntu 16.04 LTS.
After that execute on each of the node systemctl daemon-reload and restart Docker daemon systemctl restart docker
Once vm's are created you can initiate Docker swarm cluster. Export environment variables belonging to the one of the nodes:
eval $(docker-machine env node-1)and initiate a cluster:
docker swarm initthen connect to the other vm's and add them accordingly to the cluster, for test purposes you can just add only workers to the cluster. For production environment pleaae consider the appropriate amount of manager nodes to keep RAFT database secure and provide high availability for your Docker Swarm Cluster.
Make sure the cluster is working by executine command:
docker node lsID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION
iqepxq2w46nprlgm55gomf1ic * node-1 Ready Active Leader 18.09.0
1rruyl7x9s9x43rql0x2jibx0 node-2 Ready Active 18.09.0
yqoycyrs9j0cb1me7cwr77764 node-3 Ready Active 18.09.0
56yflx8vy47oi3upycouzpjpj node-4 Ready Active 18.09.0Swarm provides nice to feature to store configuration files into RAFT, so executing following commands to create appropriate configugration:
docker config create es-coordination es-coordination.yml
docker config create es-data es-data.yml
docker config create es-master es-master.ymlCreate JVM configuration for each instances. I decided to have separate configuration for Coordination, Master and Data nodes to have flexibility in assiging JVM memory to these instances. Coordination and Master nodes usually required less memory than Data nodes.
docker config create jvm-options-coordination jvm.options
docker config create jvm-options-data jvm.options
docker config create jvm-options-master jvm.optionsID NAME CREATED UPDATED
nyh52lw5lgdixdlsi1fol67wv es-coordination 6 minutes ago 6 minutes ago
owfgm5s14igyxkpn9yr619w6y es-data 5 minutes ago 5 minutes ago
jmhz9yac68wa73kmfhw1cu254 es-master 5 minutes ago 5 minutes ago
7v76ed3ahe119t2li3wxumjj4 jvm-options-coordination 3 minutes ago 3 minutes ago
qhcytwr88axtklm4b7as56eur jvm-options-data 3 minutes ago 3 minutes ago
mzj666cjkjkugfuly9z905026 jvm-options-master 3 minutes ago 3 minutes agoThan deploy a stack by executing the following command:
docker stack deploy -c docker-compose.yml esAfter a few minutes you should be able to get cluster state and list of nodes consisting the Elasticsearch cluster.
curl ${IP_ADDRESS}:9200/_cluster/health?pretty{
"cluster_name" : "docker-swarm-cluster",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 11,
"number_of_data_nodes" : 4,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}curl ${IP_ADDRESS}:9200/_cat/node10.0.1.2 27 97 45 2.90 1.39 0.52 - - jeZY0C4
10.0.1.11 44 78 33 1.57 0.82 0.31 - - TtYZYAH
10.0.1.14 28 57 24 0.84 0.43 0.16 d - cCKt4Dd
10.0.1.6 30 97 44 2.90 1.39 0.52 m - g_QL8du
10.0.1.4 40 97 43 2.90 1.39 0.52 d - 5j_gCcq
10.0.1.8 27 97 43 2.90 1.39 0.52 m - vOmxG79
10.0.1.17 43 57 23 0.88 0.39 0.14 - - Af3r11-
10.0.1.15 28 57 24 0.84 0.43 0.16 - - 1DGS4Xd
10.0.1.9 26 78 33 1.57 0.82 0.31 d - QEWrVND
10.0.1.10 35 78 33 1.57 0.82 0.31 m * XugNq4c
10.0.1.19 45 57 23 0.88 0.39 0.14 d - vSI0shWDue to service mesh feature built-in Docker Swarm you choose any of the IP addresses of your nodes. Docker Swarm will route the request to the appropriate coordination node running on virtual machines being a member of Docker Swarm cluster.
docker stack lsNAME SERVICES ORCHESTRATOR
elastic 7 Swarmdocker service lsID NAME MODE REPLICAS IMAGE PORTS
4ygj2afsednm elastic_coordination global 4/4 elasticsearch:6.5.3
c3xgunhn6c7d elastic_data1 replicated 1/1 elasticsearch:6.5.3
nz4k8f4zvkv6 elastic_data2 replicated 1/1 elasticsearch:6.5.3
7d3wjyi5s083 elastic_data3 replicated 1/1 elasticsearch:6.5.3
oqq9q0vy6t9q elastic_data4 replicated 1/1 elasticsearch:6.5.3
uszn9mxmeubh elastic_master replicated 3/3 elasticsearch:6.5.3
5o3usc3ncbb5 elastic_visualizer replicated 1/1 dockersamples/visualizer:latest *:8080->8080/tcpPlease note that service coordination has to replicate mode set to global, it means that we are requesting to have Elasticsearch coordination instance running on each of the node consisting Docker Swarm cluster. Other services have defined replicas as an integer.
docker service ps elastic_coordinationID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
y6lfnbnavy7z elastic_coordination.yqoycyrs9j0cb1me7cwr77764 elasticsearch:6.5.3 node-3 Running Running 2 minutes ago *:9200->9200/tcp
1f1xk71zug9z elastic_coordination.iqepxq2w46nprlgm55gomf1ic elasticsearch:6.5.3 node-1 Running Running 2 minutes ago *:9200->9200/tcp
fpu2bdmnnfl2 elastic_coordination.56yflx8vy47oi3upycouzpjpj elasticsearch:6.5.3 node-4 Running Running 2 minutes ago *:9200->9200/tcp
l8lozi001l2l elastic_coordination.1rruyl7x9s9x43rql0x2jibx0 elasticsearch:6.5.3 node-2 Running Running 2 minutes ago *:9200->9200/tcp