Can't start multiple Cassandra instances on same host

[KNejad]

I am following the instructions so I run these 2 commands hoping to have 2 Cassandra nodes running on the same machine:

sudo docker run --name cassandra1 -d --network host  cassandra:latest
sudo docker run --name cassandra2  --network host -e CASSANDRA_SEEDS=cassandra1 cassandra:latest

The second command exits complaining that "The seed provider lists no seeds"

WARN  [main] 2019-07-24 14:41:18,808 SimpleSeedProvider.java:60 - Seed provider couldn't lookup host cassandra1
Exception (org.apache.cassandra.exceptions.ConfigurationException) encountered during startup: The seed provider lists no seeds.
The seed provider lists no seeds.
ERROR [main] 2019-07-24 14:41:18,810 CassandraDaemon.java:749 - Exception encountered during startup: The seed provider lists no seeds.

This is the output of sudo docker ps -a:

CONTAINER ID        IMAGE                    COMMAND                  CREATED             STATUS                     PORTS                                                NAMES
94515a0ac1c0        cassandra:latest         "docker-entrypoint.s…"   8 seconds ago       Exited (3) 4 seconds ago                                                        cassandra2
5ca9fc2ad653        cassandra:latest         "docker-entrypoint.s…"   12 seconds ago      Up 11 seconds                                                                   cassandra1

I am following the instructions from the docker page. The only thing I have had to play around with is the network which I set to "host" but I also tried "bridge" and both seem to fail with the same issue.

Am I do anything wrong here?

[wglambert]

You're missing CASSANDRA_BROADCAST_ADDRESS and network=host isn't going to work for containers using the same ports

$ docker run -d --name cassandra1 -m 4g --network=net -e CASSANDRA_BROADCAST_ADDRESS=cassandra1 cassandra:latest
e0296a7562b3493fe4e2ea419a8161052659cca45cda0c41b2e3c15cf8e1128d

$ docker run -d -m 4g --name cassandra2 --network=net -e CASSANDRA_SEEDS=cassandra1 cassandra:latest
fd7511059902fa928904ead88d4b17b1e3d3377ae1ef80f900c7c212bbce0147

$ docker ps | grep -i cassandra
fd7511059902        cassandra:latest       "docker-entrypoint.s…"   23 seconds ago      Up 17 seconds       7000-7001/tcp, 7199/tcp, 9042/tcp, 9160/tcp   cassandra2
e0296a7562b3        cassandra:latest       "docker-entrypoint.s…"   43 seconds ago      Up 39 seconds       7000-7001/tcp, 7199/tcp, 9042/tcp, 9160/tcp   cassandra1

$ docker exec cassandra2 nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load       Tokens       Owns (effective)  Host ID                               Rack
UN  172.18.0.2  103.63 KiB  256          100.0%            d5a55b81-d40c-4064-b009-33159dc60ffd  rack1
UN  172.18.0.3  117.55 KiB  256          100.0%            fdc96a0a-66a1-4c64-8582-0b0e56cdf9b4  rack1

[KNejad]

Thanks @wglambert I tried yours using network as bridge as well as my-network created with sudo docker network create my-network.

With bridge I am getting:

Exception (org.apache.cassandra.exceptions.ConfigurationException) encountered during startup: Unknown broadcast_address 'cassandra1'
Unknown broadcast_address 'cassandra1'
ERROR [main] 2019-07-24 19:50:55,537 CassandraDaemon.java:749 - Exception encountered during startup: Unknown broadcast_address 'cassandra1'

And with my-network I am getting this issue:

WARN  [main] 2019-07-24 19:51:38,084 NativeLibrary.java:187 - Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK or run Cassandra as root.
INFO  [main] 2019-07-24 19:51:38,085 StartupChecks.java:140 - jemalloc seems to be preloaded from /usr/lib/x86_64-linux-gnu/libjemalloc.so.1
WARN  [main] 2019-07-24 19:51:38,085 StartupChecks.java:169 - JMX is not enabled to receive remote connections. Please see cassandra-env.sh for more info.
INFO  [main] 2019-07-24 19:51:38,087 SigarLibrary.java:44 - Initializing SIGAR library
INFO  [main] 2019-07-24 19:51:38,095 SigarLibrary.java:180 - Checked OS settings and found them configured for optimal performance.
WARN  [main] 2019-07-24 19:51:38,097 StartupChecks.java:311 - Maximum number of memory map areas per process (vm.max_map_count) 65530 is too low, recommended value: 1048575, you can change it with sysctl.
WARN  [main] 2019-07-24 19:51:38,104 StartupChecks.java:332 - Directory /var/lib/cassandra/data doesn't exist
WARN  [main] 2019-07-24 19:51:38,110 StartupChecks.java:332 - Directory /var/lib/cassandra/commitlog doesn't exist
WARN  [main] 2019-07-24 19:51:38,110 StartupChecks.java:332 - Directory /var/lib/cassandra/saved_caches doesn't exist
WARN  [main] 2019-07-24 19:51:38,111 StartupChecks.java:332 - Directory /var/lib/cassandra/hints doesn't exist
INFO  [main] 2019-07-24 19:51:38,159 QueryProcessor.java:116 - Initialized prepared statement caches with 15 MB (native) and 15 MB (Thrift)

which I'm not sure why that would be happening.

For reference the exact commands I ran was:

sudo docker network  create my-network
sudo docker run --name cassandra1 --network=my-network -m 4g -e CASSANDRA_BROADCAST_ADDRESS=cassandra1 cassandra:latest

Again just to confirm running sudo docker ps -a returns:

CONTAINER ID        IMAGE                    COMMAND                  CREATED             STATUS                        PORTS                                                NAMES
5740ec63757d        cassandra:latest         "docker-entrypoint.s…"   27 seconds ago      Exited (137) 23 seconds ago                                                        cassandra1

[wglambert]

Looks like some odd Java memory limit issue

Increase RLIMIT_MEMLOCK or run Cassandra as root.

Maybe remove the -m 4g I just have it so my system doesn't start oomkilling from two cassandras running, maybe you'll have to include something like -e HEAP_NEWSIZE=600M -e MAX_HEAP_SIZE=1024M https://github.com/docker-library/cassandra/issues/144#issuecomment-433940584

[KNejad]

Yeah I removed -m 4g and it seemed to work. I was just testing things out then I was going to let you know. Perhaps the setup instructions could be clearer that you can't use host or bridge as the networks and that CASSANDRA_BROADCAST_ADDRESS=cassandra1 needs to be set?

Not being able to use host is pretty obvious now that I think about it but why wouldn't bridge work?

[wglambert]

We would have to include the notice for every image, it's a Docker nuance https://docs.docker.com/network/bridge/

User-defined bridge networks are superior to the default bridge network.

There's a prominent section Differences between user-defined bridges and the default bridge

[KNejad]

Ah I see that's fair enough. I still see a value in specifying CASSANDRA_BROADCAST_ADDRESS since the instructions specify to run the command $ docker run --name some-cassandra --network some-network -d cassandra:tag which gives no mention to the broadcast address.

[wglambert]

On the same machine you don't need the broadcast address variable.

$ docker run -d --name cassandra1 -m 4g --network=net cassandra:latest
33ff700875738b99a79151fb181690a341fe309ffaf11db69970a771bb2a8d6f

$ docker run -d -m 4g --name cassandra2 --network=net -e CASSANDRA_SEEDS=cassandra1 cassandra:latest
4c2f215f82d05310a7dc8f97c8b644497f58721565e1267851ae8525c87cb1d1

$ docker exec cassandra2 nodetool status
Datacenter: datacenter1                                                                                                                           
=======================                                                                                                                           
Status=Up/Down                                                                                                                                    
|/ State=Normal/Leaving/Joining/Moving                                                                                                            
--  Address     Load       Tokens       Owns (effective)  Host ID                               Rack                                              
UN  172.18.0.2  108.63 KiB  256          100.0%            4e889383-4cb0-45d4-926a-ab7ed6cec616  rack1                                            
UN  172.18.0.3  93.95 KiB  256          100.0%            3d1d737f-1c7d-4dab-8f4c-812fe27b0a29  rack1

For that section in the docs https://github.com/docker-library/docs/tree/master/cassandra#make-a-cluster

there are two cluster scenarios: instances on the same machine and instances on separate machines.

The latter examples show the CASSANDRA_BROADCAST_ADDRESS

[KNejad]

Ah okay sorry I thought that was necessary now. So all I was doing wrong was that I didn't create a network and was hoping "host" or "bridge" would work. Okay in that case yeah it makes sense the docs are the way they are. I understand as well why you don't mention the "host"/"bridge" thing.

Okay thank you for all your help with this. I guess I should have just read up on docker more before getting started.