Configuring Multi-Datacenter Support

[juicedM3]

From the documentation (http://mesosphere.github.io/cassandra-mesos/docs/multi-dc.html), I need to set CASSANDRA_EXTERNALDC=. Per the example:

CASSANDRA_EXTERNAL_DC_dc1=http://multi-dc.cassandra.marathon.mesos1:10001

What port should I use? I'm assuming it's the port that is dynamically assigned by Marathon for the REST API/Command Executor. Which then makes it extremely hard to configure because I can't guess what it's going to be. I've tried using the PORT0 variable but then the command executor becomes unstable and is frequently restarted. Marathon reports it as continuously deploying.

ZooKeeper: 3.4.6 Mesos: 0.24.1 Marathon: 0.10.0 cassandra-mesos: 0.2.1 java: 7u76

[mohitsoni]

@juicedM3 if possible can you please share some logs? I am thinking of following things that can go wrong:

1. PORT0 replacement failed
2. Node which runs the executor, isnt able to access external-dc-host:port

[juicedM3]

@mohitsoni thanks for the reply. So I might actually want to open a new ticket because the PORT0 option doesn't work for me. Here are the attached logs with just starting a cassandra-mesos cluster without multi-dc support but using the PORT0 option:

  "env": {
    "CASSANDRA_ZK_TIMEOUT_MS": "10000",
    "CASSANDRA_HEALTH_CHECK_INTERVAL_SECONDS": "60",
    "MESOS_ZK": "zk://jb5.example.com:2181,jb6.example.com:2181,jb7.example.com:2181/mesos",
    "JAVA_OPTS": "-Xms256m -Xmx256m",
    "CASSANDRA_CLUSTER_NAME": "multi-dc-cassandra",
    "CASSANDRA_ZK": "zk://jb5.example.com:2181,jb6.example.com:2181,jb7.example.com:2181/cassandra-mesos0",
    "CASSANDRA_NODE_COUNT": "2",
    "CASSANDRA_SEED_COUNT": "1",
    "CASSANDRA_RESOURCE_CPU_CORES": "2.0",
    "CASSANDRA_RESOURCE_MEM_MB": "2048",
    "CASSANDRA_RESOURCE_DISK_MB": "2048",
    "PORT0": "31080"
  },

The logs:

https://drive.google.com/file/d/0BzYhrqsuOpxmMXVyeW5lUVBuUjQ/view?usp=sharing https://drive.google.com/file/d/0BzYhrqsuOpxmcEdjM1dXcUwzVjg/view?usp=sharing https://drive.google.com/file/d/0BzYhrqsuOpxmWE1kMW5CYVdFQU0/view?usp=sharing

I guess it would be good to resolve this and then try the multi-dc support.

[juicedM3]

Well, I've found something that works, but not sure why:

"CASSANDRA_DEFAULT_DC": "DC1"
"CASSANDRA_EXTERNAL_DC_DC0": "http://multi-dc.marathon.mesos-dc0:RANDOM_PORT”

And then for the command:

  "cmd": "CASSANDRA_EXTERNAL_DC_DC0=`echo $CASSANDRA_EXTERNAL_DC_DC0|sed -e \"s/RANDOM_PORT/$PORT0/\"` $(pwd)/jre*/bin/java $JAVA_OPTS -classpath cassandra-mesos-framework.jar io.mesosphere.mesos.frameworks.cassandra.framework.Main”

Why shouldn't it work? We're subbing in the port for DC1 but we really need the port for DC0. We're doing the same thing on DC0. We've tested it and we've seen replication across the two clusters. I can't explain it and it's not pretty.

[juicedM3]

Am I overthinking this? Can you just pick any port and it will work?

[BenWhitehead]

When configuring the cassandra framework to connect with another datacenter, you're essentially telling the framework where to find another framework.

So the env variable CASSANDRA_MULTI_DC_dc1=http://dcos.cassandra.dc1:5080 means that when trying to connect to the other framework instances host is dcos.cassandra.dc1 and port is 5080. What is being specified for the env variable value is actually the base URL to the framework and could be anything that could be expressed as an HTTP url (even if the cassandra framework is only accessible through a reverse-proxy like in dcos).

[triclambert]

This repository is deprecated. Please use the latest supported framework here: https://github.com/mesosphere/dcos-cassandra-service.