Cassandra is not configured to take rack placement into account during replication

Cassandra instances in docker containers use a default configuration taken from upstream packaging (/etc/cassandra/cassandra-rackdc.properties) which means that they do not override rack= settings used during replication with NetworkTopologyStrategy.

https://github.com/apache/cassandra/blob/cassandra-3.11.3/conf/cassandra-rackdc.properties#L17-L20 (upstream config)

Configuring this parameter is essential for handling rack failure scenarios.

Juju exposes availability zones configured for nodes in MAAS via JUJU_AVAILABILITY_ZONE environment variable available during hook execution which can be retrieved and used to render a proper config file for Cassandra: https://docs.jujucharms.com/juju-environment-variables#heading--juju_availability_zone

Cassandra config in all containers:

/etc/cassandra# cat cassandra-rackdc.properties 
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# These properties are used with GossipingPropertyFileSnitch and will
# indicate the rack and dc for this node
dc=dc1
rack=rack1

# Add a suffix to a datacenter name. Used by the Ec2Snitch and Ec2MultiRegionSnitch
# to append a string to the EC2 region name.
#dc_suffix=

# Uncomment the following line to make this snitch prefer the internal ip when possible, as the Ec2MultiRegionSnitch does.
# prefer_local=true

Cassandra status sample:

nodetool status -n 7200

Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address         Load       Tokens       Owns (effective)  Host ID                               Rack
UN  192.0.2.16   67.12 GiB  256          65.6%             348afae6-5b14-4303-bb43-4a1ed91aca44  rack1
UN  192.0.2.12   187.42 GiB  256          66.5%             ff7be60f-21ed-49f3-97e2-af6c2aaeaa5b  rack1
UN  192.0.2.205  85.49 GiB  256          68.0%             bff521fc-c87e-4b38-ac80-50319224ea7f  rack1

Also, to add to the context, keyspaces created in Cassandra (config and analytics) use SimpleStrategy instead of NetworkTopologyStrategy and some have replication_factor set to 1 or 2, not 3.

system_auth keyspace specifically has replication_factor set to 1 while the recommended one would be 3 for HA deployments.

https://docs.datastax.com/en/security/6.0/security/secSystemKeyspace.html

"The default replication factor for the system_auth and dse_security keyspaces is 1.
Each of these must be updated in production environments to avoid data loss. DataStax recommends changing the replication factor before enabling authentication. "

cqlsh> SELECT * FROM system_schema.keyspaces;
 keyspace_name        | durable_writes | replication
----------------------+----------------+-------------------------------------------------------------------------------------
 ContrailAnalyticsCql |           True | {'class': 'org.apache.cassandra.locator.SimpleStrategy', 'replication_factor': '2'}
          system_auth |           True | {'class': 'org.apache.cassandra.locator.SimpleStrategy', 'replication_factor': '1'}
        system_schema |           True |                             {'class': 'org.apache.cassandra.locator.LocalStrategy'}
         config_webui |           True | {'class': 'org.apache.cassandra.locator.SimpleStrategy', 'replication_factor': '1'}
   system_distributed |           True | {'class': 'org.apache.cassandra.locator.SimpleStrategy', 'replication_factor': '3'}
               system |           True |                             {'class': 'org.apache.cassandra.locator.LocalStrategy'}
        system_traces |           True | {'class': 'org.apache.cassandra.locator.SimpleStrategy', 'replication_factor': '2'}

(7 rows)

[cqlsh 5.0.1 | Cassandra 3.11.3 | CQL spec 3.4.4 | Native protocol v4]
cqlsh> SELECT * FROM system_schema.keyspaces;
 keyspace_name        | durable_writes | replication
----------------------+----------------+-------------------------------------------------------------------------------------
          system_auth |           True | {'class': 'org.apache.cassandra.locator.SimpleStrategy', 'replication_factor': '1'}
        system_schema |           True |                             {'class': 'org.apache.cassandra.locator.LocalStrategy'}
 svc_monitor_keyspace |           True | {'class': 'org.apache.cassandra.locator.SimpleStrategy', 'replication_factor': '3'}
      to_bgp_keyspace |           True | {'class': 'org.apache.cassandra.locator.SimpleStrategy', 'replication_factor': '3'}
   system_distributed |           True | {'class': 'org.apache.cassandra.locator.SimpleStrategy', 'replication_factor': '3'}
               system |           True |                             {'class': 'org.apache.cassandra.locator.LocalStrategy'}
       config_db_uuid |           True | {'class': 'org.apache.cassandra.locator.SimpleStrategy', 'replication_factor': '3'}
          dm_keyspace |           True | {'class': 'org.apache.cassandra.locator.SimpleStrategy', 'replication_factor': '3'}
        system_traces |           True | {'class': 'org.apache.cassandra.locator.SimpleStrategy', 'replication_factor': '2'}
            useragent |           True | {'class': 'org.apache.cassandra.locator.SimpleStrategy', 'replication_factor': '3'}

In the 5.1 branch this appears to be set in _cassandra_ensure_keyspace for one of the Cassandra instances based on the total number of nodes in a cluster:

https://github.com/Juniper/contrail-controller/blob/f24197a11e8401891ea1294feb28040da6f20f5a/src/config/common/vnc_cassandra.py#L606-L609

https://github.com/Juniper/contrail-controller/blob/f24197a11e8401891ea1294feb28040da6f20f5a/src/config/common/vnc_cassandra.py#L558-L560

https://github.com/Juniper/contrail-controller/blob/c86e31f2db95f3d4abcef9b5eab8d532e7bddd19/src/config/common/cfgm_common/db_json_exim.py#L43-L47

KEYSPACES = ['config_db_uuid',
            'useragent',
            'to_bgp_keyspace',
            'svc_monitor_keyspace',
            'dm_keyspace']

For contrail-analytics the replication factor is hard-coded to be set to 2 for just one keyspace (ContrailAnalyticsCql):

https://github.com/Juniper/contrail-analytics/blob/082493863e99b0656f1f7a589b37f94b45bcf33d/contrail-collector/viz.sandesh#L12 const string COLLECTOR_KEYSPACE_CQL = "ContrailAnalyticsCql"

https://github.com/Juniper/contrail-analytics/blob/3b23f1cde29893b7a147962def275284ebc36d54/contrail-collector/db_handler.cc#L97-L101

    if (cassandra_options.cluster_id_.empty()) {
        tablespace_ = g_viz_constants.COLLECTOR_KEYSPACE_CQL;
    } else {
        tablespace_ = g_viz_constants.COLLECTOR_KEYSPACE_CQL + '_' + cassandra_options.cluster_id_;
    }

https://github.com/Juniper/contrail-analytics/blob/3b23f1cde29893b7a147962def275284ebc36d54/contrail-collector/db_handler.cc#L513-L524 if (!dbif_->Db_AddSetTablespace(tablespace_, "2")) {

Juniper / contrail-charms

Cassandra is not configured to take rack placement into account during replication #97