zalando-stups / planb-cassandra

Plan B Cassandra for STUPS/AWS with static IPs
Other
27 stars 18 forks source link

WIP: add extend command to deploy side "virtual data center" #164

Closed a1exsh closed 6 years ago

a1exsh commented 7 years ago

The command takes a region and cluster name of an existing cluster to "extend". You need to specify a dc suffix for new "dc" as well.

a1exsh commented 7 years ago

Before anything else: make sure you have a superuser account other than the default "cassandra", otherwise then it comes to altering system_auth keyspace expect trouble. See #163 for details.

Usage example:

./planb.py extend --cluster-name acid-cassandra-test --region eu-central-1 --dc-suffix _xxx --no-termination-protection

This should deploy 3 more nodes within eu-central_xxx "DC":

$ nodetool status
Datacenter: eu-central
======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens       Owns (effective)  Host ID                               Rack
UN  172.31.128.14  79.66 KB   256          100.0%            2c603634-89b4-4399-926d-3585ccd54b61  1a
UN  172.31.144.13  91.26 KB   256          100.0%            319d755b-1968-4a4d-b41f-710d4e11fcd8  1b
UN  172.31.160.11  74.63 KB   256          100.0%            1828cb3c-280d-4a2b-8dab-85461b59f437  1c
Datacenter: eu-central_xxx
==========================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens       Owns (effective)  Host ID                               Rack
UN  172.31.128.15  100.8 KB   256          100.0%            1f12c879-3176-4236-b233-12c1f6610c19  1a
UN  172.31.144.14  92.82 KB   256          100.0%            05fa3ca7-2b6a-4daf-9396-299220be625d  1b
UN  172.31.160.12  89.84 KB   256          100.0%            6d61caa9-fe1b-47d1-ac0f-1c2d0b9c0652  1c

After nodes are deployed, issue nodetool rebuild eu-central on each of the new nodes to bootstrap them. You will need to alter replication strategy for system_traces/system_distributed keyspaces, etc. to use NetworkTopologyStrategy: otherwise rebuild command will fail.

a1exsh commented 7 years ago

With the latest commit now it's possible to extend cross-datacenter. Precondition: the existing ring needs to be deployed in DMZ with public IPs.

./planb.py extend \
    --from-region eu-central-1 \
    --to-region eu-west-1 \
    --use-dmz \
    --cluster-name my-cassandra-test \
    --dc-suffix _xxx \
    --ring-size 2 \
    --hosted-zone my.zone.com. \
    --no-termination-protection
Datacenter: eu-central_xxx
==========================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens       Owns (effective)  Host ID                               Rack
UN  xx.xx.25.183   326.97 KB  256          100.0%            b6a03c12-92e6-4d98-9be7-1ea1d3a82238  1a
Datacenter: eu-west_xxx
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens       Owns (effective)  Host ID                               Rack
UN  xx.xx.174.68   104.12 KB  256          100.0%            6f524c96-433a-4f66-8439-298298a544d4  1b
UN  xx.xx.6.156    116.52 KB  256          100.0%            66e595f4-84bf-4460-a45f-139c2a15fe0a  1a
slitsche commented 6 years ago

👍

a1exsh commented 6 years ago

:+1: