Bootstrap and update a Cassandra cluster on STUPS_/AWS.
Planb deploys Cassandra by means of individual EC2 instances running Taupage_ & Docker with the latest Cassandra version 3.0.x (default; the new 'tick-tock' releases 3.x and older 2.x versions are still available).
Features:
EC2 Auto Recovery_ enabledNon-Features:
STUPS Cassandra_ if you need a dynamic Cassandra cluster setupsudo pip3 install -r requirements.txt)keytool in your PATH (required to generate SSL certificates)You have created a dedicated AWS IAM user for auto-recovery. The policy document for this user should look like the following::
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "ec2:DescribeInstanceRecoveryAttribute", "ec2:RecoverInstances", "ec2:DescribeInstanceStatus", "ec2:DescribeInstances", "cloudwatch:PutMetricAlarm" ], "Resource": [ "*" ] } ] }
You have a planb_autorecovery section in your AWS credentials file
(~/.aws/credentials) with the access key of the auto-recovery user::
[planb_autorecovery] aws_access_key_id = THEKEYID aws_secret_access_key = THESECRETKEY
These credentials are only used to create the auto-recovery alarm. When triggered by the failing system status check, the recovery action is performed by this dedicated user.
.. note::
The access keys for the auto-recovery user can be rotated or made inactive at any time, without impacting its ability to perform the recovery action. The user still needs to be there, however.
To create a cluster named "mycluster" in two regions with 3 nodes per region (the default size, enough for testing):
.. code-block:: bash
$ zaws login # get temporary AWS credentials
$ ./planb.py create --cluster-name mycluster --use-dmz eu-west-1 eu-central-1The above example requires Elastic IPs to be allocated in every region (this might require to increase the AWS limits for Elastic IPs).
To create a cluster in a single region, using private IPs only, see the following example:
.. code-block:: bash
$ ./planb.py create --cluster-name mycluster eu-central-1It is possible to use Public IPs even with a single region, for example, if your application(s) connect from different VPC(s). This is currently not recommended, though, as there is no provision for client-to-server encryption.
Available options are:
=========================== ============================================================================ --cluster-name Not actually an option, you must specify the name of a cluster to create --cluster-size Number of nodes to create per AWS region. Default: 3 --dc-suffix Optional "DC suffix". --num-tokens Number of virtual nodes per node. Default: 256 --instance-type AWS EC2 instance type to use for the nodes. Default: t2.medium --volume-type Type of EBS data volume to create for every node. Default: gp2 (General Purpose SSD). --volume-size Size of EBS data volume in GB for every node. Default: 16 --volume-iops Number of provisioned IOPS for the volumes, used only for volume type of io1. Default: 100 (when applicable). --no-termination-protection Don't protect EC2 instances from accidental termination. Useful for testing and development. --use-dmz Deploy the cluster into DMZ subnets using Public IPs (required for multi-region setup). --hosted-zone Specify this to create SRV records for every region, listing all nodes' private IP addresses in that region. This is optional. --scalyr-key API Key for writing logs to Scalyr (optional). --scalyr-region Scalyr account region, such as 'eu' (optional). --artifact-name Override Pierone artifact name. Default: planb-cassandra-3.0 --docker-image Override default Docker image. --environment, -e Extend/override environment section of Taupage user data. --sns-topic Amazon SNS topic name to use for notifications about Auto-Recovery. --sns-email Email address to subscribe to Amazon SNS notification topic. See below for details. =========================== ============================================================================
In order to be able to receive notification emails in case instance
recovery is triggered, provide either SNS topic name in
--sns-topic, or email to subscribe in --sns-email (or both).
If only the email address is specified, then SNS topic name defaults
to planb-cassandra-system-event. An SNS topic will be created (if
it doesn't exist) in each of the specified regions. If email is
specified, then it will be subscribed to the topic.
If you use the Hosted Zone parameter, a full name specification is
required e.g.: --hosted-zone myzone.example.com. (note the
trailing dot.)
After the create command finishes successfully, follow the on-screen instructions to create the admin superuser, set replication factors for system_auth keyspace and then create your application user and the data keyspace.
The generated administrator password is available inside the docker
container in an environment variable ADMIN_PASSWORD.
The list of private IP contact points for the application can be obtained with the following snippet:
.. code-block:: bash
$ aws ec2 describe-instances --region $REGION --filter 'Name=tag:Name,Values=planb-cassandra' | grep PrivateIp | sed s/[^0-9.]//g | sort -u.. important::
The Jolokia port 8778 should be accessible from the Odd host. Ensure the ingress rule for your clusters security group allows connections from the Odd host.
To update the Docker image or AMI you should ensure that you are logged in to
your account and have SSH access to your Odd host. The following commands will
allow you to update the Docker image on all nodes of the cluster mycluster.
If an action is interrupted the next call will resume with the last action on
the last used node.
.. code-block:: bash
$ zaws re $ACCOUNT # for longer updates run `zaws login -r` in background
$ piu re -O $ODDHOST $ODDHOST # for longer updates add `-t 180` or bigger
$ ./planb.py update \
--region eu-central-1 \
--odd-host $ODDHOST \
--cluster-name mycluster \
--docker-image registry.opensource.zalan.do/stups/planb-cassandra-3.0:cd-69 \
--sns-topic planb-cassandra-system-event \
--sns-email test@example.comAvailable options for update:
=================== ========================================================
--region The region where the update should be applied (required)
--odd-host The Odd host in the region of your VPC (required)
--cluster-name The name of your cluster (required)
--filters Additional AWS resource filters (in JSON format)
--force-termination Disable termination protection for the duration of update
--no-prompt Don't prompt before updating every node.
--docker-image The full specified name of the Docker image
--taupage-ami-id The full specified name of the AMI
--instance-type The type of instance to deploy each node on (e.g. t2.medium)
--scalyr-key API Key for writing logs to Scalyr (optional).
--scalyr-region Scalyr account region, such as 'eu' (optional).
--environment, -e Extend/override environment section of Taupage user data.
--sns-topic Amazon SNS topic name to use for notifications about Auto-Recovery.
--sns-email Email address to subscribe to Amazon SNS notification topic. See description of create subcommand above for details.
=================== ========================================================
The cluster name parameter is used to list all EC2 instances in the region
with the matching Name tag. This parameter may contain wildcards (*).
For example, if you have multiple virtual data centers in a cluster, this
allows to update all nodes of all DCs by running only one command.
Any additional resource filters supported by AWS may be provided (only JSON
format is supported, though). For example, to limit the update operation to a
specific Availability Zone, add the following parameter: --filters '[{"Name":"availability-zone","Values":["eu-central-1c"]}]'.
By default, update is an interactive command which operates on one node at a time.
It will prompt before starting update of each node. It starts by draining the
target node and then terminates the EC2 instance that is running it. Then a new
EC2 instance is created with the same private and public IP addresses (if any),
and potentially different configuration as specified by the options. The new
instance is expected to attach the EBS volume that was previously utilized by the
node. This keeps all the node's data and identification within the cluster intact.
The command will wait for the replacement node to be back UP. You should still monitor the status of the cluster to verify that all other nodes also see the new node as UP before proceeding.
If you're confident enough in using this command, you may opt in for "fire and
forget" behavior, by specifying the --no-prompt flag.
While performing the update, which destroys the running EC2 instance and creates a blank one, the command keeps the current state in the tags of the EBS data volume.
If interrupted by some unexpected problems, the command resumes the update sequence by using the information in the EBS volume tags. This relies however on an assumption that the command is ran again with essentially the same parameters on the same machine, since some of the state is stored in a temporary file, named after the EBS volume id.
If the command enters failed state, as a safety precaution it will not try to proceed
further, even if started again. The operator is then responsible for analysing the
failure reason and removing the failed state tag from the related EBS volume before
starting the command again. One common source of failed state is forgetting to use
--force-termination flag on a cluster which was deployed with termination protection
enabled.
No provisions are made by the command to detect if a concurrent update operation is in progress for a given cluster. It makes sense to ensure that only one operator is using the command as part of routine maintenance at any given time.
There are a number of scenarios requiring to extend an existing cluster. The possible use-cases are::
Available options for extend:
============================== ============================================================================
--from-region Name of AWS region where a cluster is already running.
--to-region Name of AWS region where a new data center should be created. This can be the same as "from region", in this case a virtual data center is created.
--cluster-name The name of a cluster to extend.
--ring-size Number of nodes to create in the new data center.
--dc-suffix Optional "DC suffix". When creating a virtual data center be sure to specify a new suffix for each virtual data center you create!
--num-tokens Number of virtual nodes per node. Default: 256
--allocate-tokens-for-keyspace Use new token allocation algorithm, available starting with version 3.0.
--instance-type AWS EC2 instance type to use for the nodes. Default: t2.medium
--volume-type Type of EBS data volume to create for every node. Default: gp2 (General Purpose SSD).
--volume-size Size of EBS data volume in GB for every node. Default: 16
--volume-iops Number of provisioned IOPS for the volumes, used only for volume type of io1. Default: 100 (when applicable).
--no-termination-protection Don't protect EC2 instances from accidental termination. Useful for testing and development.
--use-dmz Deploy the new data center into DMZ subnets using Public IPs (required for multi-region setup).
--hosted-zone Specify this to create the SRV record for the new data center. This is optional.
--artifact-name Override Pierone artifact name. Default: planb-cassandra-3.0
--docker-image Override default Docker image.
--environment, -e Extend/override environment section of Taupage user data.
--sns-topic Amazon SNS topic name to use for notifications about Auto-Recovery.
--sns-email Email address to subscribe to Amazon SNS notification topic. See description of create subcommand above for details.
============================== ============================================================================
To add a new virtual data center in the same region where your existing cluster is running run the extend command like this:
.. code-block:: bash
$ planb.py extend \
--from-region eu-central-1 \
--to-region eu-central-1 \
--cluster-name mycluster \
--ring-size 3 \
--dc-suffix _new \
--hosted-zone myzone.example.com... important::
In this mode the new nodes are created with auto_bootstrap: false.
When creating a new virtual data center in the same region, you must
specify the DC suffix which doesn't exist in the region yet! Otherwise you
risk adding a number of empty nodes to the cluster, which will be serving
read requests and your client applications will suffer from apparent data
loss.
After the command has run successfully, you need to login to each of the nodes
in the new data center and run nodetool rebuild $existing_dc_name.
On version 3.0 or later it is possible to request use of the new token allocation algorithm. For that, start by including the to-be-deployed virtual DC in the replication settings of the data keyspace, by running a CQL statement like the following one on one of the existing cluster nodes:
.. code-block::
cqlsh> ALTER KEYSPACE mydata WITH replication = { 'class': 'NetworkTopologyStrategy', 'eu-central': 3, 'eu-central_new': 3 };
Then run the extend command, specifying the
--allocate-tokens-for-keyspace=mydata as one of the options.
With the new token allocation algorithm it makes sense to use a much smaller
number of tokens than the default 256. E.g. 16 tokens are generally enough to
achieve balanced ownership distribution. Use the --num-tokens option to
set the desired number of tokens per node.
.. important::
In order for the token allocation algorithm to be actually used, the
auto_bootstrap parameter has to be set to true. This is done
automatically by the deployment script. Due to this, before you can run
nodetool rebuild command on the nodes of the newly deployed ring, you
have to run manually the following CQL command on every new node:
TRUNCATE system.available_ranges.
To extend a cluster to a new AWS region, run the command like this:
.. code-block:: bash
$ planb.py extend \
--from-region eu-central-1 \
--to-region eu-west-1 \
--cluster-name mycluster \
--ring-size 3 \
--use-dmz \
--hosted-zone myzone.example.com.The DC suffix is optional in this case, unless you already have a cluster with this name in the target region. You must specify the DMZ option, and the existing cluster must already be running in the DMZ: otherwise the new and existing nodes will not be able to communicate with each other.
This is currently unsupported, due to the use of auto_bootstrap: false when
creating new nodes. In general, it should be possible to override this option
and add the nodes one by one to the existing data center, but care should be
taken while doing so.
There is a command group called remote that allows you to run arbitrary
shell commands on all nodes of a given Cassandra cluster. This can be useful
when applying a configuration change, e.g. setting compaction throughput:
.. code-block:: bash
$ planb.py remote \
--region eu-west-1 \
--odd-host $ODDHOST \
--cluster-name mycluster \
--piu "setting cassandra compaction throughput" \
nodetool \
-- \
setcompactionthroughput 50The following options are available for the remote command:
============== ==================================================
--region AWS region.
--odd-host Odd host name for the first SSH hop.
--cluster-name The name of the cluster (Name tag on the EC2 instances).
--filters Additional AWS resource filters (in JSON format)
--piu Run piu first with this parameter as reason.
--echo Print the command before running it.
--no-prompt Don't prompt before running the command.
--no-wait Don't wait for the command to exit.
--ip-label Label all output from the node with its IP address.
--help Show this message and exit.
============== ==================================================
There are 3 subcommands in the remote command group:
======== ============================== shell Run an arbitrary shell command. nodetool Run a nodetool command. cqlsh Run an administrative CQL shell command. ======== ==============================
The most basic is shell which allows to run any command on the server.
Two shorthand commands for running nodetool and cqlsh -u admin -p $ADMIN_PASSWORD are also provided.
When configuring your client application to talk to a Cassandra cluster deployed in AWS using Public IPs, be sure to enable address translation using EC2MultiRegionAddressTranslator_. Not only it saves costs when communicating within single AWS region, it also prevents availability problems when security group for your Cassandra is not configured to allow client access on Public IPs (via the region's NAT instances addresses).
Even if your client connects to the ring using Private IPs, the list of peers it gets from the first Cassandra node to be contacted only consists of Public IPs in such setup. Should that node go down at a later time, the client has no chance of reconnecting to a different node if the client traffic on Public IPs is not allowed. For the same reason the client won't be able to distribute load efficiently, as it will have to choose the same coordinator node for every request it sends (namely, the one it has first contacted via the Private IP).
To watch the cluster's node status (e.g. joining during initial bootstrap):
.. code-block:: bash
$ # on Taupage instance
$ watch docker exec -it taupageapp nodetool statusThe output should look something like this (freshly bootstrapped cluster):
::
Datacenter: eu-central
======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 52.29.137.93 66.59 KB 256 34.8% 62f50c2c-cb0f-4f62-a518-aa7b1fd04377 1a
UN 52.28.11.187 66.43 KB 256 31.1% 69d698a9-7357-46b2-93b8-6c038155f0c1 1b
UN 52.29.41.128 71.79 KB 256 35.0% b76e7ed7-78de-4bbc-9742-13adbbcfd438 1a
Datacenter: eu-west
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 52.49.209.129 91.29 KB 256 34.8% 140bc7de-9973-46fd-af8c-68148bf20524 1b
UN 52.49.192.149 81.16 KB 256 32.1% cb45fc4c-291d-4b2b-b50f-3a11048f0211 1c
UN 52.49.128.58 81.22 KB 256 32.1% 8a270de3-b419-4baf-8449-f4bc65c51d0d 1aThe following manual process may be applied whenever there is a need to scale up EC2 instances or update Taupage AMI.
For every node in the cluster, one by one:
nodetool drain; nodetool stopdaemon).erase_on_boot: false flag under mounts: /var/lib/cassandra. See documentation of Taupage_ for detailed description and syntax example. The docker image version being used can also be updated in this section, however, it is recommended to avoid changing multiple things at a time. Also, docker image can be updated without terminating the instance, by stopping and starting it with updated 'User Data' instead./dev/sdf as the device name.docker restart taupageapp.nodetool repair (optional: if the node was down for less than max_hint_window_in_ms, which is by default 3 hours, hinted hand off should take care of streaming the changes from alive nodes).nodetool status.Proceed with other nodes as long as the current one is back and everything looks OK from nodetool and application points of view.
It is possible to manually scale out already deployed cluster by following these steps:
system_auth keyspace (if needed)in every region affected. Don't set RFs to be more than 5 per region or virtual DC.
For example, if you run in two regions and want to scale to 5 nodes per region, issue the following CQL command on any of the nodes:
ALTER KEYSPACE system_auth WITH replication = {'class': 'NetworkTopologyStrategy', 'eu-central': 5, 'eu-west': 5};
nodes in every region, then update security groups in every region to include all newly allocated Elastic IP addresses.
For example, if scaling from 3 to 5 nodes in two regions you will need 2 new IP addresses in every region and both security groups need to be updated to include a total of 4 new addresses.
other EC2 instance in the VPC. You will need it on further steps.
have the same settings as for the rest of the cluster). EBS encryption is not recommended as it might prevent auto-recovery.
Name tag on the volume in the format:<cluster-name>-<private-ip>.
the tag name should be Taupage:erase-on-boot and the value True.
of the running nodes.
internal-...vs. dmz-... for public IPs setup. The subnet need to match your
private IP, which should also be assigned manually on the same page.
Public IP' is set to 'Disable'.
AUTO_BOOTSTRAP environment variableis set to true or not present. Update the referenced EBS volume to:
<cluster-name>-<private-ip>
associate one of the pre-allocated Elastic IP addresses with it.
Caution! For multi-region setup the nodes are started in DMZ subnet and thus don't have internet traffic before you give them a public IP. Be sure to do this before anything else, or the new node won't be able to ship its logs and you won't be able to ssh into it (restarting the node should help if it was too late).
nodetool status totrack its progress in joining the ring.
auto-recovery alarm for the new instance.
Check '[x] Take the action: [*] Recover this instance' and leave the rest of parameters at their default values. It is also recommended to set up a notification SNS topic for actual recovery events.
Only when the new node has fully joined, proceed to add more nodes.
After all new nodes have joined, issue nodetool cleanup command on
every node in order to free up the space that is still occupied by the
data that the node is no longer responsible for.
.. _STUPS: https://stups.io/ .. _Odd: http://docs.stups.io/en/latest/components/odd.html .. _Taupage: http://docs.stups.io/en/latest/components/taupage.html .. _Ec2MultiRegionSnitch: http://docs.datastax.com/en/cassandra/2.1/cassandra/architecture/architectureSnitchEC2MultiRegion_c.html .. _EC2MultiRegionAddressTranslator: https://datastax.github.io/java-driver/manual/address_resolution/#ec2-multi-region .. _EC2 Auto Recovery: https://aws.amazon.com/blogs/aws/new-auto-recovery-for-amazon-ec2/ .. _Jolokia: https://jolokia.org/ .. _STUPS Cassandra: https://github.com/zalando/stups-cassandra .. _Più: http://docs.stups.io/en/latest/components/piu.html
In order to upgrade your Cluster you should run the following steps. You should have in mind that this process is a rolling update, which means applying the changes for each node in your cluster one by one. After upgrading the last node in your cluster you are done.
Disclaimer: Before you actually start, you should:
curl https://registry.opensource.zalan.do/teams/stups/artifacts/planb-cassandra-3.0/tags | jq '.[-1].name'nodetool upgradesstables and nodetool drain. The latter command will flush the memtables and speed up the upgrade process later on. This command is mandatory and cannot be skipped.
Excerpt from the manual Cassandra stops listening for connections from the client and other nodes. You need to restart Cassandra after running nodetool drain.docker rm -f taupageappIf you are running cassandra with the old folder structure where the data is directly located in mounts/var/lib/cassandra/ do the following. If not go on with step 6.
**Before Move**/mounts/var/lib/cassandra$ ls commit_logs keyspace_1 saved_caches system_auth system_traces
After Move
/mounts/var/lib/cassandra$ ls -la total 28 drwxrwxrwx 4 application application 4096 Oct 10 12:21 . drwxr-xr-x 3 root root 4096 Aug 25 13:27 .. drwxrwxr-x 5 application mpickhan 4096 Oct 10 12:21 data
/mounts/var/lib/cassandra$ ls -la data/ total 36 drwxrwxr-x 5 application mpickhan 4096 Oct 10 12:21 . drwxrwxrwx 4 application application 4096 Oct 10 12:21 .. drwxr-xr-x 2 application root 20480 Oct 10 12:15 commitlog drwxrwxr-x 9 application mpickhan 4096 Oct 10 12:19 data drwxr-xr-x 2 application root 4096 Oct 10 10:52 saved_caches
/mounts/var/lib/cassandra$ ls -la data/data/ total 36 drwxrwxr-x 9 application mpickhan 4096 Oct 10 12:19 . drwxrwxr-x 5 application mpickhan 4096 Oct 10 12:21 .. drwxr-xr-x 10 application root 4096 Aug 25 14:29 keyspace_1 drwxr-xr-x 19 application root 4096 Aug 25 13:27 system drwxr-xr-x 5 application root 4096 Aug 25 13:27 system_auth drwxr-xr-x 4 application root 4096 Aug 25 13:27 system_traces
Stop the ec2-Instance and change the user details Go to Actions -> Instance Settings -> View/Change User Details Change the "source" entry to the version you want to upgrade to:
Important: Use the stop command and not terminate.
Example:
From: "source: registry.opensource.zalan.do/stups/planb-cassandra:cd89"
To: "source: registry.opensource.zalan.do/stups/planb-cassandra-3.0:cd105"nodetool upgradesstables.
Check the logs for errors and warnings. (Note: For the size of ~12GB SSTables it takes approximately one hour to convert them to the new format.)