search

jetstack / navigator

Managed Database-as-a-Service (DBaaS) on Kubernetes
Apache License 2.0
271 stars 31 forks source link

Potential data loss due to CASSANDRA_AUTO_BOOTSTRAP=false #224

Open wallrj opened 6 years ago

wallrj commented 6 years ago

We should investigate, test and comment on https://github.com/kubernetes/examples/issues/175

Setting auto_bootstrap to false when bootstrapping a new node exposes the cluster to huge inconsistencies. This is because all the other steps in the process are carried out but no data is streamed to the node. Hence, the node would be in the UN state without having any data for the token ranges it has been allocated! Furthermore, the new node without data will be serving reads and nodes that previously owned the tokens will no longer be serving reads. Effectively, the token ranges for that replica would be replaced with no data.

http://thelastpickle.com/blog/2017/05/23/auto-bootstrapping-part1.html

If it is an issue we need to handle this safely in Navigator.

/kind bug

wallrj commented 6 years ago

Note that in #222 we've switched to different image which does not allow the bootstrap mode to be configured from an environment variable. We need to investigate what the default setting is in the Docker maintained images and how to set it....assuming it's necessary to set it.