Discover Cassandra nodes via Kubernetes DNS during replication

CatalystCode / project-fortis

Repository for all parts of the Fortis architecture

https://aka.ms/fortis-story

MIT License

29 stars 17 forks source link

Discover Cassandra nodes via Kubernetes DNS during replication #33

Open c-w opened 6 years ago

c-w commented 6 years ago

Currently, the helm chart that sets up Cassandra has nodes discovering each other by IP during data replication. In rare situations we've seen this cause issues if Cassandra and Kubernetes get out of sync as to what are the correct IP addresses for the Cassandra workers. To prevent this issue and shield Cassandra from IP changes, we want to front each Cassandra node with a DNS name of a kubernetes service. @xtophs is the expert in this area for further questions.

c-w commented 6 years ago

Error screenshots from yet another outage caused by this:

error1 error2 error3

c-w commented 6 years ago

Tried to fix the issue with kubectl -n cassandra exec -it cassandra-cluster-cassan-0 -- nodetool removenode 7d0841ec-09fc-4c5c-b67b-412d0d8a0afb and kubectl -n cassandra exec -it cassandra-cluster-cassan-0 -- nodetool removenode 860d09c0-1571-4771-9574-6d537fee57c1.