Remove the loadbalanced CQL service and document an alternative way to connect.

[wallrj]

In no particular order:

• Removed the CQL service.
• Added a headless nodes service.
• Renamed the seedprovider service to seeds.
• Added documentation explaining the two headless services and how to connect a CQL client.
• Refactored the two headless services into a single service control
• Removed the service update code, which was not tested and which added unecessary complication.
• Navigator will now only touch the services if they are missing.
• Updated the E2E tests so that cql_connect always attempts to connect to the nodes service.
• Removed the ServiceName from the NodePool statefulsets because it no longer made sense with multiple NodePools / StatefulSets. The SS servicename is supposed to match a service dedicated to that statefulset, not a single seedprovider service. We probably should dynamically create a service for each nodepool.
• Removed the, also broken, CASSANDRA_SEEDS configuration which was pointing seedProviderServiceName rather than at a service name matching the name of the statefulset.
• In E2E tests Reverted to a better mechanism for simulating node failure. Decommission leaves the node in a decommissioned state causing the C* process to immediately exit.
• Added CONSISTENCY checks to the E2E CQL queries to verify that both C* nodes are reachable and have the test data.

Fixes: #232

Release note:

NONE

[munnerz]

Removed the, also broken, CASSANDRA_SEEDS configuration which was pointing seedProviderServiceName rather than at a service name matching the name of the statefulset.

Perhaps I misunderstand this bullet - when a node bootstraps, surely it should (attempt to) connect to all seeds in the cluster, not just its own node pool, in order to start?

[wallrj]

In https://github.com/jetstack/navigator/pull/278#issuecomment-372641959 @munnerz wrote:

Perhaps I misunderstand this bullet - when a node bootstraps, surely it should (attempt to) connect to all seeds in the cluster, not just its own node pool, in order to start?

There are two things:

• As far as we can see, this environment variable is obsolete, now that we're using the Kubernetes seed provider. The CASSANDRA_SEEDS env variable results in a Cassandra configuration stanza which is specifically for the SimpleSeedProvider.
• The StatefulSet.Spec.ServiceName had previously been set to a single, cluster wide service name. So when there are multiple node pools and pods from multiple statefulsets, the DNS names no longer reflect the controlling statefulset...instead they'd be something like euwest1a-0.cass-seeds.my-namespace.srv.cluster.local which works (somehow)....but
  • The domain name label cass-seeds is no longer ideal, because there will only be DNS names for pods that have been labelled as seeds.
  • Instead we want stable DNS names for all the pods in all statefulsets, not just for the seed pods.

I think we should follow this up with a branch that enhances CreateNodePool action so that it also creates a headless service per nodepool....and that dynamically created service can be the name we use in the statefulset.spec.serviceName

[jetstack-ci-bot]

@wallrj PR needs rebase

[jetstack-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kragniz

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files: - ~~[OWNERS](https://github.com/jetstack/navigator/blob/master/OWNERS)~~ [kragniz] You can indicate your approval by writing `/approve` in a comment You can cancel your approval by writing `/approve cancel` in a comment

[jetstack-ci-bot]

/test all [submit-queue is verifying that this PR is safe to merge]

[jetstack-ci-bot]

Automatic merge from submit-queue.