search

jetstack / navigator

Managed Database-as-a-Service (DBaaS) on Kubernetes
Apache License 2.0
271 stars 31 forks source link

Use ordered pod management #329

Closed wallrj closed 6 years ago

wallrj commented 6 years ago

Our ScaleOut action already adds pods one-at-a-time, but using the parallel management policy in the statefulset was definitely wrong.

Release note:

NONE
munnerz commented 6 years ago

How come we can't use a parallel strategy? Does cassandra recommend nodes be brought online one at a time?

In some cases, could waiting until each node is Ready before proceeding cause a cluster to not start due to a lack of quorum, causing the first pods to never become Ready?

wallrj commented 6 years ago

How come we can't use a parallel strategy? Does cassandra recommend nodes be brought online one at a time?

There are mixed messages about this. The official docs don't really mention it:

But elsewhere it is implied that adding nodes, one-at-a-time is important when using the auto_bootstrap=true configuration option. E.g. this question and the answer from https://stackoverflow.com/users/1054558/aaron (top rated for Cassandra https://stackoverflow.com/tags/cassandra/topusers): https://stackoverflow.com/questions/37283424/best-way-to-add-multiple-nodes-to-existing-cassandra-cluster

In some cases, could waiting until each node is Ready before proceeding cause a cluster to not start due to a lack of quorum, causing the first pods to never become Ready?

I don't think so. Adding nodes in parallel might cause problems for clients reads and writes where they require quorum consistency and data has not yet replicated to the new nodes.

I'll add some notes and links to the docs.

wallrj commented 6 years ago

Added a note to the docs.

wallrj commented 6 years ago

/retest

munnerz commented 6 years ago

/lgtm /approve

jetstack-bot commented 6 years ago

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: munnerz

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files: - ~~[OWNERS](https://github.com/jetstack/navigator/blob/master/OWNERS)~~ [munnerz] Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment