Automatic cluster detach and rebalancing on graceful instance shutdown

misterbisson commented 8 years ago

When planning to gracefully stop a Couchbase instance, it should mark itself for removal and cleanly detach from the cluster so as not to lose any data or risk data loss by reducing the cluster's data redundancy. This could hook to Containerbuddy's preStop event, which is run before the application is stopped, and waits for the hook to return before stopping the main application.

I haven't tested it, but I think the right command to trigger the node removal and rebalance:

couchbase-cli rebalance -c 127.0.0.1:8091 -u $COUCHBASE_USER -p $COUCHBASE_PASS --server-remove=${IP_PRIVATE}:8091

And when that is done, it should be safe to stop (and remove/delete) the container. The stop timeout value would need to be large enough to accommodate the rebalancing.

tgross commented 8 years ago

This would be an excellent example of a preStop handler.

misterbisson commented 8 years ago

Yes, that was exactly the use case I had in mind when the preStop handler was initially proposed. You're right to point out that people reading this might not be aware of that history, however, so I updated the OP to add that info.

Previous discussion elsewhere:

autopilotpattern / couchbase

Automatic cluster detach and rebalancing on graceful instance shutdown #14