Introduce a few new rabbitmq-upgrade commands

michaelklishin commented 4 years ago

Proposed Changes

See #408 for the background.

Types of Changes

[ ] Bug fix (non-breaking change)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause an observable behavior change in existing systems)
[ ] Documentation improvements (corrections, new content, etc)
[ ] Cosmetic change (whitespace, formatting, etc)

Checklist

[x] I have read the CONTRIBUTING.md document
[x] I have signed the CA (see https://cla.pivotal.io/sign/rabbitmq)
[x] All tests pass locally with my changes
[x] I have added tests that prove my fix is effective or that my feature works
[ ] I have added necessary documentation (if appropriate)
[x] Any dependent changes have been merged and published in related repositories

Further Comments

To QA this, start a three node cluster and declare a quorum queue in it. It will default to three replicas.

Then

Run ./sbin/rabbitmq-upgrade await_online_quorum_plus_one -t 30 --formatter=json which will succeed
./sbin/rabbitmq-upgrade await_online_quorum_plus_one -t 30 which will succeed
Stop node two
./sbin/rabbitmq-upgrade await_online_quorum_plus_one -t 10 --formatter=json which will fail
./sbin/rabbitmq-upgrade await_online_quorum_plus_one -t 10 which will fail
Run * ./sbin/rabbitmq-upgrade await_online_quorum_plus_one -t 60 and start node two again in the background, the command will succeed after a period of waiting

michaelklishin commented 4 years ago

Since this is now a starting point of https://github.com/rabbitmq/rabbitmq-server/issues/2321 which has a lot of eye balls on it and will be extensively QA'ed by at least two deployment automation tools/teams, let's merge this early and continue.

michaelklishin commented 4 years ago

Backported to v3.8.x.

gerhard commented 4 years ago

I was just reviewing this code while updating the release notes for 3.8.4-beta.1 and noticed that the default timeout for both commands is set to 120 milliseconds. Was it mean to be 120 seconds instead?

rabbitmq-upgrade await_online_quorum_plus_one
Will wait for a quorum + 1 of nodes to be online for all quorum queues for 0 seconds...

rabbitmq-upgrade await_online_quorum_plus_one -t 120
Will wait for a quorum + 1 of nodes to be online for all quorum queues for 120 seconds...

gerhard commented 4 years ago

I also noticed this:

rabbitmq-upgrade await_online_synchronized_mirror
Will wait for a synchronised mirror be online for all classic mirrored queues for 0 seconds...
Error:
{:undef, [{:rabbit_upgrade_preparation, :await_online_synchronised_mirror, 'x', []}]}

This is in 3.8.4-alpha.28

gerhard commented 4 years ago

FYI https://github.com/rabbitmq/rabbitmq-website/commit/d103ce668a8752670ef4d7b99fd1e7a4dc430760

michaelklishin commented 4 years ago

Good catch. There was a typo in a function name and a few commands had the same default timeout problem. Addressed in #418.

michaelklishin commented 4 years ago

Unfortunately 3.8.4-rc.1 has already shipped. We will have to produce a 3.8.4-rc.2 before we can announce an RC.

rabbitmq / rabbitmq-cli