Respect not ready addresses

johanneswuerbach commented 4 years ago

Proposed Changes

We are using tooling of top of Kubernetes which respects the pod disruption budget of a StatefulSets https://kubernetes.io/docs/tasks/run-application/configure-pdb/

The problem with PDBs and the current approach of rabbitmq peer discovery is that pods are required to be ready, before joining the cluster.

Ideally we would like to change pods to be only marked as ready, once they actually joined the cluster.

To achieve that we return also the IPs of not ready nodes and rely on rabbitmq to add them to the cluster once the pod is responding.

Instead of the current recommended command: ["rabbitmq-diagnostics", "status"], would the stage 4 check command: ["rabbitmq-diagnostics", "check_port_connectivity"] be sufficient to detect a pod not yet in a cluster?

Types of Changes

What types of changes does your code introduce to this project? Put an x in the boxes that apply

[x] Bug fix (non-breaking change which fixes issue #NNNN)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause an observable behavior change in existing systems)
[ ] Documentation improvements (corrections, new content, etc)
[ ] Cosmetic change (whitespace, formatting, etc)

Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask on the mailing list. We're here to help! This is simply a reminder of what we are going to look for before merging your code.

[x] I have read the CONTRIBUTING.md document
[x] I have signed the CA (see https://cla.pivotal.io/sign/rabbitmq)
[x] All tests pass locally with my changes
[x] I have added tests that prove my fix is effective or that my feature works
[ ] I have added necessary documentation (if appropriate)
[ ] Any dependent changes have been merged and published in related repositories

Further Comments

If this is a relatively large or complex change, kick off the discussion by explaining why you chose the solution you did and what alternatives you considered, etc.

pivotal-issuemaster commented 4 years ago

@johanneswuerbach Please sign the Contributor License Agreement!

Click here to manually synchronize the status of this Pull Request.

See the FAQ for frequently asked questions.

pivotal-issuemaster commented 4 years ago

@johanneswuerbach Thank you for signing the Contributor License Agreement!

johanneswuerbach commented 4 years ago

Scratch that, there was a misunderstanding how the initial discovery works. Sorry for the noise.

michaelklishin commented 4 years ago

@johanneswuerbach thank you for taking the time to contribute :) If you have any Kubernetes-specific suggestions, including the examples in this repository or Peer Discovery doc guide, your PRs and mailing list suggestions would be appreciated 👍

michaelklishin commented 4 years ago

As for your question about the readiness probe, it's your call. See Health Checks for a more detailed description of how our team approaches health checks. Specifically rabbitmq-diagnostics check_port_connectivity is a popular option.

rabbitmq / rabbitmq-peer-discovery-k8s