Open achton opened 9 months ago
Related discussion: https://github.com/helm/charts/issues/13485
This may very well be addressed by the changes in #3586 which also have to do with clustering. They are included in Lagoon v2.17.0.
We've rolled it out to the cluster which has had this problem twice now.
Describe the bug
In our hosted Lagoon cluster (v2.16.0 on AKS 1.25.15), we are seeing issues with the
lagoon-core-broker
StatefulSet. From time to time, the pods error out with errors like these continuously. When I arrive at the scene, there is usually only one pod alive (lagoon-core-broker-0
), which is not running.I am able to recover from this state by exec'ing into the pod, stopping RabbitMQ and restarting it with the
force_boot
setting enabled:To Reproduce
I am unsure how to get the cluster into this state. RabbitMQ docs write this about shutdown:
So maybe this is due to pods being evicted during automated node shutdown and not starting in the correct order.
Expected behavior
I expected the broker pods to restart correctly when they are evicted or shutdown automatically for any reason.