ajmalrahman
commented
2 years ago Tracked elsewhere
Closed ajmalrahman closed 2 years ago
ajmalrahman
commented
2 years ago Tracked elsewhere
lukebakken
commented
2 years ago
When RabbitMQ (3-node downstream cluster) is restarted, the federation status (on the admin page) shows no links - which were previously running. The policies and the upstream configurations are still present after the restart, but the links are missing.
This happens on Production, but can easily reproduce it also on the test environments when the services are restarted one by one. The links disappear, when the specific node as per the status (Node in Federation status) is restarted. In my case, that is ha-1.
If I delete the policy, and create it again - the link shows up again. This is quite frustrating - as if for some reason the node restarts - the federation breaks. And we have to create the policy again.
Steps to reproduce:
Created the policy to federate the exchange on the downstream server Created a federation upstream on the downstream server Checked the status of the federation - the link status was Running (on node ha-1) Restarted all 3 nodes in the 3-node Rabbitmq cluster Checked the status of the federation, while restarting (ha-1) it went to Starting and then got removed. Created the policy again (manually) - the federation link came up again.
RabbitMQ versions used: RabbitMQ 3.8.17 Erlang 23.3.4.2
Logs from the pods: Attached
rmq-ha-2.txt rmq-ha-0.txt rmq-ha-1.txt
I believe I have provided sufficient info, also could see similar posts from others with no resolution (missing info on how to reproduce) . It would be great if I could get some help on what's going wrong here and how I could fix or work around it.
For example, is it possible to detect missing links are automatically re-create the policies should such a situation arise?