I'm running a rabbitmq cluster on eks. (rabbitmq-cluster-operator + 3 servers)
most of the time, everything works fine.
every now and then, I see a message in unack, but in the pod i don't see that the message is being worked on...
it's stuck until I restart the pod, and then it works.
Where i see it the most is when I have about 50 pods (of the same service) consuming messages from the same queue, but ~10 of them are stuck.
I can see 50 consumers in the consumers column in the rabbitmq console (meaning they are connected), but in the pod no log that indicates they are working on the message.
after I'm restarting the pods, some of them starts to work fine, others need another restart.
I'm running a rabbitmq cluster on eks. (rabbitmq-cluster-operator + 3 servers) most of the time, everything works fine. every now and then, I see a message in unack, but in the pod i don't see that the message is being worked on... it's stuck until I restart the pod, and then it works.
Where i see it the most is when I have about 50 pods (of the same service) consuming messages from the same queue, but ~10 of them are stuck. I can see 50 consumers in the consumers column in the rabbitmq console (meaning they are connected), but in the pod no log that indicates they are working on the message. after I'm restarting the pods, some of them starts to work fine, others need another restart.
anyone encountered such a behaviour?