search

mosquito / aio-pika

AMQP 0.9 client designed for asyncio and humans.
https://aio-pika.readthedocs.org/
Apache License 2.0
1.18k stars 186 forks source link

RobustConnection is not restored after "Server connection was stuck" #588

Open tsdaemon opened 8 months ago

tsdaemon commented 8 months ago

aio-pika==9.2.3 aiormq==6.7.7 RabbitMQ: 3.10.10, remote cluster

Related issues: https://github.com/mosquito/aio-pika/issues/577 https://github.com/mosquito/aio-pika/issues/563

After period of inactivity we observed an error in logs:

Server connection was stuck. No frames were received in 183 seconds.

This error was logged but it didn't stopped the process: it would help if consumer just fail, then K8s would restart it. But it was left hanging.

I was not able to reproduce it locally. We will be looking into options how to self-heal a connection on a callback, but any advice is appreciated.

dimbler commented 7 months ago

+1 aio-pika=9.3.1 aiormq=6.7.7 No reconnection attempt after Server connection <Connection: "amqp://******@******:5672/apsv?heartbeat=60" at 0x7f042d374040> was stuck. No frames were received in 3 seconds. but in brocker side is closing AMQP connection <0.7020.2515> (*******:62649 -> *******:5672, vhost: '******', user: 'test'): client unexpectedly closed TCP connection You can reproduce this issuer by blocking all connection between server and client for time more that heartbeat timeout.

guishake commented 7 months ago

I get this issue too, but it's not bound to the 9.3 version, I encountered it also with 9.0 and thought the 9.1.3 would help unfortunately not. Is there any progress on this ? It would help if we could get back the control over the connection to close it properly and re-open it for instance, would it be possible ?