Recurrent RPC timeout on queue.cancel()

[odesenfans]

Hello,

I am experiencing a rather annoying issue when cleaning up a queue. I set up a queue to feed a websocket. Once the websocket closes, I sometimes experience a "ChannelInvalidStateError: Channel closed by RPC timeout" exception on await queue.cancel(consumer_tag). I have no idea why this occurs.

Setup

The code is open source: https://github.com/aleph-im/pyaleph/blob/64c97e2a47071e028d0e929f512b0a7439c94661/src/aleph/web/controllers/messages.py#L362

• I create a channel when starting the API process.
• I create the queue when opening the websocket. The queue is exclusive and auto-delete.
• I create a consumer with queue.consume() and then wait for events from the websocket.
• Once I receive a close event, I cancel the consumer and then delete the queue.

In terms of stats, my app has:

• 19 open connections: 3 for a background process, and 2 per API process (used to be 1, but I had to create a second one to change the heartbeat timeout).
• 27 channels: 3 for the background process, and 3 per API process.
• ~100 queues.

Note that these queues are not very active. There's only a message every few seconds.

Any idea?

[mosquito]

Is is possible to make traffic dump beetween python proccess and broker?

[odesenfans]

Sure, what's the easiest way to achieve that?

[mosquito]

• wireshark on local machine
• tcpdump -pni any port 5672 on remote through the ssh

[odesenfans]

I started a tcpdump process on that server, I'll send you the dump once the error occurs again.

[odesenfans]

Okay, I logged all traffic to rabbitmq with tcpdump. I'm curious to understand how you make use of that, it's rather verbose! The "channel closed by RPC timeout" error occurred at 18:47:09 UTC (20:47:09 CET). I cut the logs from 20:40 to 20:50. Let me know if this is what you need.

rabbitmq-log.txt

[odesenfans]

Did you have the chance to take a look?

[Aurvandill]

Hello @odesenfans

I think the logs are no good for what @mosquito would need. He needs the packet content from some specific packages. The log you provided just contains what connections were made and not the content. I think @mosquito could maybe specify what packets he needs exactly.