RabbitMQ timeout errors

[jdavid]

I've found these errors in the /var/log/rabbitmq/rabbit@raspberrypi.log file:

=ERROR REPORT==== 24-Feb-2019::06:25:28 ===
closing AMQP connection <0.848.0> ([::1]:59846 -> [::1]:5672):
missed heartbeats from client, timeout: 60s

=ERROR REPORT==== 28-Feb-2019::18:56:53 ===
closing AMQP connection <0.801.0> ([::1]:49532 -> [::1]:5672):
{writer,send_failed,{error,timeout}}

I've reproduced the first one locally, introducing a time.sleep().

[jdavid]

So, there were 2 issues.

missed heartbeats from client, timeout: 60s

Clients connections to RabbitMQ timeout after 60s of inactivity, a feature known as heartbeat allows to keep alive the connection.

The wsn_data_django.py program makes a POST request to the Django server. When the quota of the 4G network was consumed the 4G network slowed down, making the POST request to take more than 60s. This produced the timeout error, RabbitMQ closing the connection to the client (the wsn_data_django.py program).

The solution to this has been:

• Use sessions (from the requests library) to speed up requests, by persisting the TCP connection from the Pi to the server.
• Make the request to timeout: 5s to establish the connection, and 30s of read timeout (30s max between bytes received). By default the request is blocking.

{writer,send_failed,{error,timeout}}

The 4G network speed was restored, but the first error has produced the messages to pile up in the queue, producing a new error.

RabbitMQ pushing messages faster to wsn_data_django.py than this was able to deliver, RabbitMQ eventually fails to send a new message and closes the connection to the client.

Solution:

• Set the QOS prefetch count to 20 (by default unlimited). This means RabbitMQ will send at most 20 messages, then wait for an ack by the client before sending a new one.

[jdavid]

Fixed.