Sensu client hangs with "Detected missing amqp heartbeats"

allantalver commented 7 years ago

We are detecting sensu-client crashing at random time with the error: [amqp] Detected missing amqp heartbeats

Stracing the process shows: select(8, [7], [], [], {18446738800258, 664909}

While it's in hanged state, the memory usage of the process is high. Varying between 3-4GB on different nodes. SIGTERM is not able to stop the process.

From the sample of 4 servers:

After SIGKILL, sensu-client could be started and it resumed normal operations on 1 server.
On 3 servers had to remove sensu package and reinstall. Currently didn't have time to go deeper into the reasons why it doesn't start, but probably it's not related to transport failure and crashing.

Sensu version is 0.26.5-2.

portertech commented 7 years ago

@allantalver was the sensu-client consuming 3-4GB of memory? Or RabbitMQ? What platform and release is the service running on?

allantalver commented 7 years ago

@portertech, it was sensu-client. And it was on Ubuntu 16.04.

portertech commented 7 years ago

@allantalver are you able to upgrade Sensu to 0.29.x and see if the problem persists?

portertech commented 7 years ago

Closing due to inactivity. Please feel free to create a new issue if this matter is not resolved.

sensu / sensu-transport

Sensu client hangs with "Detected missing amqp heartbeats" #49