pjk25
commented
4 years ago After doing some investigation, I believe this is caused by the rebuilding of message refcounts by rabbit_msg_store. We can the disk asynchronously, and in a few limited tests, I saw that we scan and enqueue faster than we consume, resulting in a very large backlog of messages in memory. The [erlang] queues in question are unbounded, so my initial thought is to bound those queues in a matter that puts are blocking, which will throttle the disk reads to limit the data transiently in memory.
michaelklishin
gerhard
Part of TGIR S01E02: Help! RabbitMQ ate my RAM! I came across the following unexpected behaviour when rabbit app gets restart via
rabbitmqctl stop_app ; rabbitmqctl start_appA worker process is using 56GB of physical memory while reading through all queue segments, across all queues. This operation takes ~30 minutes to complete (13:44:16 - 14:15:46) with the disk reads starting at 410MiB/s and maxing out at 965MiB/s for 15 minutes straight:
This is what the system physical memory breakdown looks like:
Notice:
I have captured the runtime thread stats to show the 88% idle regular schedulers (most time is spent in
send) and 94% idle dirty_io schedulers. All other thread types are 98% or above idle.I am attaching the netdata export: netdata-c8a3200a7c2f-20200224-130158-6846.snapshot.zip
There is no Prometheus metrics snapshot for the Erlang VM since this behaviour happens before the Prometheus app is started: