Closed gerhard closed 4 years ago
After doing some investigation, I believe this is caused by the rebuilding of message refcounts by rabbit_msg_store. We can the disk asynchronously, and in a few limited tests, I saw that we scan and enqueue faster than we consume, resulting in a very large backlog of messages in memory. The [erlang] queues in question are unbounded, so my initial thought is to bound those queues in a matter that puts are blocking, which will throttle the disk reads to limit the data transiently in memory.
@pjk25 we already have a mechanism for the producer-consumer problem and unbounded buffers in RabbitMQ. It's called credit flow. It's a producer side solution so if we don't control the "producer side" of this process interaction it cannot be used but sounds like we do. No need for blocking puts, producers can self-limit using credit_flow
.
After further exploration, I have learned that the gatherer
managing the above mentioned queues blocks its publishers until messages are consumed, invalidating the above theory.
@michaelklishin thanks for the insight. The queues I were looking at were these https://github.com/rabbitmq/rabbitmq-server/blob/f8d47976d999e8816dacf981f86b71f387dc18fc/src/gatherer.erl#L90, so not "rabbit" queues, although I'm sure I could have made my comment more clear.
It's worth noting that even though we shut down the broker with rabbitmqctl stop_app
first, we still follow this code path, which I believe corresponds to an unclean shutdown https://github.com/rabbitmq/rabbitmq-server/blob/abc0b522de83e36650e873d2065646dc1f4ae1d4/src/rabbit_msg_store.erl#L1735
Also, the call to https://github.com/rabbitmq/rabbitmq-server/blob/abc0b522de83e36650e873d2065646dc1f4ae1d4/src/rabbit_msg_store.erl#L1737 takes about 10 minutes to complete, before we get to the point where memory is consumed rapidly.
I've just tested this in the context of TGIR S01E03 and can confirm that #2274 fixes this.
The memory usage doesn't go above 8GiB and even though the recovery time didn't improve (it still took ~45 minutes for the node to restart), the memory usage was perfect: the ETS data structures used the expected memory, all other components only used what they needed.
Watch the video if you're interested to see how this was confirmed 🙌🏻
TL;DW
Part of TGIR S01E02: Help! RabbitMQ ate my RAM! I came across the following unexpected behaviour when rabbit app gets restart via
rabbitmqctl stop_app ; rabbitmqctl start_app
A worker process is using 56GB of physical memory while reading through all queue segments, across all queues. This operation takes ~30 minutes to complete (13:44:16 - 14:15:46) with the disk reads starting at 410MiB/s and maxing out at 965MiB/s for 15 minutes straight:
This is what the system physical memory breakdown looks like:
Notice:
I have captured the runtime thread stats to show the 88% idle regular schedulers (most time is spent in
send
) and 94% idle dirty_io schedulers. All other thread types are 98% or above idle.I am attaching the netdata export: netdata-c8a3200a7c2f-20200224-130158-6846.snapshot.zip
There is no Prometheus metrics snapshot for the Erlang VM since this behaviour happens before the Prometheus app is started: