rabbitmq / ra

A Raft implementation for Erlang and Elixir that strives to be efficient and make it easier to use multiple Raft clusters in a single system.
Other
813 stars 96 forks source link

Avoid growing notifications map during recovery #385

Closed kjnilsson closed 1 year ago

kjnilsson commented 1 year ago

As this could negatively affect recovery performance of long logs.

Proposed Changes

Please describe the big picture of your changes here to communicate to the RabbitMQ team why we should accept this pull request. If it fixes a bug or resolves a feature request, be sure to link to that issue.

A pull request that doesn't explain why the change was made has a much lower chance of being accepted.

If English isn't your first language, don't worry about it and try to communicate the problem you are trying to solve to the best of your abilities. As long as we can understand the intent, it's all good.

Types of Changes

What types of changes does your code introduce to this project? Put an x in the boxes that apply

Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask on the mailing list. We're here to help! This is simply a reminder of what we are going to look for before merging your code.

Further Comments

If this is a relatively large or complex change, kick off the discussion by explaining why you chose the solution you did and what alternatives you considered, etc.

mkuratczyk commented 1 year ago

Nice one! In my test scenario - a quorum queue with 10 million 5kb messages, the recovery process on node startup went down from ~250s to ~170s. On top of that, the memory consumption during recovery went down as well as expected.

Here's the memory usage during the node startup (with just that one queue), first with this PR, then with main: Screenshot 2023-06-27 at 15 32 04