search

rabbitmq / ra

A Raft implementation for Erlang and Elixir that strives to be efficient and make it easier to use multiple Raft clusters in a single system.
Other
813 stars 96 forks source link

Make snapshot compression options configurable #287

Closed kjnilsson closed 2 years ago

kjnilsson commented 2 years ago

This removes the use of snapshot compression from the ra_log_snapshot module completely. Users that want to use snapshot compression can implement their own ra_snapshot behaviour but is clear that using compression only uses excessive cpu which is likely to be a more scarce resource than disk space.

Use term_to_iovec instead of term_to_binary

kjnilsson commented 2 years ago

Using RabbitMQ whilst consuming 1M messages (which results in snapshots being taken) we have the following before perf capture: out_with_snapshot_comp

And after (With no snapshot compression):

out_snapshot_comp

It is notable that the time spent in longest_match is no longer done. Also on my devbox I got a higher consumption rate whilst capturing with perf (75k msg/sec vs 65k msg/sec). This could be anecdotal however.