Instead of reading batches into memory then processing we now
process each entry as it is read from the log by introducing a new
ra_log:fold/5 function. This function replaces all prior uses of
ra_log:take which has been removed.
This substantially reduces peak memory use during recovery when, e.g. a quorum queue has a long backlog of largish messages. As a consequence of this recovery takes less time and thus has a positive availability property.
I tested this with a few clustered workloads and could not detect any noticeable performance regressions elsewhere.
Instead of reading batches into memory then processing we now process each entry as it is read from the log by introducing a new ra_log:fold/5 function. This function replaces all prior uses of ra_log:take which has been removed.
This substantially reduces peak memory use during recovery when, e.g. a quorum queue has a long backlog of largish messages. As a consequence of this recovery takes less time and thus has a positive availability property.
I tested this with a few clustered workloads and could not detect any noticeable performance regressions elsewhere.