There are two change_queues of concern: in aae_treecache used in cached tree rebuilds; in aae_keystore used in parallel store rebuilds.
As aae_treecache rebuilds are generally short, then having the queue unbounded here is ok? So a format_status/2 is added just to tidy up this queue on a crash.
For aae_keystore parallel rebuilds, they can take a long time, and so the change_queue could get very large - certainly large enough to threaten available memory. However, functionally speaking discards are not acceptable. So this has now been switched to a disk_log - slower, but eradicates the memory risk.
This current implementation may not be an efficient use of disk_log. Perhaps larger batch sizes should be used. Perhaps compression should be enabled (e.g. take a larger batch, then term_to_binary with compression before disk_log:ba_log/2
There are two change_queues of concern: in aae_treecache used in cached tree rebuilds; in aae_keystore used in parallel store rebuilds.
As aae_treecache rebuilds are generally short, then having the queue unbounded here is ok? So a format_status/2 is added just to tidy up this queue on a crash.
For aae_keystore parallel rebuilds, they can take a long time, and so the change_queue could get very large - certainly large enough to threaten available memory. However, functionally speaking discards are not acceptable. So this has now been switched to a disk_log - slower, but eradicates the memory risk.
This current implementation may not be an efficient use of disk_log. Perhaps larger batch sizes should be used. Perhaps compression should be enabled (e.g. take a larger batch, then term_to_binary with compression before disk_log:ba_log/2