Building bitpacking has a peek memory usage of 1.9Gio on my system, and caused issue in CI and for some users building on lower-end hardware (see Plume-org/Plume#632).
This seems to be directly related to heavy usage of macro for code expansion. Once code is expended, I measured a bit more than 330k loc. This is distributed approximately as 1/5 of bitpacker1x, 2/5 of bitpacker4x, and 2/5 bitpacker8x.
Commenting mod bitpacker8x;reduced peek memory usage to 1.2Gio, so it might be interesting to have features to enable/disable individual bitpackers depending on what is required. For instance, Tantivy seems to use only 4x, so it's probably possible to go sub 1G by just disabling unneeded bitpackers
Building bitpacking has a peek memory usage of 1.9Gio on my system, and caused issue in CI and for some users building on lower-end hardware (see Plume-org/Plume#632). This seems to be directly related to heavy usage of macro for code expansion. Once code is expended, I measured a bit more than 330k loc. This is distributed approximately as 1/5 of bitpacker1x, 2/5 of bitpacker4x, and 2/5 bitpacker8x. Commenting
mod bitpacker8x;
reduced peek memory usage to 1.2Gio, so it might be interesting to have features to enable/disable individual bitpackers depending on what is required. For instance, Tantivy seems to use only 4x, so it's probably possible to go sub 1G by just disabling unneeded bitpackers