search

quickwit-oss / bitpacking

SIMD algorithms for integer compression via bitpacking. This crate is a port of a C library called simdcomp.
MIT License
272 stars 30 forks source link

Compilation require heavy amount of memory #14

Closed trinity-1686a closed 5 years ago

trinity-1686a commented 5 years ago

Building bitpacking has a peek memory usage of 1.9Gio on my system, and caused issue in CI and for some users building on lower-end hardware (see Plume-org/Plume#632). This seems to be directly related to heavy usage of macro for code expansion. Once code is expended, I measured a bit more than 330k loc. This is distributed approximately as 1/5 of bitpacker1x, 2/5 of bitpacker4x, and 2/5 bitpacker8x. Commenting mod bitpacker8x;reduced peek memory usage to 1.2Gio, so it might be interesting to have features to enable/disable individual bitpackers depending on what is required. For instance, Tantivy seems to use only 4x, so it's probably possible to go sub 1G by just disabling unneeded bitpackers

fulmicoton commented 5 years ago

This is a great suggestion. Thanks for having investigated.

trinity-1686a commented 5 years ago

fixed by #15 and tantivy-search/tantivy#589