manticoresoftware / columnar

Manticore Columnar Library
Apache License 2.0
82 stars 15 forks source link

columnar: integrate streamvbyte library #30

Closed githubmanticore closed 1 year ago

githubmanticore commented 1 year ago

streamvbyte is a codec that uses AVX2, it is included in the fastpfor library. However, most recent version is developed as a separate library, https://github.com/lemire/streamvbyte. We need to integrate that library with MCL as this will allow us to use a most recent version of the codec and use a version of it that has built-in differential coding. Using that built-in version should be faster than using external code to calc deltas and inverse deltas. The downside is that it requires implementing a specialized codec in the MCL.

githubmanticore commented 1 year ago

➤ githubmanticore commented:

➤ Sergey Nikolaev commented:

rought estimates:

glookka commented 1 year ago

Done in 52c59922