search

lemire / FastPFor

The FastPFOR C++ library: Fast integer compression
Apache License 2.0
883 stars 124 forks source link

Hi, can you help verify this simple 8b? I actually am quite confused over two things. #78

Closed hiqsociety closed 3 years ago

hiqsociety commented 3 years ago 
                • The selector value 2 corresponds to b = 1. This allows us to store 60 integers having values in {0,1}, which are packed in the data bits.

                • The selector value 3 corresponds to b = 2 and allows one to pack 30 integers having values in [0, 4] in the data bits.

shouldn't this be [0,3] instead of [0,4]? i mean it should only allow for 0,1,2,3 values right for bit length of 2. 

                 • Selector values 0 or 1 represent sequences containing 240 and 120 zeros, respectively. In this instance the 60 data bits are ignored.

i'm not sure how to use this 240 or 120 zeros. is it for rare situations only?

hiqsociety commented 3 years ago

  1. I'm asking because I saw online there could be misunderstanding / misrepresentation of what simple8b can be. hope you can clarify.

  2. I've yet to see it being implemented in golang. possible for you to write one?

lemire commented 3 years ago

@hiqsociety This does not appear to be an issue with respect to the code repository FastPFor. It is a not a valid software issue.

It seems, rather, that you are studying schemes from the Simple family introduced by Moffat and collaborators. Moffat and his collaborators are unrelated to the current repository. This scheme was first described in the following reference :

Anh VN, Moffat A. Index compression using 64-bit words. Software: Practice and Experience 2010; 40(2):131– 14

I believe that they have made their source code available. I refer you to this reference and their source code as authoritative.

FastPFor provides a C++ implementation of some Simple schemes which you are invited to study, of course.

I've yet to see it being implemented in golang. possible for you to write one?

FastPFor is a community-maintained C++ library. We are unlikely to add Go code to the project.