lemire / FastPFor

The FastPFOR C++ library: Fast integer compression
Apache License 2.0
883 stars 124 forks source link

codecs --clusterdynamic runtime failure #104

Closed andrewthad closed 1 year ago

andrewthad commented 1 year ago

The clusterdynamic benchmark consistently fails. The first part, 1024 arrays of size 32768, succeeds. The second part, 32 arrays of size 1048576, fails. Here is what is printed to stderr:

Input's out of range: 321214499
terminate called after throwing an instance of 'std::runtime_error'
  what():  You tried to apply Simple16 to an incompatible set of integers: they should be in [0,2^28).

And here is what is printed to stdout leading up to the failure (I doubt this is useful):

# found clusterdynamic
# dynamic clustered data generation...
# generated 1024 arrays
# their size is  32768
#BP32+VariableByte  @BP32+VariableByte  JustCopy    @JustCopy   FastBinaryPacking16+VariableByte    @FastBinaryPacking16+VariableByte   FastBinaryPacking32+VariableByte    @FastBinaryPacking32+VariableByte   FastBinaryPacking8+VariableByte @FastBinaryPacking8+VariableByte    FastPFor128+VariableByte    @FastPFor128+VariableByte   FastPFor256+VariableByte    @FastPFor256+VariableByte   MaskedVByte @MaskedVByte    NewPFor<4,Simple16>+VariableByte    @NewPFor<4,Simple16>+VariableByte   OPTPFor<4,Simple16>+VariableByte    @OPTPFor<4,Simple16>+VariableByte   PFor+VariableByte   @PFor+VariableByte  PFor2008+VariableByte   @PFor2008+VariableByte  SIMDBinaryPacking+VariableByte  @SIMDBinaryPacking+VariableByte SIMDFastPFor128+VariableByte    @SIMDFastPFor128+VariableByte   SIMDFastPFor256+VariableByte    @SIMDFastPFor256+VariableByte   SIMDGroupSimple+VariableByte    @SIMDGroupSimple+VariableByte   SIMDGroupSimple_RingBuf+VariableByte    @SIMDGroupSimple_RingBuf+VariableByte   SIMDNewPFor<4,Simple16>+VariableByte    @SIMDNewPFor<4,Simple16>+VariableByte   SIMDOPTPFor<4,Simple16>+VariableByte    @SIMDOPTPFor<4,Simple16>+VariableByte   SIMDPFor+VariableByte   @SIMDPFor+VariableByte  SIMDSimplePFor+VariableByte @SIMDSimplePFor+VariableByte    Simple16    @Simple16   Simple8b    @Simple8b   Simple8b_RLE    @Simple8b_RLE   Simple9 @Simple9    Simple9_RLE @Simple9_RLE    SimplePFor+VariableByte @SimplePFor+VariableByte    streamvbyte @streamvbyte    VariableByte    @VariableByte   VarIntG8IU  @VarIntG8IU varintgb    @varintgb   VByte   @VByte  VSEncoding  @VSEncoding 
# for each scheme we give compression speed (million int./s) decompression speed and bits per integer
14  792.7   821.6   15.11       883.1   1125    16.18       2213    2161    32      2612    2714    32      778.7   789.5   15.54       824.4   852.7   16.65       1013    1009    15.11       1105    1124    16.18       476.9   559.2   16.59       527.3   621.9   18      289.5   928.4   14.71       308.7   628.4   16.09       230.6   1063    14.71       385.6   1216    16.1        296.6   764.4   17.13       560.1   895 20.67       134.71151   15.58       134.3   1240    17.48       6.817   649.1   14.76       8.486   486.5   16.57       325.3   1113    17.65       346.9   943.6   19.23       310.3   1213    17.65       306.7   1093    19.24       1953    243615.53       2258    2958    16.51       344 1862    14.71       418.5   2397    16.09       396.2   2074    14.71       466.2   2499    16.11       280.8   1231    18.64       268.9   1345    22.96       289.9   1193    19.08259.9  1382    23.26       137.1   1760    15.58       140.6   2322    17.48       6.844   839.2   14.76       7.955   650.2   16.57       417.7   2185    17.65       441.5   2586    19.23       343.5   2035    14.7        419.22419   16.09       172.4   330.5   20.32       251.8   430.9   25.69       242.3   431.8   16.23       344.4   420.7   18.12       77.4    508.4   16.23       86.88   505.1   18.12       198 325.6   20.42       255.8   421.425.72      66.28   320.4   20.42       64.18   252.4   25.72       313.4   1071    14.7        355.2   1166    16.09       384.4   2234    17.19       461.7   2629    20.01       254.8   244.5   17.13       390.2   368.6   20.67161.2  1621    17.75       164.4   1847    22.61       440.9   770.7   17.19       475.3   809.4   20.01       253.9   263.7   17.13       389.6   409.3   20.67       11.93   693.8   16.44       14.22   791.4   18.08       
# generated 32 arrays
# their size is  1048576
#BP32+VariableByte  @BP32+VariableByte  JustCopy    @JustCopy   FastBinaryPacking16+VariableByte    @FastBinaryPacking16+VariableByte   FastBinaryPacking32+VariableByte    @FastBinaryPacking32+VariableByte   FastBinaryPacking8+VariableByte @FastBinaryPacking8+VariableByte    FastPFor128+VariableByte    @FastPFor128+VariableByte   FastPFor256+VariableByte    @FastPFor256+VariableByte   MaskedVByte @MaskedVByte    NewPFor<4,Simple16>+VariableByte    @NewPFor<4,Simple16>+VariableByte   OPTPFor<4,Simple16>+VariableByte    @OPTPFor<4,Simple16>+VariableByte   PFor+VariableByte   @PFor+VariableByte  PFor2008+VariableByte   @PFor2008+VariableByte  SIMDBinaryPacking+VariableByte  @SIMDBinaryPacking+VariableByte SIMDFastPFor128+VariableByte    @SIMDFastPFor128+VariableByte   SIMDFastPFor256+VariableByte    @SIMDFastPFor256+VariableByte   SIMDGroupSimple+VariableByte    @SIMDGroupSimple+VariableByte   SIMDGroupSimple_RingBuf+VariableByte    @SIMDGroupSimple_RingBuf+VariableByte   SIMDNewPFor<4,Simple16>+VariableByte    @SIMDNewPFor<4,Simple16>+VariableByte   SIMDOPTPFor<4,Simple16>+VariableByte    @SIMDOPTPFor<4,Simple16>+VariableByte   SIMDPFor+VariableByte   @SIMDPFor+VariableByte  SIMDSimplePFor+VariableByte @SIMDSimplePFor+VariableByte    Simple16    @Simple16   Simple8b    @Simple8b   Simple8b_RLE    @Simple8b_RLE   Simple9 @Simple9    Simple9_RLE @Simple9_RLE    SimplePFor+VariableByte @SimplePFor+VariableByte    streamvbyte @streamvbyte    VariableByte    @VariableByte   VarIntG8IU  @VarIntG8IU varintgb    @varintgb   VByte   @VByte  VSEncoding  @VSEncoding 
# for each scheme we give compression speed (million int./s) decompression speed and bits per integer
lemire commented 1 year ago

Try/catch added in our main branch just now. Please sync.

Closing.