search

Blosc / c-blosc

A blocking, shuffling and loss-less compression library that can be faster than `memcpy()`.
https://www.blosc.org
Other
986 stars 158 forks source link

Update blosc lz4 version to r126 #74

Closed crspeller closed 9 years ago

crspeller commented 9 years ago

When lz4 r125 is released blosc should update to this revision. There is a rare case that causes a crash and a potential security issue that will be fixed in this revision.

FrancescAlted commented 9 years ago

I tried that, and I am getting segfaults on my machine (Ubuntu 14.10, gcc 4.9.1, Intel CPU):

$ cd bench
$ gcc-4.9 -g -O3 -msse2 ../blosc/*.c ../internal-complibs/lz4-r125/*.c  bench.c -o bench -lpthread -I../blosc -DHAVE_LZ4 -I ../internal-complibs/lz4-r125/
faltet@francesc-Latitude-E6430:~/blosc/c-blosc/bench$ ./bench lz4 single 1 1000
Blosc version: 1.5.1.dev ($Date:: 2014-12-14 #$)
List of supported compressors in this build: blosclz,lz4,lz4hc
Supported compression libraries:
  BloscLZ: 1.0.3
  LZ4: 1.4.1
  Snappy: unknown
  Zlib: unknown
Using compressor: lz4
Running suite: single
--> 1, 1000, 8, 19, lz4
********************** Run info ******************************
Blosc version: 1.5.1.dev ($Date:: 2014-12-14 #$)
Using synthetic data with 19 significant bits (out of 32)
Dataset size: 1000 bytes        Type size: 8 bytes
Working set: 31.2 MB            Number of threads: 1
********************** Running benchmarks *********************
memcpy(write):              0.2 us, 5994.2 MB/s
memcpy(read):               0.1 us, 6408.1 MB/s
Compression level: 0
comp(write):        0.2 us, 4325.3 MB/s   Final bytes: 1016  Ratio: 0.98
decomp(read):       0.1 us, 6786.1 MB/s   OK
Compression level: 1
comp(write):        1.9 us, 491.9 MB/s    Final bytes: 634  Ratio: 1.58
Violació de segment (bolcat de la imatge del nucli)

and valgrind if pointing to this:

Compression level: 1
comp(write):       77.1 us, 12.4 MB/s     Final bytes: 634  Ratio: 1.58
==6717== 
==6717== Process terminating with default action of signal 11 (SIGSEGV)
==6717==  General Protection Fault
==6717==    at 0x40AED0: LZ4_copy8 (lz4.c:270)
==6717==    by 0x40AED0: LZ4_wildCopy (lz4.c:285)
==6717==    by 0x40AED0: LZ4_decompress_generic (lz4.c:1021)
==6717==    by 0x40AED0: LZ4_decompress_fast (lz4.c:1131)
==6717==    by 0x401E4D: lz4_wrap_decompress (blosc.c:405)
==6717==    by 0x401E4D: blosc_d.isra.4 (blosc.c:629)
==6717==    by 0x40403E: serial_blosc (blosc.c:736)
==6717==    by 0x40403E: do_job (blosc.c:794)
==6717==    by 0x404929: blosc_run_decompression_with_context (blosc.c:1195)
==6717==    by 0x404929: blosc_decompress (blosc.c:1224)
==6717==    by 0x41ADA7: do_bench (bench.c:261)
==6717==    by 0x40158A: main (bench.c:525)

After several trials, I determined that the problem only appears when using gcc 4.9.1 and -O3 flag (I suppose this triggers the use of SSE instructions on Intel). Other compilers like gcc 4.8.3 or 4.7.4 and clang 3.3, 3.4 and 3.5 work just fine.

After a bit of googling, this seems to be exactly the problem as reported in: https://github.com/Cyan4973/lz4/issues/44

So I am afraid that GCC 4.9.1 is really flawed and that LZ4 1.4.1 (r125) is unfortunately triggering the issue. As GCC 4.9.1 is the default in Ubuntu 14.10 (and god knows how many other platforms out there), I don't know which the solution would be. For the time being will reference this issue from the original LZ4 github issue and hopefully someone could offer a decent solution for this.

FrancescAlted commented 9 years ago

So it seems that Yann is going to address this in a next release, so let's wait for it.

crspeller commented 9 years ago

Sounds good. Changed the title to r126.

FrancescAlted commented 9 years ago

Done in 29cec23. Can you give it a go? I will leave this open until r126 actually hits the official release.

FrancescAlted commented 9 years ago

Updated to rc126 in rev 3c2cf61. I think this can be closed already.