janelia-flyem / dvid

Distributed, Versioned, Image-oriented Dataservice
http://dvid.io
Other
196 stars 33 forks source link

Ubuntu 16+ and newer Linux builds with lz4 #251

Closed DocSavage closed 6 years ago

DocSavage commented 6 years ago

Some dvid builds use the older cloudflare/lz4 CGo library, which binds directly to old reference C libraries. This library is now unsupported and has build issues with newer OSes due to glibc.

An attempt was made to go to a native Go library (github.com/pierrec/lz4) but a number of issues were detected. Tests of seemingly compressible data returned "incompressible" results from the block compress functions, and the stream compressions were incompatible with previous lz4 clients. Also the compression speed seemed much slower than the bindings to the native C library (decompression seemed sufficient). So this attempt to use native Go library was reverted.

The best near-term solution appears to be updating the CGo library to use newer C reference library, and also testing for compatibility with legacy clients. Once the library has been updated, dvid can move to the newer library.