search

dkpro / dkpro-similarity

Word and text similarity measures
https://dkpro.github.io/dkpro-similarity
Other
53 stars 22 forks source link

Alternative compression for vector space models #45

Open iokuznetsov opened 8 years ago

iokuznetsov commented 8 years ago

Current implementation of VectorBinding supports compression/decompression via GZIP and BZIP2. However, it might be useful to have more advanced compression methods as well, e.g. LZO, LZ4 or Snappy, since I/O and decompression are one of the bottlenecks in similarity-intensive applications. Some benchmarking results can be found here: http://catchchallenger.first-world.info/wiki/Quick_Benchmark:_Gzip_vs_Bzip2_vs_LZMA_vs_XZ_vs_LZ4_vs_LZO. At least LZO and Snappy provide Input/OutputStream objects and their Java implementations are on Maven Central, so it should be relatively easy to integrate. However, some knowledge of those libraries is required to get an optimal compression/speed ratio.

reckart commented 8 years ago

I think that Apache Commons Compress supports most (all?) of the methods you are suggesting and using it here should make it pretty trivial to switch between different algorithms.