Compress/decompress in memory, add LZ4/ZSTD support

McStasMcXtrace / iFit

a simple library to analyze data (with McCode and Phonons/DFT hooks). :warning: this project has been moved to https://gitlab.com/soleil-data-treatment/soleil-software-projects/remote-desktop

http://ifit.mccode.org

Other

5 stars 5 forks source link

Compress/decompress in memory, add LZ4/ZSTD support #197

Closed farhi closed 4 years ago

farhi commented 4 years ago

Comment from

https://stackoverflow.com/questions/18803822/matlab-direct-efficient-untar-to-memory-to-avoid-slow-disk-interactions

indicates that a decompress can extract into /run/shm (linux) i.e. directly in memory. Must check size of it before (file size < df('/rm/shm')). Same could be done for e.g. LZ4, ZSTD, etc compress, and then copy result on disk.

Also, looking at untar.m and unzip, we see that the actual implementation derives from Java. Could use native LZ4/ZSRT port then...

farhi commented 4 years ago

Just saw LZ4 Java implementation:

https://github.com/lz4/lz4-java
http://people.apache.org/~rmuir/jacoco-core/org.apache.lucene.codecs.compressing/LZ4.java.html (no dependencies, full source)

farhi commented 4 years ago

A much simpler solution is to add support for LZ4 in:

https://github.com/McStasMcXtrace/iFit/blob/master/Libraries/Loaders/iLoad.m (around line 335)

and just call the system command. Extraction should be possibly in tmpdir or better in /run/shm on Linux systems.