lucianopaz / compress_pickle

Standard python pickle, thinly wrapped with standard compression libraries

MIT License

42 stars 11 forks source link

added lz4 compression. use compression_level=0 in kwargs. #11

Closed ctmakro closed 4 years ago

ctmakro commented 4 years ago

lz4(compression_level=0) is much(~50%) faster than gzip for sending pickles over 1Gb Ethernet/ SATA2 connections. compressed files are larger thus slower to transmit, but the time spent on compression is greatly reduced.

codecov[bot] commented 4 years ago

Codecov Report

Merging #11 into master will decrease coverage by 0.20%. The diff coverage is n/a.

@@             Coverage Diff             @@
##            master      #11      +/-   ##
===========================================
- Coverage   100.00%   99.79%   -0.21%     
===========================================
  Files            7        7              
  Lines          493      496       +3     
===========================================
+ Hits           493      495       +2     
- Misses           0        1       +1

ctmakro commented 4 years ago

some tests not passing, close.