add transform LZ4 comression

ornladios / ADIOS

The old ADIOS 1.x code repository. Look for ADIOS2 for new repo

https://csmd.ornl.gov/adios

Other

54 stars 41 forks source link

add transform LZ4 comression #112

Closed psychocoderHPC closed 7 years ago

psychocoderHPC commented 7 years ago

Implement issue #101 (LZ4 lossless compression transformation layer)

add adios_transformlz4*
add LZ4 to cmake build
add LZ4 to configure build
add LZ4 section to the latex documentation

plugin options

threshold=128 - boundary to store small data uncompressed
lvl=1 - compression level (default=1) higher means less compression ratio but more performance

scaling within a real word example PIConGPU

picongpu_adios_scaling

Node configuration: 2 x Intel(R) Xeon(R) CPU E5-2609 0 @ 2.40GHz and 4 x NVIDIA k20

CC-ing: @ax3l

pnorbert commented 7 years ago

Thanks a lot. What is the orange dashed line in your figure?

psychocoderHPC commented 7 years ago

The orange dashed line is the theoretical ideal weak io scaling based on one node (4GPUs) for LZ4 and the blue dashed line is the some for non compression.

ax3l commented 7 years ago

We just extrapolated a bit naïvely from one node in both cases, so one gets a guide for the eye for the offset to an "ideal" scaling. Of course, the actual BW hitting the filesys is always a bit higher, but we keep our data-preparation overhead in the "effective BW" since that's the actual overhead a simulation with and without I/O will experience.

Thanks again to @psychocoderHPC for the implementation and verification!