search

szcompressor / SZ

Error-bounded Lossy Data Compressor (for floating-point/integer datasets)
http://szcompressor.org
Other
151 stars 56 forks source link

SZ principle work #94

Closed mearcla closed 2 years ago

mearcla commented 2 years ago

Is there any paper explain the principle work of the sz algorithm? Thanks,

disheng222 commented 2 years ago

I forgot to mention that SZ2.1 is maintained here: https://github.com/szcompressor/SZ3

On Tue, Jul 19, 2022 at 12:24 PM Sheng Di @.***> wrote:

For SZ2.1, please check this paper: Xin Liang, Sheng Di, Dingwen Tao, Sihuan Li, Shaomeng Li, Hanqi Guo, Zizhong Chen, Franck Cappello, "Error-Controlled Lossy Compression Optimized for High Compression Ratios of Scientific Datasets", in IEEE Bigdata2018, 2018.

You may also be interested in SZ3.1, which has a much higher compression ratio than SZ2.1 in many cases but slightly lower compression speed. SZ3.1's principle is explained in this paper: Kai Zhao, Sheng Di, Maxim Dmitriev, Thierry-Laurent D. Tonellot, Zizhong Chen, and Franck Cappello. “Optimizing Error-Bounded Lossy Compression for Scientific Data by Dynamic Spline Interpolation.” In 2021 IEEE International Conference on Data Engineering (ICDE), Chania, Crete, Greece, pp. 1643-1654, Apr 19 - 22, 2021.

On Mon, Jul 18, 2022 at 6:47 AM mearcla @.***> wrote:

Is there any paper explain the principle work of the sz algorithm?

— Reply to this email directly, view it on GitHub https://github.com/szcompressor/SZ/issues/94, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACK3KSKHHTOKLXGRWVNGHW3VUUY3NANCNFSM533ZL2RQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>

mearcla commented 2 years ago

I used SZ 2.1 so I think this paper explains well about the algorithm too "Fast Error-bounded Lossy HPC Data Compression with SZ"

disheng222 commented 2 years ago

FYI, the paper "Fast Error-bounded Lossy HPC Data Compression with SZ" corresponds to SZ0.1, which is a deprecated version. The paper "Significantly Improving Lossy Compression for Scientific Data Sets Based on Multidimensional Prediction and Error-Controlled Quantization" corresponds to the SZ1.4, which has been integrated in SZ2.1 SZ 2.1's paper is this one: Xin Liang, Sheng Di, Dingwen Tao, Sihuan Li, Shaomeng Li, Hanqi Guo, Zizhong Chen, Franck Cappello, "Error-Controlled Lossy Compression Optimized for High Compression Ratios of Scientific Datasets", in IEEE Bigdata2018, 2018.

On Wed, Jul 20, 2022 at 10:15 AM mearcla @.***> wrote:

I used SZ 2.1 so I think this paper explains well about the algorithm too "Fast Error-bounded Lossy HPC Data Compression with SZ"

— Reply to this email directly, view it on GitHub https://github.com/szcompressor/SZ/issues/94#issuecomment-1190343473, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACK3KSMCXRKD7C2DROU6XF3VVACZPANCNFSM533ZL2RQ . You are receiving this because you commented.Message ID: @.***>

mearcla commented 2 years ago

Yes, I read it now thank you for this clarification. What about this [SZ 2.1.12.2], please?

disheng222 commented 2 years ago

It's in the paper published by IEEE Bigdata2018, 2018.

On Thu, Jul 21, 2022 at 2:55 AM mearcla @.***> wrote:

Yes, I read it now thank you for this clarification. What about this [SZ 2.1.12.2], please?

— Reply to this email directly, view it on GitHub https://github.com/szcompressor/SZ/issues/94#issuecomment-1191165765, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACK3KSISS7ENNXV24XS2YB3VVD673ANCNFSM533ZL2RQ . You are receiving this because you commented.Message ID: @.***>

disheng222 commented 1 year ago

For SZ2.1, please check this paper: Xin Liang, Sheng Di, Dingwen Tao, Sihuan Li, Shaomeng Li, Hanqi Guo, Zizhong Chen, Franck Cappello, "Error-Controlled Lossy Compression Optimized for High Compression Ratios of Scientific Datasets", in IEEE Bigdata2018, 2018.

You may also be interested in SZ3.1, which has a much higher compression ratio than SZ2.1 in many cases but slightly lower compression speed. SZ3.1's principle is explained in this paper: Kai Zhao, Sheng Di, Maxim Dmitriev, Thierry-Laurent D. Tonellot, Zizhong Chen, and Franck Cappello. “Optimizing Error-Bounded Lossy Compression for Scientific Data by Dynamic Spline Interpolation.” In 2021 IEEE International Conference on Data Engineering (ICDE), Chania, Crete, Greece, pp. 1643-1654, Apr 19 - 22, 2021.

On Mon, Jul 18, 2022 at 6:47 AM mearcla @.***> wrote:

Is there any paper explain the principle work of the sz algorithm?

— Reply to this email directly, view it on GitHub https://github.com/szcompressor/SZ/issues/94, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACK3KSKHHTOKLXGRWVNGHW3VUUY3NANCNFSM533ZL2RQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>