search

szcompressor / SZ

Error-bounded Lossy Data Compressor (for floating-point/integer datasets)
http://szcompressor.org
Other
151 stars 56 forks source link

SZ fails decompressing N-Dimensional Data #70

Closed SilkeDH closed 3 years ago

SilkeDH commented 3 years ago

Hi everybody,

I'm currently trying to compress N-Dimensional numpy arrays using SZ with a set absolute error e.g. 0.1. For that, I'm generating some random data like this:

# Array size
n = 1000000
np.random.seed(123)
zexact = np.random.randn(n)
# Reshape 3-D
zexact = zexact.reshape((100,100,100))
zexact = zexact.astype(np.float32)

# Save file:
filename = 'zexact.dat'
zexact.tofile(filename)

Next, I run the following commands:

!./sz -z -f  -M ABS -A 0.1 -i zexact.dat -3 100 100 100
!./sz -x -f -s zexact.dat.sz -3 100 100 100 -a -i zexact.dat

The output I get is:

Min=-4.6071300506591796875, Max=4.6275801658630371094, range=9.234710693359375
Max absolute error = 179516227584.0000000000
Max relative error = 19439290368.000000
Max pw relative error = 1672789715114.853271
PSNR = -158.353887, NRMSE= 82735967.982407003641
normError = 764042728252.572754, normErr_norm = 1.000000
acEff=-0.001861
compressionRatio=5.487985
decompression time = 0.058381 seconds.
decompressed data file: zexact.dat.sz.out

As you can see, the error bound is not respected. However, when I reshape the array to be 1-D, it works.

!./sz -z -f  -M ABS -A 0.1 -i zexact.dat -1 1000000
!./sz -x -f -s zexact.dat.sz -1 1000000 -a -i zexact.dat

The output:

Min=-4.6071300506591796875, Max=4.6275801658630371094, range=9.234710693359375
Max absolute error = 0.1000547409
Max relative error = 0.010835
Max pw relative error = 9727.824053
PSNR = 44.072084, NRMSE= 0.0062574274392154221464
normError = 57.785532, normErr_norm = 0.057676
acEff=0.998335
compressionRatio=6.511529
decompression time = 0.024181 seconds.
decompressed data file: zexact.dat.sz.out

Is there any way you could help me? Thank you 🦊

disheng222 commented 3 years ago

Is it possible to send us the zexact.dat file for a test? You can send it to @.*** Thanks. Best, Sheng

On Thu, Jun 24, 2021 at 5:02 AM Silke Donayre @.***> wrote:

Hi everybody,

I'm currently trying to compress N-Dimensional numpy arrays using SZ with a set absolute error e.g. 0.1. For that, I'm generating some random data like this:

Array size

n = 1000000 np.random.seed(123) zexact = np.random.randn(n)

Reshape 3-D

zexact = zexact.reshape((100,100,100)) zexact = zexact.astype(np.float32)

Save file:

filename = 'zexact.dat' zexact.tofile(filename)

Next, I run the following commands:

!./sz -z -f -M ABS -A 0.1 -i zexact.dat -3 100 100 100

!./sz -x -f -s zexact.dat.sz -3 100 100 100 -a -i zexact.dat

The output I get is:

Min=-4.6071300506591796875, Max=4.6275801658630371094, range=9.234710693359375

Max absolute error = 179516227584.0000000000

Max relative error = 19439290368.000000

Max pw relative error = 1672789715114.853271

PSNR = -158.353887, NRMSE= 82735967.982407003641

normError = 764042728252.572754, normErr_norm = 1.000000

acEff=-0.001861

compressionRatio=5.487985

decompression time = 0.058381 seconds.

decompressed data file: zexact.dat.sz.out

As you can see, the error bound is not respected. However, when I reshape the array to be 1-D, it works.

!./sz -z -f -M ABS -A 0.1 -i zexact.dat -1 1000000

!./sz -x -f -s zexact.dat.sz -1 1000000 -a -i zexact.dat

The output:

Min=-4.6071300506591796875, Max=4.6275801658630371094, range=9.234710693359375

Max absolute error = 0.1000547409

Max relative error = 0.010835

Max pw relative error = 9727.824053

PSNR = 44.072084, NRMSE= 0.0062574274392154221464

normError = 57.785532, normErr_norm = 0.057676

acEff=0.998335

compressionRatio=6.511529

decompression time = 0.024181 seconds.

decompressed data file: zexact.dat.sz.out

Is there any way you could help me? Thank you 🦊

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/szcompressor/SZ/issues/70, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACK3KSLYZ2BB6IHUV6V7Z63TUL7ENANCNFSM47HRWNSA .

disheng222 commented 3 years ago

I have generated the data file using your script. I will check it later on.

On Thu, Jun 24, 2021 at 8:59 AM Sheng Di @.***> wrote:

Is it possible to send us the zexact.dat file for a test? You can send it to @.*** Thanks. Best, Sheng

On Thu, Jun 24, 2021 at 5:02 AM Silke Donayre @.***> wrote:

Hi everybody,

I'm currently trying to compress N-Dimensional numpy arrays using SZ with a set absolute error e.g. 0.1. For that, I'm generating some random data like this:

Array size

n = 1000000 np.random.seed(123) zexact = np.random.randn(n)

Reshape 3-D

zexact = zexact.reshape((100,100,100)) zexact = zexact.astype(np.float32)

Save file:

filename = 'zexact.dat' zexact.tofile(filename)

Next, I run the following commands:

!./sz -z -f -M ABS -A 0.1 -i zexact.dat -3 100 100 100

!./sz -x -f -s zexact.dat.sz -3 100 100 100 -a -i zexact.dat

The output I get is:

Min=-4.6071300506591796875, Max=4.6275801658630371094, range=9.234710693359375

Max absolute error = 179516227584.0000000000

Max relative error = 19439290368.000000

Max pw relative error = 1672789715114.853271

PSNR = -158.353887, NRMSE= 82735967.982407003641

normError = 764042728252.572754, normErr_norm = 1.000000

acEff=-0.001861

compressionRatio=5.487985

decompression time = 0.058381 seconds.

decompressed data file: zexact.dat.sz.out

As you can see, the error bound is not respected. However, when I reshape the array to be 1-D, it works.

!./sz -z -f -M ABS -A 0.1 -i zexact.dat -1 1000000

!./sz -x -f -s zexact.dat.sz -1 1000000 -a -i zexact.dat

The output:

Min=-4.6071300506591796875, Max=4.6275801658630371094, range=9.234710693359375

Max absolute error = 0.1000547409

Max relative error = 0.010835

Max pw relative error = 9727.824053

PSNR = 44.072084, NRMSE= 0.0062574274392154221464

normError = 57.785532, normErr_norm = 0.057676

acEff=0.998335

compressionRatio=6.511529

decompression time = 0.024181 seconds.

decompressed data file: zexact.dat.sz.out

Is there any way you could help me? Thank you 🦊

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/szcompressor/SZ/issues/70, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACK3KSLYZ2BB6IHUV6V7Z63TUL7ENANCNFSM47HRWNSA .

disheng222 commented 3 years ago

Hi, I didn't encounter that issue in my test, as shown below. @. example]$ ./sz -z -f -M ABS -A 0.1 -i zexact.dat -3 100 100 100 compression time = 0.042818 compressed data file: zexact.dat.sz @. example]$ ./sz -x -f -s zexact.dat.sz -3 100 100 100 -a -i zexact.dat Min=-4.6071300506591796875, Max=4.6275801658630371094, range=9.234710693359375 Max absolute error = 0.0999999046 Max relative error = 0.010829 Max pw relative error = 38791.086342 PSNR = 44.083308, NRMSE= 0.00624934655209032075 normError = 57.710907, normErr_norm = 0.057619 acEff=0.998339 compressionRatio=5.487804 decompression time = 0.031787 seconds. decompressed data file: zexact.dat.sz.out @.*** example]$

Which version are you using? I am using the 'git clone' master version.

Best, Sheng

On Thu, Jun 24, 2021 at 9:11 AM Sheng Di @.***> wrote:

I have generated the data file using your script. I will check it later on.

On Thu, Jun 24, 2021 at 8:59 AM Sheng Di @.***> wrote:

Is it possible to send us the zexact.dat file for a test? You can send it to @.*** Thanks. Best, Sheng

On Thu, Jun 24, 2021 at 5:02 AM Silke Donayre @.***> wrote:

Hi everybody,

I'm currently trying to compress N-Dimensional numpy arrays using SZ with a set absolute error e.g. 0.1. For that, I'm generating some random data like this:

Array size

n = 1000000 np.random.seed(123) zexact = np.random.randn(n)

Reshape 3-D

zexact = zexact.reshape((100,100,100)) zexact = zexact.astype(np.float32)

Save file:

filename = 'zexact.dat' zexact.tofile(filename)

Next, I run the following commands:

!./sz -z -f -M ABS -A 0.1 -i zexact.dat -3 100 100 100

!./sz -x -f -s zexact.dat.sz -3 100 100 100 -a -i zexact.dat

The output I get is:

Min=-4.6071300506591796875, Max=4.6275801658630371094, range=9.234710693359375

Max absolute error = 179516227584.0000000000

Max relative error = 19439290368.000000

Max pw relative error = 1672789715114.853271

PSNR = -158.353887, NRMSE= 82735967.982407003641

normError = 764042728252.572754, normErr_norm = 1.000000

acEff=-0.001861

compressionRatio=5.487985

decompression time = 0.058381 seconds.

decompressed data file: zexact.dat.sz.out

As you can see, the error bound is not respected. However, when I reshape the array to be 1-D, it works.

!./sz -z -f -M ABS -A 0.1 -i zexact.dat -1 1000000

!./sz -x -f -s zexact.dat.sz -1 1000000 -a -i zexact.dat

The output:

Min=-4.6071300506591796875, Max=4.6275801658630371094, range=9.234710693359375

Max absolute error = 0.1000547409

Max relative error = 0.010835

Max pw relative error = 9727.824053

PSNR = 44.072084, NRMSE= 0.0062574274392154221464

normError = 57.785532, normErr_norm = 0.057676

acEff=0.998335

compressionRatio=6.511529

decompression time = 0.024181 seconds.

decompressed data file: zexact.dat.sz.out

Is there any way you could help me? Thank you 🦊

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/szcompressor/SZ/issues/70, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACK3KSLYZ2BB6IHUV6V7Z63TUL7ENANCNFSM47HRWNSA .

SilkeDH commented 3 years ago

Hi @disheng222,

weird 😕, I have the latest version as well, I've cloned the master repository again, just to be sure I've got everything right, but the issue is still there.

I really don't know what could be the issue here, i've installed SZ following the installation way 2 and that's basically it.

disheng222 commented 3 years ago

I tested it on two other machines, which also show correct results. Could you run ./sz -v to check the version number. My run shows 2.1.12. @.*** example]$ ./sz -v version: 2.1.12

Best, Sheng

On Thu, Jun 24, 2021 at 2:50 PM Silke Donayre @.***> wrote:

Hi @disheng222 https://github.com/disheng222,

weird 😕, I have the latest version as well, I've cloned the master repository again, just to be sure I've got everything right, but the issue is still there.

I really don't know what could be the issue here, i've installed SZ following the installation way 2 and that's basically it.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/szcompressor/SZ/issues/70#issuecomment-867908505, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACK3KSN4QHOSWO6O2O54TTLTUOEBZANCNFSM47HRWNSA .

SilkeDH commented 3 years ago

Sure, it shows:

!./sz -v version: 2.1.12

it is the same version. I run the sz executable in build/bin/sz.

disheng222 commented 3 years ago

That's weird. Could you pack the whole package and send it to my email: @.***?

Best, Sheng

On Fri, Jun 25, 2021 at 10:11 AM Silke Donayre @.***> wrote:

Sure, it shows:

!./sz -v version: 2.1.12

it is the same version. I run the sz executable in build/bin/sz.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/szcompressor/SZ/issues/70#issuecomment-868568783, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACK3KSMZJ5XOZXDS57LD5QTTUSMARANCNFSM47HRWNSA .

disheng222 commented 3 years ago

In fact, your 1D test is also inconsistent with my test, as shown below. The Max absolute error bound should be "completely" respected. Your result shows a max error of 0.1000547409, which used to happen in a very old version of SZ.

@.*** example]$ ./sz -x -f -s zexact.dat.sz -1 1000000 -a -i zexact.dat Min=-4.6071300506591796875, Max=4.6275801658630371094, range=9.234710693359375 Max absolute error = 0.0999999046 Max relative error = 0.010829 Max pw relative error = 9733.318911 PSNR = 44.072084, NRMSE= 0.0062574273273227985076 normError = 57.785531, normErr_norm = 0.057676 acEff=0.998335 compressionRatio=6.511529 decompression time = 0.049010 seconds. decompressed data file: zexact.dat.sz.out

On Fri, Jun 25, 2021 at 10:15 AM Sheng Di @.***> wrote:

That's weird. Could you pack the whole package and send it to my email: @.***?

Best, Sheng

On Fri, Jun 25, 2021 at 10:11 AM Silke Donayre @.***> wrote:

Sure, it shows:

!./sz -v version: 2.1.12

it is the same version. I run the sz executable in build/bin/sz.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/szcompressor/SZ/issues/70#issuecomment-868568783, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACK3KSMZJ5XOZXDS57LD5QTTUSMARANCNFSM47HRWNSA .

SilkeDH commented 3 years ago

I tried to install SZ on another machine and it worked! The output I get is consistent with yours and also works for N-dimensions 😊:

!./sz -z -f  -M ABS -A 0.1 -i zexact.dat -3 100 100 100
!./sz -x -f -s zexact.dat.sz -3 100 100 100 -a -i zexact.dat

Output:

compression time = 0.090085
compressed data file: zexact.dat.sz
Min=-4.6071300506591796875, Max=4.6275801658630371094, range=9.234710693359375
Max absolute error = 0.0999999046
Max relative error = 0.010829
Max pw relative error = 38791.086342
PSNR = 44.083308, NRMSE= 0.00624934655209032075
normError = 57.710907, normErr_norm = 0.057619
acEff=0.998339
compressionRatio=5.487804
decompression time = 0.083607 seconds.
decompressed data file: zexact.dat.sz.out

I really don't know what is happening on the other machine..., I'll try to run some tests, maybe I find the bug... For now, I will be using this machine for the compression.

Thank you for helping me!