disheng222
commented
1 year ago Hi, Is it possible that the data at ith time step are affected by the compression errors in previous time steps (such as (i-1)th)? That is, The plotted issue seems related to the error propagation. If you think the execution was not affected by compression errors, can you generate the one snapshot data in binary format and test the compression using SZ’s executable to see if the error still happens?
Best, Sheng
Oriol Tintó Prims @.***>于2023年6月6日 周二上午6:21写道：
Dear developers,
I've been using SZ for a while as a hdf5 filter through python and the hdf5plugin library.
Recently I was setting up some examples and I found a case in which the compressor is failing.
Specifically I was setting up some examples using Xarray's tutorial datasets and for a reason I didn't manage to understand when I use the absolute threshold mode the recovered data the threshold is not respected and the magnitude of the error grows in time (first axis) :
[image: image] https://user-images.githubusercontent.com/34112954/243690576-846159fc-ac15-4a64-85b5-44a8727b2548.png
Any clue on what can be happening?
I tried to reproduce this error with other datasets and I didn't manage to do it.
The same plot using SZ3 would be: [image: image] https://user-images.githubusercontent.com/34112954/243690908-b66e79ea-85dc-4249-8279-b5c8781eaf00.png
The code to produce this plot:
import io import h5pyimport hdf5pluginimport numpy as npimport xarray as xrfrom matplotlib import pyplot as plt
def get_xarray_tutorial_data(): with xr.tutorial.open_dataset("air_temperature") as ds: return ds.air.values
TOLERANCE = 2 ** -4
def main():
Get encoding for the sz HDF5 filter
filter_encoding = hdf5plugin.SZ(absolute=TOLERANCE) # Get dummy data from xarray tutorial data = get_xarray_tutorial_data() # open a file in memory with io.BytesIO() as bio: # Save compressed file with h5py.File(bio, mode='w') as temporary_file: temporary_file.create_dataset("tmp", data=data, **filter_encoding, chunks=data.shape) # Recover data with h5py.File(bio, mode='r') as temporary_file: recovered_data = temporary_file["tmp"][:] # Compute difference diff = recovered_data - data # Get maximum difference for each time-step diff = np.abs(np.max(diff, axis=(1, 2))) # Plot difference plt.plot(diff, label="max diff") # Add theoretical threshold for reference plt.plot([0, len(diff)], [TOLERANCE, TOLERANCE], ":", label=f"tolerance={TOLERANCE}") plt.xlabel("Time") plt.ylabel("Max difference") plt.yscale("log") plt.legend() plt.show() plt.clf()if name == "main": main()
It can run in a container built with:
FROM python:3.11 RUN pip install xarray hdf5plugin h5netcdf pooch matplotlib netCDF4
Thanks in advance.
— Reply to this email directly, view it on GitHub https://github.com/szcompressor/SZ/issues/108, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACK3KSPAFUR7MXWGI6EI6T3XJ4OEBANCNFSM6AAAAAAY4KP3O4 . You are receiving this because you are subscribed to this thread.Message ID: @.***>
-- Sent from my iPhone
orioltinto
robertu94
t20100
epasveer
Dear developers,
I've been using SZ for a while as a hdf5 filter through python and the hdf5plugin library.
Recently I was setting up some examples and I found a case in which the compressor is failing.
Specifically I was setting up some examples using Xarray's tutorial datasets and for a reason I didn't manage to understand when I use the absolute threshold mode the recovered data the threshold is not respected and the magnitude of the error grows in time (first axis) :
Any clue on what can be happening?
I tried to reproduce this error with other datasets and I didn't manage to do it.
The same plot using SZ3 would be:
The code to produce this plot:
It can run in a container built with:
Thanks in advance.