search

HDFGroup / h5pyd

h5py distributed - Python client library for HDF Rest API
Other
114 stars 38 forks source link

FillValue no longer working #44

Closed rsignell-usgs closed 1 year ago

rsignell-usgs commented 6 years ago

I upgraded to the latest h5pyd, and now the fillvalues in my dataset are not working: https://gist.github.com/rsignell-usgs/c3555fd60c391699197d53dcd0cb007c

The fill values in the original netcdf4 file were 1.0e37 but after writing with h5load the new h5pyd thinks the fillvalue is 0.0.

This same notebook worked at the ESIP summer meeting.

ghost commented 6 years ago

Would it be possible to get the original file?

My theory (without a shred of evidence) is that there is some confusion as to from where the fill value should come. dset.fillvalue gives the fill value from the HDF5 dataset's creation property list. And it's default value is 0.0. What is the fill value you are referring to? The one stored in the dataset's _FillValue attribute?

jreadey commented 6 years ago

I have the source file - I'll investigate.

jreadey commented 6 years ago

I tried reloading the Sandy_ocean_his_nc4.nc file. Looks like both the dataset.fillvalue and dataset _FillValue attribute are set to 1.0e37.

@rsignell-usgs - would you try loading the file again? I think the files on jetstream got corrupted at some point.

rsignell-usgs commented 6 years ago

The fill values work as expected with this HSDS "file": /home/rsignell/Sandy_ocean_his_nc4.nc
but get returned as 0.0 with this "file": /home/rsignell/Sandy_ocean_his_nc4c.nc

I created fresh versions of these today using "hsload".

rsignell-usgs commented 6 years ago

I wonder if it's a function of compression?

jreadey commented 1 year ago

I think this should be fixed with the commit in issue #119.
Please re-open if you are still seeing this.