Unidata / netcdf4-python

netcdf4-python: python/numpy interface to the netCDF C library
http://unidata.github.io/netcdf4-python
MIT License
736 stars 259 forks source link

Bug: quantize_mode="GranularBitRound" replaces nans by "-0". #1314

Open garciampred opened 3 months ago

garciampred commented 3 months ago

This is likely an upstream bug, but I don't know how to reproduce it with netcdf-c. BitRound and BitGroom are not affected.

import numpy as np
import netCDF4 as ncdf
data = np.array([5.3, 6.2, 7.3, np.nan])
nc = ncdf.Dataset("test.nc", "w")
nc = ncdf.Dataset("test.nc", "w")
nc.createDimension("x", 4)
v = nc.createVariable("tas", "float32", "x", shuffle=True, compression="zlib", significant_digits=4, quantize_mode="GranularBitRound")
v[:] = data
nc.close

Then ncdump returns

netcdf test {
dimensions:
    x = 4 ;
variables:
    float tas(x) ;
        tas:_QuantizeGranularBitRoundNumberOfSignificantDigits = 4 ;
        tas:_Storage = "chunked" ;
        tas:_ChunkSizes = 4 ;
        tas:_DeflateLevel = 4 ;
        tas:_Shuffle = "true" ;
        tas:_Endianness = "little" ;

// global attributes:
        :_NCProperties = "version=2,netcdf=4.9.2,hdf5=1.14.3" ;
        :_SuperblockVersion = 2 ;
        :_IsNetcdf4 = 1 ;
        :_Format = "netCDF-4" ;
data:

 tas = 5.299805, 6.200195, 7.299805, -0 ;
}
``

I am using the following versions 

hdf5 1.14.3 nompi_h4f84152_100 conda-forge libnetcdf 4.9.2 nompi_h9612171_113 conda-forge netcdf4 1.6.5 nompi_py312h26027e0_100 conda-forge

jswhit commented 3 months ago

not sure you should expect the quantization to leave nans alone, unless the _FillValue or missing_value is set to nan

jswhit2 commented 3 months ago

it turns out that even if the _FillValue is set to nan, the same problem occurs.

however, you can use

data = np.ma.masked_invalid(np.array([5.3, 6.2, 7.3, np.nan]))

and you end up with

netcdf test {
dimensions:
    x = 4 ;
variables:
    float tas(x) ;
        tas:_QuantizeGranularBitRoundNumberOfSignificantDigits = 4 ;
data:

 tas = 5.299805, 6.200195, 7.299805, _ ;
}
jswhit commented 3 months ago

did you report this upstream?