search

JuliaGeo / CommonDataModel.jl

CommonDataModel.jl is a module that defines types common to NetCDF and GRIB data
MIT License
11 stars 4 forks source link

Support for the `_Unsigned` attribute? #24

Open asinghvi17 opened 1 month ago

asinghvi17 commented 1 month ago

Describe the bug

Values read by NCDatasets and ZarrDatasets were inconsistent with xarray for this file, because it was interpreted as Int16 data but the attributes had an _Unsigned = true key, so the element type should have been UInt16 instead.

using AWSS3, FilePathsBase
download(p"s3://noaa-goes16/ABI-L2-SSTF/2020/210/00/OR_ABI-L2-SSTF-M6_G16_s20202100000205_e20202100059513_c20202100105456.nc", "noaa.nc")
using Rasters, NCDatasets
Raster("noaa.nc")
# or whichever other way you care to load it

You can see the debugging history in https://github.com/rafaqz/Rasters.jl/issues/735

To Reproduce

Please provide a minimal julia code example which reproduces the behavior (bug, performance regression, ...).

Expected behavior

I would expect that the array is somehow reinterpreted to the appropriate unsigned type.

Environment

Alexander-Barth commented 3 weeks ago

Here is some previous discussion:

https://github.com/Alexander-Barth/NCDatasets.jl/issues/133

In short, this is a lot of work for some feature that never made it to the CF conventions and has been long been superseded by native unsigned types in NetCDF4.

But if somebody would like to make a PR, I am happy to review it. I guess that this can be used similar to other special attributes like scale_factor, units which affect also the element type of an array:

https://github.com/JuliaGeo/CommonDataModel.jl/blob/main/src/cfvariable.jl#L167

And add another function in the "transformation pipeline":

https://github.com/JuliaGeo/CommonDataModel.jl/blob/main/src/cfvariable.jl#L328

(but maybe I will find to do it myself should there be a shortage of volunteers... :-) )