search

JuliaDataCubes / YAXArrays.jl

Yet Another XArray-like Julia package
https://juliadatacubes.github.io/YAXArrays.jl/
Other
89 stars 12 forks source link

Netcdf _FillValue and scale_factor not applied properly when using open_dataset() #396

Open dpabon opened 1 month ago

dpabon commented 1 month ago

In the following netCDF: MYD21C3_2017_cut_issue.zip

when I try:

using YAXArraysToolbox
using NetCDF
using YAXArrays
using DimensionalData
lst_monthly_high = open_dataset("/home/dpabon/Documents/MYD21C3_2017_cut_issue.nc") # years 2002 to 2011
lst_monthly_high = Cube(lst_monthly_high[["__xarray_dataarray_variable__"]])
lst_monthly_high

I got:

╭───────────────────────────────────╮
│ 7200×3600×1×12 YAXArray{UInt16,4} │
├───────────────────────────────────┴────────────────────────────────────────────────────────────── dims ┐
  ↓ lon  Sampled{Float64} -179.975:0.05000000000000001:179.97500000000005 ForwardOrdered Regular Points,
  → lat  Sampled{Float64} 89.975:-0.05:-89.97500000000002 ReverseOrdered Regular Points,
  ↗ band Sampled{Int64} 1:1 ForwardOrdered Regular Points,
  ⬔ Ti   Sampled{DateTime} [2017-01-01T00:00:00, …, 2017-12-01T00:00:00] ForwardOrdered Irregular Points
├────────────────────────────────────────────────────────────────────────────────────────────── metadata ┤
  Dict{String, Any} with 6 entries:
  "AREA_OR_POINT" => "Area"
  "name"          => "__xarray_dataarray_variable__"
  "add_offset"    => 0.0
  "scale_factor"  => 1.0
  "grid_mapping"  => "spatial_ref"
  "_FillValue"    => 0x0000
├───────────────────────────────────────────────────────────────────────────────────────────── file size ┤ 
  file size: 593.26 MB
└────────────────────────────────────────────────────────────────────────────────────────────────────────┘

here "_FillValue" => 0x0000

However when I check the data:

collect(lst_monthly_high.data)

7200×3600×1×12 Array{UInt16, 4}:
[:, :, 1, 1] =
 0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  …  0x29e8  0x296b  0x2b25  0x28b2  0x29fe  0x29bf  0x2aa5  0x295d  0x29fd  0x28b4  0x0000
 0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000     0x297b  0x2975  0x29ad  0x29ef  0x2a63  0x2c07  0x0000  0x2895  0x2a9d  0x0000  0x0000
 0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000     0x2a41  0x2a2d  0x2977  0x2891  0x2931  0x2a2d  0x0000  0x0000  0x29d2  0x0000  0x0000
 0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000     0x2948  0x29dd  0x0000  0x2a45  0x0000  0x299d  0x2b69  0x0000  0x28b3  0x0000  0x2892
 0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000     0x2a17  0x0000  0x2a1d  0x295e  0x0000  0x2a71  0x0000  0x0000  0x0000  0x0000  0x0000
 0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  …  0x28b5  0x2987  0x2951  0x299f  0x2995  0x2a84  0x28cd  0x2991  0x0000  0x0000  0x0000
 0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000     0x29ee  0x2976  0x29c4  0x28ec  0x2a0f  0x290e  0x29f7  0x0000  0x0000  0x28ad  0x0000
 0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000     0x2974  0x293e  0x29eb  0x2932  0x0000  0x29fe  0x2943  0x2967  0x2844  0x0000  0x0000
 0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000     0x29b4  0x2888  0x28c4  0x29ff  0x0000  0x2c00  0x2a0c  0x2a25  0x2a8c  0x0000  0x0000
 0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000     0x28ed  0x2a00  0x0000  0x29c1  0x2a30  0x0000  0x0000  0x2a46  0x0000  0x0000  0x0000
 0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  …  0x28cf  0x0000  0x28c9  0x29d7  0x2abb  0x295c  0x2a5b  0x0000  0x2b6e  0x0000  0x0000
 0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000     0x29e9  0x2a65  0x299c  0x0000  0x29cc  0x286c  0x0000  0x295c  0x29ca  0x0000  0x0000

I got 0x0000 instead of NaN or missing values. Most likely because in Julia missing and NaN cannot be Int. But also because the scale factor is not applied.