search

JuliaClimate / ClimateBase.jl

Tools to analyze and manipulate climate (spatiotemporal) data. Also used by ClimateTools and ClimatePlots
https://juliaclimate.github.io/ClimateBase.jl/dev/
Other
39 stars 3 forks source link

ERA5 ncwrite fails #61

Open Balinus opened 3 years ago

Balinus commented 3 years ago

Tried to save a ClimArray containing ERA5 tasmax data and it failed due to a potential wrong _FillValue.

I can provide the file, but it is ~750MB. I do not remember where and when I downloaded it.

julia> fileERA5 = "/media/proy/Data/DATA/ERA5_tasmax.nc"
julia> A = ncread(fileERA5, "t2m")
julia> A
ClimArray (named t2m) with dimensions:
 Longitude (type Lon): 0.0f0:0.25f0:359.75f0 (Sampled: Ordered Regular Points)
 Latitude (type Lat): 90.0f0:-0.25f0:-90.0f0 (Sampled: Ordered Regular Points)
 Time (type Ti): DateTime[2000-01-01T00:00:00, 2000-01-02T00:00:00, …, 2000-12-30T00:00:00, 2000-12-31T00:00:00] (Sampled: Ordered Irregular Points)
attributes: Dict{String, Any} with 6 entries:
  "missing_value" => -32767
  "units"         => "K"
  "add_offset"    => 256.979
  "long_name"     => "2 metre temperature"
  "scale_factor"  => 0.00192051
  "_FillValue"    => -32767
and data: 1440×721×366 Array{Float64, 3}
[:, :, 1]
 264.693  264.063  263.793  263.725  263.631  263.608  263.585  263.608  263.821  …  244.088  244.011  244.009  243.971  243.958  243.84   243.721  243.441
 264.693  264.063  263.793  263.725  263.631  263.612  263.595  263.624  263.829     244.079  244.006  244.006  243.969  243.956  243.84   243.721  243.441
 264.693  264.063  263.793  263.725  263.633  263.616  263.604  263.639  263.837     244.071  244.0    244.002  243.969  243.956  243.84   243.721  243.441
 264.693  264.063  263.793  263.727  263.633  263.618  263.614  263.656  263.844     244.061  243.994  243.998  243.967  243.956  243.838  243.721  243.441
   ⋮                                            ⋮                                 ⋱                      ⋮                                            ⋮
 264.693  264.063  263.798  263.737  263.654  263.608  263.562  263.572  263.766     244.107  244.032  244.023  243.977  243.961  243.842  243.721  243.441
 264.693  264.063  263.796  263.733  263.645  263.608  263.57   263.583  263.785     244.102  244.027  244.019  243.975  243.959  243.842  243.721  243.441
 264.693  264.063  263.795  263.729  263.637  263.608  263.577  263.597  263.804     244.094  244.019  244.015  243.973  243.958  243.84   243.721  243.441
[and 365 more slices...]

julia > ncwrite("testClimateBase.nc", A)
processing variable t2m...
writing dimension lon...
writing dimension lat...
writing dimension time...
writing the CF-variable...
ERROR: NetCDF error: NetCDF: Not a valid data type or _FillValue type mismatch (NetCDF error code: -45)
Stacktrace:
  [1] check
    @ ~/.julia/packages/NCDatasets/eDnxY/src/errorhandling.jl:33 [inlined]
  [2] nc_put_att(ncid::Int32, varid::Int32, name::String, data::Int16)
    @ NCDatasets ~/.julia/packages/NCDatasets/eDnxY/src/netcdf_c.jl:521
  [3] setindex!(a::NCDatasets.Attributes{NCDataset{Nothing}}, data::Int16, name::String)
    @ NCDatasets ~/.julia/packages/NCDatasets/eDnxY/src/attributes.jl:57
  [4] defVar(ds::NCDataset{Nothing}, name::String, vtype::DataType, dimnames::Tuple{String, String, String}; kwargs::Base.Iterators.Pairs{Symbol, Vector{Pair{String, Any}}, Tuple{Symbol}, NamedTuple{(:attrib,), Tuple{Vector{Pair{String, Any}}}}})
    @ NCDatasets ~/.julia/packages/NCDatasets/eDnxY/src/cfvariable.jl:158
  [5] _defVar(ds::NCDataset{Nothing}, name::String, data::Array{Float64, 3}, nctype::Type, dimnames::Tuple{String, String, String}; attrib::Dict{String, Any}, kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ NCDatasets ~/.julia/packages/NCDatasets/eDnxY/src/cfvariable.jl:234
  [6] #defVar#40
    @ ~/.julia/packages/NCDatasets/eDnxY/src/cfvariable.jl:187 [inlined]
  [7] ncwrite(file::String, Xs::Tuple{ClimArray{Float64, 3, Tuple{Lon{StepRangeLen{Float32, Float64, Float64}, DimensionalData.Sampled{DimensionalData.Ordered{DimensionalData.ForwardIndex, DimensionalData.ForwardArray, DimensionalData.ForwardRelation}, DimensionalData.Regular{Float32}, DimensionalData.Points}, Dict{String, String}}, Lat{StepRangeLen{Float32, Float64, Float64}, DimensionalData.Sampled{DimensionalData.Ordered{DimensionalData.ReverseIndex, DimensionalData.ForwardArray, DimensionalData.ForwardRelation}, DimensionalData.Regular{Float32}, DimensionalData.Points}, Dict{String, String}}, Ti{Vector{DateTime}, DimensionalData.Sampled{DimensionalData.Ordered{DimensionalData.ForwardIndex, DimensionalData.ForwardArray, DimensionalData.ForwardRelation}, DimensionalData.Irregular{DimensionalData.AutoBounds}, DimensionalData.Points}, Dict{String, String}}}, Tuple{}, Array{Float64, 3}, Dict{String, Any}}}; globalattr::Dict{Any, Any})
    @ ClimateBase ~/.julia/packages/ClimateBase/klYxO/src/core/nc_io.jl:348
  [8] #ncwrite#34
    @ ~/.julia/packages/ClimateBase/klYxO/src/core/nc_io.jl:319 [inlined]
  [9] ncwrite(file::String, X::ClimArray{Float64, 3, Tuple{Lon{StepRangeLen{Float32, Float64, Float64}, DimensionalData.Sampled{DimensionalData.Ordered{DimensionalData.ForwardIndex, DimensionalData.ForwardArray, DimensionalData.ForwardRelation}, DimensionalData.Regular{Float32}, DimensionalData.Points}, Dict{String, String}}, Lat{StepRangeLen{Float32, Float64, Float64}, DimensionalData.Sampled{DimensionalData.Ordered{DimensionalData.ReverseIndex, DimensionalData.ForwardArray, DimensionalData.ForwardRelation}, DimensionalData.Regular{Float32}, DimensionalData.Points}, Dict{String, String}}, Ti{Vector{DateTime}, DimensionalData.Sampled{DimensionalData.Ordered{DimensionalData.ForwardIndex, DimensionalData.ForwardArray, DimensionalData.ForwardRelation}, DimensionalData.Irregular{DimensionalData.AutoBounds}, DimensionalData.Points}, Dict{String, String}}}, Tuple{}, Array{Float64, 3}, Dict{String, Any}})
    @ ClimateBase ~/.julia/packages/ClimateBase/klYxO/src/core/nc_io.jl:319
 [10] top-level scope
    @ REPL[113]:1

pkg> st
      Status `~/Project.toml`
  [35604d93] ClimateBase v0.12.3
  [85f8d34a] NCDatasets v0.11.3
Datseris commented 3 years ago

Yeah, the type of the Fill Value is Integer, while the type of the data is Float.

Should we check and autoconvert the type of fill value to the type of the array?

Datseris commented 3 years ago

(notice: this is not our problem per se. It's a problem of NCDatasets.jl that doesn't provide the convenience of automatic conversion)

Balinus commented 3 years ago

ah I see! I guess we could add an optional parameter to ncwrite?

wulan87 commented 3 years ago

Hi! I got the same problem here. Is there are any ways to solve this?

Datseris commented 3 years ago

Yeap, the solution is quite trivial. Either do a PR here and modify the function ncwrite to automatically convert the _FillValue argument to the element type of the data. Or, do a similar PR at NCDatasets.jl that does the same thing in the function NCDatasets.write.

Or just do this on your end. Just convert the _FillValue field of the metadata dictionary to be float instead of integer.