Cannot open NetCDF file

[felixcremer]

When I am trying to open this NetCDF file naively, I get the error below. When I specify the key argument in the Raster call it works as expected, I would hope, that we could give a better error message and mention the possible keys for the data. Also I would have expected a RasterStack to work on that file to give me a collection of all Rasters but that also fails. I can open the data with NCDatasets or YAXArrays

using Rasters
using Downloads: download # Standard Library to handle downloads to get the data

# Now we download the monthly data for the year 1995 to first have a look at the data

url = "http://esgf1.dkrz.de/thredds/fileServer/cosmo-rea/reanalysis/EUR-6km/DWD/ECMWF-ERAINT/REA6/r1i1p1f1/COSMO/v1/mon/atmos/tas/v20230314/tas_EUR-6km_ECMWF-ERAINT_REA6_r1i1p1f1_COSMO_v1_mon_199501-199512.nc"
filename = split(url, "/")[end]
mkpath("data/")
path = download(url, "/tmp/$filename")

Error for plain Raster call

```julia Raster(path) ERROR: NetCDF error: Variable 'vertices' not found in file /tmp/tas_EUR-6km_ECMWF-ERAINT_REA6_r1i1p1f1_COSMO_v1_mon_199501-199512.nc (NetCDF error code: -49) Stacktrace: [1] nc_inq_varid(ncid::Int32, name::String) @ NCDatasets ~/.julia/packages/NCDatasets/vRl1m/src/netcdf_c.jl:1508 [2] _variable @ ~/.julia/packages/NCDatasets/vRl1m/src/variable.jl:72 [inlined] [3] variable @ ~/.julia/packages/NCDatasets/vRl1m/src/variable.jl:83 [inlined] [4] layerkeys(ds::NCDataset{Nothing}) @ RastersNCDatasetsExt ~/.julia/packages/Rasters/9ST4J/ext/RastersNCDatasetsExt/ncdatasets_source.jl:215 [5] Raster(ds::NCDataset{Nothing}, filename::String, key::Nothing; kw::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}) @ RastersNCDatasetsExt ~/.julia/packages/Rasters/9ST4J/ext/RastersNCDatasetsExt/ncdatasets_source.jl:49 [6] Raster(ds::NCDataset{Nothing}, filename::String, key::Nothing) @ RastersNCDatasetsExt ~/.julia/packages/Rasters/9ST4J/ext/RastersNCDatasetsExt/ncdatasets_source.jl:46 [7] (::Rasters.var"#53#54"{Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, String})(ds::NCDataset{Nothing}) @ Rasters ~/.julia/packages/Rasters/9ST4J/src/array.jl:305 [8] _open(f::Rasters.var"#53#54"{Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, String}, ::Type{Rasters.NCDsource}, ds::NCDataset{Nothing}; key::Nothing, kw::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}) @ RastersNCDatasetsExt ~/.julia/packages/Rasters/9ST4J/ext/RastersNCDatasetsExt/ncdatasets_source.jl:261 [9] _open @ ~/.julia/packages/Rasters/9ST4J/ext/RastersNCDatasetsExt/ncdatasets_source.jl:259 [inlined] [10] #39 @ ~/.julia/packages/Rasters/9ST4J/ext/RastersNCDatasetsExt/ncdatasets_source.jl:256 [inlined] [11] NCDataset(::RastersNCDatasetsExt.var"#39#40"{Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, Rasters.var"#53#54"{Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, String}}, ::String, ::Vararg{String}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}) @ NCDatasets ~/.julia/packages/NCDatasets/vRl1m/src/dataset.jl:255 [12] NCDataset(::Function, ::String, ::Vararg{String}) @ NCDatasets ~/.julia/packages/NCDatasets/vRl1m/src/dataset.jl:252 [13] #_open#38 @ ~/.julia/packages/Rasters/9ST4J/ext/RastersNCDatasetsExt/ncdatasets_source.jl:255 [inlined] [14] _open(f::Function, ::Type{Rasters.NCDsource}, filename::String) @ RastersNCDatasetsExt ~/.julia/packages/Rasters/9ST4J/ext/RastersNCDatasetsExt/ncdatasets_source.jl:252 [15] #_open#187 @ ~/.julia/packages/Rasters/9ST4J/src/sources/sources.jl:68 [inlined] [16] _open @ ~/.julia/packages/Rasters/9ST4J/src/sources/sources.jl:67 [inlined] [17] Raster(filename::String; name::Nothing, key::Nothing, source::Nothing, kw::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}) @ Rasters ~/.julia/packages/Rasters/9ST4J/src/array.jl:303 [18] Raster(filename::String) @ Rasters ~/.julia/packages/Rasters/9ST4J/src/array.jl:299 [19] top-level scope @ REPL[84]:1 ```

Error for RasterStack

```julia julia> RasterStack(path) ERROR: NetCDF error: Variable 'vertices' not found in file /tmp/tas_EUR-6km_ECMWF-ERAINT_REA6_r1i1p1f1_COSMO_v1_mon_199501-199512.nc (NetCDF error code: -49) Stacktrace: [1] nc_inq_varid(ncid::Int32, name::String) @ NCDatasets ~/.julia/packages/NCDatasets/vRl1m/src/netcdf_c.jl:1508 [2] _variable @ ~/.julia/packages/NCDatasets/vRl1m/src/variable.jl:72 [inlined] [3] variable @ ~/.julia/packages/NCDatasets/vRl1m/src/variable.jl:83 [inlined] [4] layerkeys(ds::NCDataset{Nothing}) @ RastersNCDatasetsExt ~/.julia/packages/Rasters/9ST4J/ext/RastersNCDatasetsExt/ncdatasets_source.jl:215 [5] layerdims(ds::NCDataset{Nothing}) @ RastersNCDatasetsExt ~/.julia/packages/Rasters/9ST4J/ext/RastersNCDatasetsExt/ncdatasets_source.jl:182 [6] (::Rasters.var"#84#85"{EPSG, EPSG, DataType, Nothing, String})(ds::NCDataset{Nothing}) @ Rasters ~/.julia/packages/Rasters/9ST4J/src/stack.jl:291 [7] #_open#41 @ ~/.julia/packages/Rasters/9ST4J/ext/RastersNCDatasetsExt/ncdatasets_source.jl:261 [inlined] [8] _open @ ~/.julia/packages/Rasters/9ST4J/ext/RastersNCDatasetsExt/ncdatasets_source.jl:259 [inlined] [9] #39 @ ~/.julia/packages/Rasters/9ST4J/ext/RastersNCDatasetsExt/ncdatasets_source.jl:256 [inlined] [10] NCDataset(::RastersNCDatasetsExt.var"#39#40"{Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, Rasters.var"#84#85"{EPSG, EPSG, DataType, Nothing, String}}, ::String, ::Vararg{String}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}) @ NCDatasets ~/.julia/packages/NCDatasets/vRl1m/src/dataset.jl:255 [11] NCDataset(::Function, ::String, ::Vararg{String}) @ NCDatasets ~/.julia/packages/NCDatasets/vRl1m/src/dataset.jl:252 [12] #_open#38 @ ~/.julia/packages/Rasters/9ST4J/ext/RastersNCDatasetsExt/ncdatasets_source.jl:255 [inlined] [13] _open @ ~/.julia/packages/Rasters/9ST4J/ext/RastersNCDatasetsExt/ncdatasets_source.jl:252 [inlined] [14] _open(f::Function, filename::String; source::Type, kw::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}) @ Rasters ~/.julia/packages/Rasters/9ST4J/src/sources/sources.jl:68 [15] _open @ ~/.julia/packages/Rasters/9ST4J/src/sources/sources.jl:67 [inlined] [16] _layer_stack(filename::String; dims::Nothing, refdims::Tuple{}, metadata::Nothing, crs::Nothing, mappedcrs::Nothing, layerdims::Nothing, layermetadata::Nothing, missingval::Nothing, source::Type, name::Nothing, keys::Nothing, resize::Nothing, kw::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}) @ Rasters ~/.julia/packages/Rasters/9ST4J/src/stack.jl:288 [17] _layer_stack @ ~/.julia/packages/Rasters/9ST4J/src/stack.jl:281 [inlined] [18] RasterStack(filename::String; source::Nothing, name::Nothing, keys::Nothing, lazy::Bool, dropband::Bool, kw::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}) @ Rasters ~/.julia/packages/Rasters/9ST4J/src/stack.jl:252 [19] RasterStack(filename::String) @ Rasters ~/.julia/packages/Rasters/9ST4J/src/stack.jl:231 [20] top-level scope @ REPL[85]:1 ```

[rafaqz]

What is vertices

I guess it needs to be skipped somehow.

[felixcremer]

It seems as if vertices is parts of the dimensions and seems to be something similar to "bnds". YAXArrays filters them out as part of the list of dimensions and therefore doesn't have them in a possible layer list. I am going to discuss this further with @meggart.

[rafaqz]

Ok I wasnt aware of those. Maybe link whats on your ignore list in YAX and I'll make sure to include them

[meggart]

This is the code we use in YAXArrays to filter "real" variables https://github.com/JuliaDataCubes/YAXArrays.jl/blob/ee64a8608ed94c399edfebcdca9f188642c1349c/src/DatasetAPI/Datasets.jl#L294-L300:

dnames = string.(keys(dimlist))
varlist = filter(get_varnames(g)) do vn
    upname = uppercase(vn)
    !occursin("BNDS", upname) &&
    !occursin("BOUNDS", upname) &&
    !any(i -> isequal(upname, uppercase(i)), dnames)
end

basically everything that does not have _bnds in its name (should probably use endswith instead of occursin) and which is never listed as a dimension in the dataset will be treated as a data variable

[rafaqz]

Ok thanks that looks a little more thourough than the check for bounds variables here.

[rafaqz]

Hmm this file is gone too, would be good to check this works now...

[felixcremer]

This is still not working. I uploaded a failing dataset here.

tas_EUR-6km_ECMWF-ERAINT_REA6_r1i1p1f1_COSMO_v1_mon_199501-199512.zip

[rafaqz]

Thanks. Now we have all the CommonDataModel changes I want to fix all these little bugs.