CCSDS support not enabled

[lazarusA]

Hi, great package. Unfortunately, I'm hitting the following error for a particular file. How and where I'm suppose to apply the suggestions?

ECCODES ERROR   :  grib_accessor_data_ccsds_packing: CCSDS support not enabled. Please rebuild with -DENABLE_AEC=ON (Adaptive Entropy Coding library)
ECCODES ERROR   :  unable to get values as double array (Functionality not enabled)
ECCODES ERROR   :  Geoiterator factory: Error instantiating iterator gaussian_reduced (Functionality not enabled)
ERROR: Error: Value mismatch
Stacktrace:
 [1] errorcheck
   @ ~/.julia/packages/GRIB/6rlik/src/GRIB.jl:119 [inlined]
 [2] data(handle::GRIB.Message)
   @ GRIB ~/.julia/packages/GRIB/6rlik/src/message.jl:350
 [3] (::GRIBDatasets.var"#27#29"{Vector{String}, DataStructures.DefaultDict{AbstractString, Vector{Any}, GRIBDatasets.var"#26#28"}, Vector{GRIBDatasets.MessageIndex}})(f::GRIB.GribFile)
   @ GRIBDatasets ~/.julia/packages/GRIBDatasets/44bt5/src/index.jl:66
 [4] GRIB.GribFile(f::GRIBDatasets.var"#27#29"{Vector{String}, DataStructures.DefaultDict{AbstractString, Vector{Any}, GRIBDatasets.var"#26#28"}, Vector{GRIBDatasets.MessageIndex}}, filename::String; mode::String)
   @ GRIB ~/.julia/packages/GRIB/6rlik/src/gribfile.jl:52
 [5] GribFile
   @ ~/.julia/packages/GRIB/6rlik/src/gribfile.jl:49 [inlined]
 [6] FileIndex(grib_path::String; index_keys::Vector{String})
   @ GRIBDatasets ~/.julia/packages/GRIBDatasets/44bt5/src/index.jl:62
 [7] FileIndex
   @ ~/.julia/packages/GRIBDatasets/44bt5/src/index.jl:55 [inlined]
 [8] GRIBDataset(filepath::String)
   @ GRIBDatasets ~/.julia/packages/GRIBDatasets/44bt5/src/dataset.jl:79
 [9] top-level scope
   @ ~/Documents/tmp/openGrib.jl:2

[tcarion]

Hi, thanks! GRIBDatasets.jl relies on the eccodes library to decode the GRIB files. Apparently, your file needs eccodes to be built with Adaptive Entropy Coding library (-DENABLE_AEC=ON), which is not the case in the Julia build. I couldn't find the AEC library in Yggdrasil, so that's probably why @barche disabled it. So I guess the only workaround would be to build the AEC library with BinaryBuilder.jl, so eccodes can be built with it. I can try to do it when I have time, but if you have some experience with BinaryBuilder, any help would be welcomed :relaxed:

[lazarusA]

Oh... thanks for taking a look. Unfortunately, I haven't use BinaryBuilder.jl [yet], so at the moment I can only help with some testing.

[jbbremnes]

I have similar problems with our own (MET Norway) grib files, but it seems to work fine with ECMWF IFS grib files (only tested one, though). It would be very useful if the problem somehow can be sorted out, but unfortunately I do not have any experience with the BinaryBuilder.jl either, at least for the moment being.

ECCODES ERROR   :  grib_accessor_data_ccsds_packing: CCSDS support not enabled. Please rebuild with -DENABLE_AEC=ON (Adaptive Entropy Coding library)
ECCODES ERROR   :  unable to get values as double array (Functionality not enabled)
ECCODES ERROR   :  Geoiterator factory: Error instantiating iterator lambert_conformal (Functionality not enabled)

[benelsen]

I added libAEC/CCSDS to the eccodes_jll last month to solve the issue with the new ECMWF cycle grib files and that should just filter through GRIB/GRIBDatasets when doing an update with Pkg.

Can you check if you can read your grib files in a new environment w/o any other dependencies, or maybe with plain GRIB.jl? Maybe something is holding back the update.

If not we might need to update the version of eccodes_jll to the latest one ECMWF offers and drop 32 bit platforms.

[jbbremnes]

Yes, I was indeed using an old version. However, after upgrading to v0.2.1 (and eccodes 2.28) I get another error, please see below. Here is also a selection of the grib messages (257 in total) as listed by grib_ls. I could also provide the grib file if needed.

edition      centre       date         dataType     gridType     stepRange    typeOfLevel  level        shortName    packingType  
2            251          20230813     cf           lambert      10           heightAboveGround  0            t            grid_ccsds  
2            251          20230813     cf           lambert      10           heightAboveGround  0            z            grid_ccsds  
2            251          20230813     cf           lambert      10           heightAboveGround  0            sd           grid_ccsds  
2            251          20230813     cf           lambert      10           isobaricInhPa  850          u            grid_ccsds  
2            251          20230813     cf           lambert      10           isobaricInhPa  925          u            grid_ccsds  
2            251          20230813     cf           lambert      10           isobaricInhPa  1000         u            grid_ccsds  
2            251          20230813     cf           lambert      10           heightAboveGround  80           u            grid_ccsds  
2            251          20230813     cf           lambert      10           heightAboveGround  120          u            grid_ccsds  
2            251          20230813     cf           lambert      0-10         heightAboveGround  0            nswrf        grid_ccsds  
2            251          20230813     cf           lambert      0-10         heightAboveGround  0            nlwrf        grid_ccsds
2            251          20230813     cf           lambert      10           tropopause   0            t            grid_ccsds
2            251          20230813     cf           lambert      10           unknown      0            h            grid_ccsds
2            251          20230813     cf           lambert      10           adiabaticCondensation  0            h            grid_ccsds.
2            251          20230813     cf           lambert      10           entireAtmosphere  0            tcwv         grid_ccsds
2            251          20230813     cf           lambert      10           isothermal   27315        h            grid_ccsds  
2            251          20230813     cf           lambert      10           isothermZero  0            h            grid_ccsds  
2            251          20230813     cf           lambert      10           heightAboveSea  0            pres         grid_ccsds 

julia> ds = GRIBDataset(file)
Dataset: /lustre/storeB/project/metproduction/products/meps/member_00/grib/fp/met/20230813_1200_h_10.grib2
Group: /

Dimensions
   x = 949
   y = 1069
   heightAboveGround = 2
   heightAboveGround_2 = 1
   heightAboveGround_3 = 1
   heightAboveGround_4 = 1
   heightAboveGround_5 = 3
   heightAboveGround_6 = 4
   isobaricInhPa = 13
   valid_time = 1

Variables
Error showing value of type GRIBDataset{Float64, 10}:
ERROR: MethodError: no method matching GRIBDatasets.Variable(::GRIBDataset{Float64, 10}, ::String, ::Tuple{GRIBDatasets.MessageDimension{GRIBDatasets.Horizontal}}, ::Matrix{Float64}, ::Dict{String, Any})

Closest candidates are:
  GRIBDatasets.Variable(::TP, ::String, ::Tuple{Vararg{GRIBDatasets.AbstractDim, N}}, ::TA, ::Dict{String, Any}) where {T, N, TA<:Union{Array{T, N}, DiskArrays.AbstractDiskArray{T, N}}, TP}
   @ GRIBDatasets ~/.julia/packages/GRIBDatasets/Ia2Sr/src/variables.jl:101
  GRIBDatasets.Variable(::GRIBDataset, ::Any)
   @ GRIBDatasets ~/.julia/packages/GRIBDatasets/Ia2Sr/src/variables.jl:129
  GRIBDatasets.Variable(::GRIBDataset, ::GRIBDatasets.AbstractDim)
   @ GRIBDatasets ~/.julia/packages/GRIBDatasets/Ia2Sr/src/variables.jl:163

Stacktrace:
  [1] GRIBDatasets.Variable(ds::GRIBDataset{Float64, 10}, dim::GRIBDatasets.MessageDimension{GRIBDatasets.Horizontal})
    @ GRIBDatasets ~/.julia/packages/GRIBDatasets/Ia2Sr/src/variables.jl:166
  [2] GRIBDatasets.Variable(ds::GRIBDataset{Float64, 10}, key::String)
    @ GRIBDatasets ~/.julia/packages/GRIBDatasets/Ia2Sr/src/variables.jl:134
  [3] cfvariable(ds::GRIBDataset{Float64, 10}, varname::String)
    @ GRIBDatasets ~/.julia/packages/GRIBDatasets/Ia2Sr/src/cfvariables.jl:4
  [4] getindex(ds::GRIBDataset{Float64, 10}, key::String)
    @ GRIBDatasets ~/.julia/packages/GRIBDatasets/Ia2Sr/src/dataset.jl:83
  [5] show(io::IOContext{Base.TTY}, ds::GRIBDataset{Float64, 10})
    @ CommonDataModel ~/.julia/packages/CommonDataModel/RSBF3/src/dataset.jl:95
  [6] show(io::IOContext{Base.TTY}, #unused#::MIME{Symbol("text/plain")}, x::GRIBDataset{Float64, 10})
    @ Base.Multimedia ./multimedia.jl:47
  [7] (::REPL.var"#55#56"{REPL.REPLDisplay{REPL.LineEditREPL}, MIME{Symbol("text/plain")}, Base.RefValue{Any}})(io::Any)
    @ REPL /disk2/software/julia-1.9.0/share/julia/stdlib/v1.9/REPL/src/REPL.jl:276
  [8] with_repl_linfo(f::Any, repl::REPL.LineEditREPL)
    @ REPL /disk2/software/julia-1.9.0/share/julia/stdlib/v1.9/REPL/src/REPL.jl:557
  [9] display(d::REPL.REPLDisplay, mime::MIME{Symbol("text/plain")}, x::Any)
    @ REPL /disk2/software/julia-1.9.0/share/julia/stdlib/v1.9/REPL/src/REPL.jl:262
 [10] display
    @ /disk2/software/julia-1.9.0/share/julia/stdlib/v1.9/REPL/src/REPL.jl:281 [inlined]
 [11] display(x::Any)
    @ Base.Multimedia ./multimedia.jl:340
 [12] #invokelatest#2
    @ ./essentials.jl:816 [inlined]
 [13] invokelatest
    @ ./essentials.jl:813 [inlined]
 [14] print_response(errio::IO, response::Any, show_value::Bool, have_color::Bool, specialdisplay::Union{Nothing, AbstractDisplay})
    @ REPL /disk2/software/julia-1.9.0/share/julia/stdlib/v1.9/REPL/src/REPL.jl:305
 [15] (::REPL.var"#57#58"{REPL.LineEditREPL, Pair{Any, Bool}, Bool, Bool})(io::Any)
    @ REPL /disk2/software/julia-1.9.0/share/julia/stdlib/v1.9/REPL/src/REPL.jl:287
 [16] with_repl_linfo(f::Any, repl::REPL.LineEditREPL)
    @ REPL /disk2/software/julia-1.9.0/share/julia/stdlib/v1.9/REPL/src/REPL.jl:557
 [17] print_response(repl::REPL.AbstractREPL, response::Any, show_value::Bool, have_color::Bool)
    @ REPL /disk2/software/julia-1.9.0/share/julia/stdlib/v1.9/REPL/src/REPL.jl:285
 [18] (::REPL.var"#do_respond#80"{Bool, Bool, REPL.var"#93#103"{REPL.LineEditREPL, REPL.REPLHistoryProvider}, REPL.LineEditREPL, REPL.LineEdit.Prompt})(s::REPL.LineEdit.MIState, buf::Any, ok::Bool)
    @ REPL /disk2/software/julia-1.9.0/share/julia/stdlib/v1.9/REPL/src/REPL.jl:899
 [19] #invokelatest#2
    @ ./essentials.jl:816 [inlined]
 [20] invokelatest
    @ ./essentials.jl:813 [inlined]
 [21] run_interface(terminal::REPL.Terminals.TextTerminal, m::REPL.LineEdit.ModalInterface, s::REPL.LineEdit.MIState)
    @ REPL.LineEdit /disk2/software/julia-1.9.0/share/julia/stdlib/v1.9/REPL/src/LineEdit.jl:2647
 [22] run_frontend(repl::REPL.LineEditREPL, backend::REPL.REPLBackendRef)
    @ REPL /disk2/software/julia-1.9.0/share/julia/stdlib/v1.9/REPL/src/REPL.jl:1300
 [23] (::REPL.var"#62#68"{REPL.LineEditREPL, REPL.REPLBackendRef})()
    @ REPL ./task.jl:514

[jbbremnes]

Using GRIB.jl on the same file seems to work. At least the following example loops through the messages without errors

using GRIB
@time GribFile(file) do f
    for message in f
        out = data(message)
        println(message["parameterName"], " (", message["shortName"], ") at level ",
                message["level"], ": value = ", out[3][1,1])
    end
end 

[tcarion]

Thank you @benelsen for adding libAEC to eccodes!

@jbbremnes, this seems to be a GRIBDatasets bug (unrelated to CCSDS). Could you open another issue, if possible with a link to the file, so I can try to solve it?

I will close this issue as CCSDS is now supported.