ecmwf / fdb

Fdb is a domain-specific object store for meteorological objects
Apache License 2.0
26 stars 11 forks source link

error when running fdb-write #5

Open shambakey1 opened 1 year ago

shambakey1 commented 1 year ago

Hi

Sorry for posting questions here, as I don't know if you have a mailing list, and I'm completely new to FDB.

I have installed fdb5 using apt install fdb5. I copied the schema from test folder into /usr/lib/etc/fdb/schema, and tried to to fdb-write on some free grib files downloaded from https://marine.meteoconsult.co.uk/marine-services/grib-files. I receive the following error:

Processing uEuMsTxTnoSYmtRzKDl0qDApvweRD87zv2F.grb
Exception: SeriousBug: Key::get() failed for [class] in {levtype=sfc,date=20230404,time=0000,step=12,param=3049}  in  (./src/fdb5/database/Key.cc +192 get)
SeriousBug: Key::get() failed for [class] in {levtype=sfc,date=20230404,time=0000,step=12,param=3049}  in  (./src/fdb5/database/Key.cc +192 get)
backtrace [2] stack has 17 addresses
(/usr/bin/../lib/x86_64-linux-gnu/libeckit.so.0d+eckit::BackTrace::dump[abi:cxx11]())0x1eb 
(/usr/bin/../lib/x86_64-linux-gnu/libeckit.so.0d+eckit::SeriousBug::SeriousBug(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, eckit::CodeLocation const&))0x3e2 
(/usr/bin/../lib/x86_64-linux-gnu/libfdb5.so.0d+)0x6a15c 
(/usr/bin/../lib/x86_64-linux-gnu/libfdb5.so.0d+fdb5::Rule::expand(fdb5::Key const&, __gnu_cxx::__normal_iterator<fdb5::Predicate* const*, std::vector<fdb5::Predicate*, std::allocator<fdb5::Predicate*> > >, unsigned long, std::vector<fdb5::Key, std::allocator<fdb5::Key> >&, fdb5::Key&, fdb5::WriteVisitor&) const)0xfc 
(/usr/bin/../lib/x86_64-linux-gnu/libfdb5.so.0d+fdb5::Rule::expand(fdb5::Key const&, fdb5::WriteVisitor&, unsigned long, std::vector<fdb5::Key, std::allocator<fdb5::Key> >&, fdb5::Key&) const)0x3e 
(/usr/bin/../lib/x86_64-linux-gnu/libfdb5.so.0d+fdb5::Schema::expand(fdb5::Key const&, fdb5::WriteVisitor&) const)0xe6 
(/usr/bin/../lib/x86_64-linux-gnu/libfdb5.so.0d+fdb5::Archiver::archive(fdb5::Key const&, fdb5::BaseArchiveVisitor&))0x49 
(/usr/bin/../lib/x86_64-linux-gnu/libfdb5.so.0d+fdb5::Archiver::archive(fdb5::Key const&, void const*, unsigned long))0x48 
(/usr/bin/../lib/x86_64-linux-gnu/libfdb5.so.0d+fdb5::FDB::archive(fdb5::Key const&, void const*, unsigned long))0x51 
(/usr/bin/../lib/x86_64-linux-gnu/libfdb5.so.0d+fdb5::MessageArchiver::archive(eckit::DataHandle&))0x283 
(fdb-write+)0x52b3 
(/usr/bin/../lib/x86_64-linux-gnu/libfdb5.so.0d+fdb5::FDBTool::run())0x13b 
(/usr/bin/../lib/x86_64-linux-gnu/libeckit.so.0d+eckit::Tool::start())0x24 
(fdb-write+)0x3a86 
(/usr/bin/../lib/x86_64-linux-gnu/libc.so.6+)0x29d90 
(/usr/bin/../lib/x86_64-linux-gnu/libc.so.6+__libc_start_main)0x80 
(fdb-write+)0x3c95 

end of backtrace dump ...
fdb::service::archive: 0.425946 second elapsed, 0.014444 second cpu
** SeriousBug: Key::get() failed for [class] in {levtype=sfc,date=20230404,time=0000,step=12,param=3049}  in  (./src/fdb5/database/Key.cc +192 get) Caught in  (./src/eckit/runtime/Tool.cc +32 start)
** Exception terminates fdb-write

I'm not sure if the error is related to the free downloaded data, or the copied schema, or something else?

I wonder if you have an already working example with sample data/schema

Regards

kinow commented 1 year ago

I'm not sure if the error is related to the free downloaded data, or the copied schema, or something else?

I am not too experienced in FDB/GRIB, but I downloaded the data for North Iberian, and then used grib_ls (apt install libeccodes-tools) and the data from that site doesn't have the mars key class:

kinow@ranma:~/Development/cpp/workspace/fdb/tests/regressions/FDB-238$ grib_ls -p class ~/Downloads/METEOCONSULT12Z_VENT_Nord_Iberique_uEuLusExcGwb2.grb | head -n 4
/home/kinow/Downloads/METEOCONSULT12Z_VENT_Nord_Iberique_uEuLusExcGwb2.grb
class       
not_found  
not_found 

I wonder if you have an already working example with sample data/schema

Maybe try the test schema, and one of the regression test files? My understanding is that those files were used to report bugs, which were fixed, and then they are now used as regression-tests to avoid the error being added again. I believe that this must mean they should work with the test schema.

You can confirm the class key exists in those files, e.g.

kinow@ranma:~/Development/cpp/workspace/fdb/tests/regressions/FDB-238$ grib_ls -p class,expver,stream,type,number,centre 12.grib 
12.grib
class       expver      stream      type        number      centre      
rd          xxxx        oper        fc          0           ecmf       
1 of 1 messages in 12.grib

1 of 1 total messages in 1 files

Hope that helps! :+1: