Error: can't decode messages - IEDX01

[vsouvan]

We are encountering an error while parsing a defined set of files. For every file in the subset below we obtain the following (number can change of course) output:

Message header : "0004593100\001\015\015\012000\015\015\012IEDX01\040EUMP\040260824\015\015\012"

BUFR Edition : 4

length : 45896

Section 0

length : 8

Section 1

length : 22

BUFR master table : 0

originating center : 254

sub center : 0

update sequence number : 0

Data category : 3

International sub category : 255

Local sub category : 223

master table version : 13

local table version : 1

Year : 2014

Month : 9

Day : 26

Hour : 8

Min : 24

Sec : 0

octet 8 : 0

optional section : No

Section 2

length : 0

Section 3

length : 131

datasubsets : 240

octet 7 : 192

compression : Yes

observed data

Section 4

length : 45731

Section 5

length : 4

Error: can't decode messages

On the other side the BUFR/CREX format checker at ECMWF (http://old.ecmwf.int/products/data/d/check) reports no error and decodes correctly the bufr.

Message 1 Section 0 Length of Section 0: 8 byte(s) Total length of BUFR message: 45896 byte(s) BUFR edition number: 4 Section 1 Length of Section 1: 22 byte(s) Originating subcentre: 0 Originating centre: 254 Update sequence number: 0 Flag (Presence of section 2): 0 Local table version number: 1 Data category: 3 Data subcategory: 255 Local data sub-category: 223 Year: 2014 Month: 9 Day: 26 Hour: 8 Minute: 24 Second: 0 BUFR master table: 0 BUFR master table version number: 13 Section 3 Length of Section 3: 131 byte(s) Reserved: 0 Number of data subsets: 240 Data type/data compression: 192 Data descriptors (unexpanded) 1 001007 2 001031 3 002019 4 002020 5 004001 6 004002 7 004003 8 004004 9 004005 10 004006 11 005040 12 201133 13 005041 14 201000 15 005001 16 006001 17 005043 18 013038 19 008012 20 008013 21 002022 22 008065 23 040192 24 040193 25 040194 26 040195 27 040196 28 040197 29 040198 30 007024 31 005021 32 007025 33 005022 34 008003 35 010004 36 008049 37 012061 38 008049 39 202126 40 007001 41 202000 42 008003 43 106000 44 031001 45 202131 46 201138 47 007004 48 201000 49 202000 50 012101 51 110000 52 031001 53 202131 54 201138 55 007004 56 201000 57 202000 58 202129 59 201134 60 013002 61 201000 62 202000 Data descriptors (expanded) 1 001007 SATELLITE IDENTIFIER 2 001031 IDENTIFICATION OF ORIGINATING/GENERATING CENTRE (SEE NOTE 10) 3 002019 SATELLITE INSTRUMENTS 4 002020 SATELLITE CLASSIFICATION 5 004001 YEAR 6 004002 MONTH 7 004003 DAY 8 004004 HOUR 9 004005 MINUTE 10 004006 SECOND 11 005040 ORBIT NUMBER 12 005041 SCAN LINE NUMBER 13 005001 LATITUDE (HIGH ACCURACY) 14 006001 LONGITUDE (HIGH ACCURACY) 15 005043 FIELD OF VIEW NUMBER 16 013038 SUPERADIABATIC INDICATOR

[...]

Also the utilities that comes with Geo::BUFR Perl module decodes the file correctly.

If need we can provide some example files.

Names of the files for which we encouter the problem:

W_XX-EUMETSAT-Darmstadt,SOUNDING+SATELLITE,METOPA+IASI_CEUMC??????????????_?????_eps_o_twt_l2.bin W_XX-EUMETSAT-Darmstadt,SOUNDING+SATELLITE,METOPB+IASI_CEUMP??????????????_?????_eps_o_twt_l2.bin W_XX-EUMETSAT-Darmstadt,SOUNDING+SATELLITE,METOPA+IASI_CEUMP??????????????_?????_eps_o_clp_l2.bin W_XX-EUMETSAT-Darmstadt,SOUNDING+SATELLITE,METOPA+IASI_CEUMC??????????????_?????_eps_o_ozo_l2.bin W_XX-EUMETSAT-Darmstadt,SOUNDING+SATELLITE,METOPB+IASI_CEUMP??????????????_?????_eps_o_ozo_l2.bin W_XX-EUMETSAT-Darmstadt,SOUNDING+SATELLITE,METOPA+IASI_CEUMC??????????????_?????_eps_o_ems_l2.bin W_XX-EUMETSAT-Darmstadt,SOUNDING+SATELLITE,METOPB+IASI_CEUMP??????????????_?????_eps_o_ems_l2.bin W_XX-EUMETSAT-Darmstadt,SOUNDING+SATELLITE,METOPA+IASI_CEUMC??????????????_?????_eps_o_trg_l2.bin W_XX-EUMETSAT-Darmstadt,SOUNDING+SATELLITE,METOPB+IASI_CEUMP??????????????_?????_eps_o_trg_l2.bin W_XX-EUMETSAT-Darmstadt,SURFACE+SATELLITE,SARAL+OGDR_CEUMS??????????????T?????????????????????.bin W_C_ISFX04EGRR??????????????_UK-MetOffice-Sferics-Africa-BUFR-15minute_concatenation.bin S-O3M_GOME_O3-NO2L2??????????????_001METOPA?????_DLR_03.BUFR

---

Imported from Launchpad using lp2gh.

• date created: 2014-10-31T13:31:54Z
• owner: sdk
• the launchpad url was https://bugs.launchpad.net/bugs/1388088

[vsouvan]

(by chris-beauregard) Yes, please attach at least a couple of the files it's having trouble with.

[vsouvan]

(by sdk) Attachment of the example file with the problem

[vsouvan]

(by chris-beauregard) What BUFR tables are you using to decode it?

The default tables with LibECBUFR don't know about local descriptors.

When you bufr_decoder, check for a DEBUG.decoder file in the local directory. In my case, I'm seeing:

cpb@delta:~/bufr$ more DEBUG.decoder Descriptor 40192 ?? Error: unknown descriptor 40192 Descriptor 40193 ?? Error: unknown descriptor 40193 Descriptor 40194 ?? Error: unknown descriptor 40194 Descriptor 40195 ?? Error: unknown descriptor 40195 Descriptor 40196 ?? Error: unknown descriptor 40196 Descriptor 40197 ?? Error: unknown descriptor 40197 Descriptor 40198 ?? Error: unknown descriptor 40198 Error: Template definition contains error(s) Error: Unable to create Template Error: can't decode messages

Obviously, the stock code tables available to ECMWF and Geo::BUFR know about these local descriptors.

[vsouvan]

(by sdk) You are right, we are using the stock tables for each of the products, so I think the problem can be connected to this missing descriptor. I'd been probably misleading by the question https://answers.launchpad.net/libecbufr/+question/240912 in thinking that was only an "wait for an update problem" sure we were not able to understand it from the screen output. (but our fault because we didn't check the debug file)

Do you know if there's a way or a tool to import their table for use together with libecubufr?

[vsouvan]

(by chris-beauregard)

Do you know if there's a way or a tool to import their table for use together with libecubufr?

Not that I'm aware of. I've written code to handle the ECMWF tables from LibECBUFR as a proof-of-concept (see the Examples/foreign_table.c in the source) and it wouldn't be difficult to incorporate into the decoder using, say, a bufr_load_ecmwf_tables() function, but generally speaking we've elected to leave the details of table management outside of the LibECBUFR core.

The master-table-version-checking blueprint sheds a bit more light on the whole issue, and Yves could probably shed even more if that doesn't scare you away.

[vsouvan]

(by chris-beauregard) Indeed... in a pinch, the foreign_table.c example is a usable, is plain, BUFR decoder to use with ECMWF tables. I.e.

cpb@delta:~/src/libecbufr/Examples$ ./foreign_table -tableb B0000000000098013001.TXT -tableb B0000000000254010001.TXT -tabled D0000000000098013001.TXT ~/bufr/W_XX-EUMETSAT-Darmstadt\,SOUNDING+SATELLITE\,METOPB+IASI_C_EUMP_20140926082400_10494_eps_o_twt_l2.bin cpb@delta:~/src/libecbufr/Examples$ more OUTPUT.TXT BUFR_EDITION=4 HEADER_STRING="0004593100\001\015\015\012000\015\015\012IEDX01\040EUMP\040260824 \015\015\012" BUFR_MASTER_TABLE=0 ORIG_CENTER=254 ORIG_SUB_CENTER=0 UPDATE_SEQUENCE=0 DATA_CATEGORY=3 INTERN_SUB_CATEGORY=255 LOCAL_SUB_CATEGORY=223 MASTER_TABLE_VERSION=13 LOCAL_TABLE_VERSION=1 YEAR=2014 MONTH=9 DAY=26 HOUR=8 MINUTE=24 SECOND=0 DATA_FLAG=192 COMPRESSED=1 DATASUBSET 1 : 1486 codes 001007 3 001031 254 --More--(0%)

[vsouvan]

(by chris-beauregard) As you can see, with the correct tables LibECBUFR handles the message perfectly fine. As such (and no offense intended), I've changed the bug status to Invalid. Whether we should directly handle ECMWF tables is another topic entirely which is worthwhile debating in a separate feature request, I think.