Wrong decoding of MODE S message

[dcesari]

The attached bufr message mode-s.zip, presumably correct, a "MODE S" (new kind of aircraft report), shows decoding errors with wrep:

Value 59776.5 is outside the range [0,409.4] for B11002 (WIND SPEED)

Other messages of the same kind show errors on different descriptors, e.g.

Value -30706 is outside the range [0,62] for 002170 (AIRCRAFT HUMIDITY SENSORS)
When decoding compressed BUFR data, the difference bit length must be 0 (and not 4 like in this case) when the base value is missing
Value 212308 is outside the range [0,2] for 002064 (AIRCRAFT ROLL ANGLE QUALITY)
etc.

I can attach other examples, but I think the source of error is the same. Are they using some non managed modifiers?

N.B. Decoding requires the new bufr tables just committed.

[spanezz]

I dumped the data descriptor section of the MODE S samples. What is the intention of the data representation? They seem to have one measurement at the top, followed by a sequence of different measurements at the bottom:

src/wrep --dds testdata/bufr/mode-s.bufr 
311010 (group)
  001008 AIRCRAFT REGISTRATION NUMBER OR OTHER IDENTIFICATION[CCITTIA5]
  001023 OBSERVATION SEQUENCE NUMBER[NUMERIC]
  001006 AIRCRAFT FLIGHT NUMBER[CCITTIA5]
  001110 AIRCRAFT TAIL NUMBER[CCITTIA5]
  001111 ORIGINATION AIRPORT[CCITTIA5]
  001112 DESTINATION AIRPORT[CCITTIA5]
  204002 2 bits of associated field
  301011 (group)
    004001 YEAR[A]
    004002 MONTH[MON]
    004003 DAY[D]
  301013 (group)
    004004 HOUR[H]
    004005 MINUTE[MIN]
    004006 SECOND[S]
  301021 (group)
    005001 LATITUDE (HIGH ACCURACY)[DEG]
    006001 LONGITUDE (HIGH ACCURACY)[DEG]
  007010 FLIGHT LEVEL[M]
  010053 GLOBAL NAVIGATION SATELLITE SYSTEM ALTITUDE[M]
  008009 DETAILED PHASE OF FLIGHT[CODE TABLE]
  011001 WIND DIRECTION[DEG]
  011002 WIND SPEED[M/S]
  002064 AIRCRAFT ROLL ANGLE QUALITY[CODE TABLE]
  011100 AIRCRAFT TRUE AIRSPEED[M/S]
  011101 AIRCRAFT GROUND SPEED U-COMPONENT[M/S]
  011102 AIRCRAFT GROUND SPEED V-COMPONENT[M/S]
  011103 AIRCRAFT GROUND SPEED W-COMPONENT[M/S]
  011104 TRUE HEADING OF AIRCRAFT, SHIP OR OTHER MOBILE PLATFORM[DEG]
  012101 TEMPERATURE/AIR TEMPERATURE[K]
  002170 AIRCRAFT HUMIDITY SENSORS[CODE TABLE]
  201144 change data width to 16
  202133 change data scale to 5
  013002 MIXING RATIO[KG/KG]
  202000 change data scale to 0
  201000 change data width to 0
  201135 change data width to 7
  202130 change data scale to 2
  013003 RELATIVE HUMIDITY[%]
  202000 change data scale to 0
  201000 change data width to 0
  101000 replicate 1 descriptors (delayed 031000) times
    012103 DEWPOINT TEMPERATURE[K]
  033026 MOISTURE QUALITY[CODE TABLE]
  101000 replicate 1 descriptors (delayed 031000) times
    020042 AIRFRAME ICING PRESENT[CODE TABLE]
  103000 replicate 3 descriptors (delayed 031000) times
    020043 PEAK LIQUID WATER CONTENT[KG M-3]
    020044 AVERAGE LIQUID WATER CONTENT[KG M-3]
    020045 SUPERCOOLED LARGE DROPLET (SLD) CONDITIONS[CODE TABLE]
  101000 replicate 1 descriptors (delayed 031000) times
    033025 ACARS INTERPOLATED VALUES INDICATOR[CODE TABLE]
  103000 replicate 3 descriptors (delayed 031001) times
    011075 MEAN TURBULENCE INTENSITY (EDDY DISSIPATION RATE)[M2/3 S-1]
    011076 PEAK TURBULENCE INTENSITY (EDDY DISSIPATION RATE)[M2/3 S-1]
    011039 EXTENDED TIME OF OCCURRENCE OF PEAK EDDY DISSIPATION RATE[CODE TABLE]
  102000 replicate 2 descriptors (delayed 031000) times
    011037 TURBULENCE INDEX[CODE TABLE]
    011077 REPORTING INTERVAL OR AVERAGING TIME FOR EDDY DISSIPATION RATE[S]
  103000 replicate 3 descriptors (delayed 031000) times
    011034 VERTICAL GUST VELOCITY[M/S]
    011035 VERTICAL GUST ACCELERATION[M S-2]
    011036 MAXIMUM DERIVED EQUIVALENT VERTICAL GUST SPEED[M/S]
  204000 0 bits of associated field
  119000 replicate 19 descriptors (delayed 031001) times
    301011 (group)
      004001 YEAR[A]
      004002 MONTH[MON]
      004003 DAY[D]
    301013 (group)
      004004 HOUR[H]
      004005 MINUTE[MIN]
      004006 SECOND[S]
    301021 (group)
      005001 LATITUDE (HIGH ACCURACY)[DEG]
      006001 LONGITUDE (HIGH ACCURACY)[DEG]
    007007 HEIGHT[M]
    011105 EDR ALGORITHM VERSION[NUMERIC]
    204007 7 bits of associated field
    011076 PEAK TURBULENCE INTENSITY (EDDY DISSIPATION RATE)[M2/3 S-1]
    011075 MEAN TURBULENCE INTENSITY (EDDY DISSIPATION RATE)[M2/3 S-1]
    204000 0 bits of associated field
    011106 RUNNING MINIMUM CONFIDENCE[NUMERIC]
    011107 MAXIMUM NUMBER BAD INPUTS[NUMERIC]
    011108 PEAK LOCATION[NUMERIC]
    011109 NUMBER OF GOOD EDR[NUMERIC]
    012101 TEMPERATURE/AIR TEMPERATURE[K]
    011001 WIND DIRECTION[DEG]
    201130 change data width to 2
    011084 WIND SPEED[KT]
    201000 change data width to 0
025061 SOFTWARE IDENTIFICATION AND VERSION NUMBER[CCITTIA5]
001015 STATION OR SITE NAME[CCITTIA5]
001022 NAME OF FEATURE[CCITTIA5]
001065 ICAO REGION IDENTIFIER[CCITTIA5]
033002 QUALITY INFORMATION[CODE TABLE]

[spanezz]

Posso avere un dump di questo BUFR fatto da un altro tool, per riferimento?

[dcesari]

The command bufr_dump -p (eccodes) gives mode-s-dump.zip. What do you mean by "top" and "bottom" here? As far as I understand these messages collect many reports from different aircrafts taken at the same time or short time interval.

[spanezz]

Thanks for the tip. It seems that it decodes well until 008009 DETAILED PHASE OF FLIGHT[CODE TABLE] (both values and associated fields), and it starts failing from 011001 WIND DIRECTION[DEG]. Looking at the bitstream, I can see the beginning of 011001 but 200 bytes after 008009 instead of right away. I need to investigate more

[spanezz]

I confirm that skipping those 200bits the bitstream resumes to something meaningful. For the next 011002 WIND SPEED[M/S], there are again those extra 200bits to be skipped; same for the later B12101.

I tried hardcoding skipping those:

    if (info->code == WR_VAR(0, 11, 1) || info->code == WR_VAR(0, 11, 2) || info->code == WR_VAR(0, 12, 101))
        skip_bits(200);

and the file decodes properly.

I have no idea what are those 200 bytes. I dumped the three skipped bit sequences and they are these:

1:00000010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000001000000000000
2:00000010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000001000000000000
3:00000010000000000000000000000100000000000000000000000100000000000000000000010000000000000000000000000000000000000000000000000000000000010000000001000000000000000000000000000000000000000000010000000000

The BUFR has 100 subsets, and so I'd say it's 2 bits per subset, which is incidentally the size of the associated field. But then the associated field data is present in the later B11001, B11002, and B12101 data, and was present in the previously encoded values.

They even look like associated field values: windDirection->associatedField in bufr_dump does have only two values set to 1, like in the sequence 1: above. And the B11001 values are encoded with 0 bits of associated field data. But why encode associated field data like that, before the field for associated field base and difference bits that is indeed present later?

I think I would need better documentation of the BUFR format when associated fields are present, or a way to discuss this with other people who have experience with BUFR encoding and decoding. Would it be possible to arrange something?

[spanezz]

Wait, perhaps I found better documentation in MANUAL ON CODES, WMO306_Vol_I.2_2012_en.pdf, FM 94 BUFR, SECTION 4, note 4. Let me try implementing it