search

ywangd / pybufrkit

Pure Python toolkit to work with WMO BUFR messages
http://pybufrkit.readthedocs.io/
MIT License
72 stars 28 forks source link

BUFR file with all date values off-by-one #33

Closed akrherz closed 6 months ago

akrherz commented 6 months ago

Thanks for the work on this excellent library. I have been having great success with processing surface observations transmitted in near realtime in BUFR format. I have been doing this for a couple of months now and occasionally hit an oddity with a random BUFR file seemingly having a timestamp that is one-off from what the current time is. An example is this:

example.zip

pybufrkit appears to decode this as a timestamp of 2025-02-26 19:01+00, which is strangely +1 in all year, month, day, hour, and minute from the current timestamp. I am unsure of a means to fully debug this outside of pybufrkit and I could totally see this just being an encoding issue with the upstream generator, but figured I would drop an example here to see, if by chance, there is something else going on!

Thank you.

ywangd commented 6 months ago

Thanks for reporting this issue. It turned out to be an bug in decoding compressed data. The issue is that an one-bit increment of value one should be considered as missing data instead of an normal increment. Quote from WMO specs

94.1.6 The convention for representing missing data for compressed data within the binary Data section shall be to set the corresponding increments to fields of all ones.

Previously the code had a special logic for 1-bit increment. Apperantly this is not what the specs says. I have now pushed a fix (be91cb6) and published a new version 0.2.22 to PyPI.

akrherz commented 6 months ago

Oh wow, thank you. I was very worried that this was a goose chase! I've pushed a conda-forge version update now as well.