Datasets with incorrect values read

nmangue / NGrib

NGrib is a .NET library to read GRIB (GRid in Binary) files. GRIB is a gridded data standard from WMO (World Meteorological Organisation) and is used by many meteorological organisation. Fork of GribCS.

GNU Lesser General Public License v3.0

17 stars 10 forks source link

Datasets with incorrect values read #23

Open mabianKT opened 2 months ago

mabianKT commented 2 months ago

We found some grib files with data that cannot be read correctly by NGrib.

I attached a zip file with a grib file in which three datasets are decoded incorrectly. For convenience, I also attached csv files for the three datasets with details of values as read by:

wgrib2 output (used as "true" reference)
Grib API library https://github.com/GribApiDotNet/GribApi.NET - It is able to read the values but we cannot use it due to Windows runtime dependencies
NGrib 0.10 (we also tried 0.11 and 0.12 pre but the results are the same)

In particular, it seems that some "missing values" are not handled properly, but not missing values are wrong too.

Is it possible to take a look? Thanks!

grib2_ngrib_Readissues.zip

Lodeli commented 2 months ago

Out of 9 messages in the previous GRIB file, I've extracted the 3rd message. The library appears to correctly extract the information from the Indicator Section, but the values are all significantly higher than expected. The error increases as the values are further from the beginning of the message, which leads me to suspect it might be related to the complex packing compression.

message3.zip

nmangue commented 1 month ago

I have taken a look at the issue. The problem effectively lies in the Complex packing and spatial differencing Grid Data Representation. Unfortunately I haven't work on this part of the implementation is from the original implementation and I don't master the 5.3 template implementation. I was not able to pin down the issue.

So I have use the eccodes implementation to completely rewrite the unpack method. I have add a unit test with the data extract and it gave good result.

Could you run more test on your side with the branch fix/issue-23 to confirm that the decoding is ok ?

mabianKT commented 1 month ago

Thank you very much, we are going to check very soon and revert with our findings :)

Lodeli commented 1 month ago

I've been checking the fix and I can confirm that with this same file the values seem now correct. There are still some differences from GribAPI values but they're now in the order of floating point rounding.