search

jswhit / pygrib

Python interface for reading and writing GRIB data
https://jswhit.github.io/pygrib
MIT License
319 stars 93 forks source link

Problem with replacing variable value in pygrib library #227

Open terence0o opened 5 months ago

terence0o commented 5 months ago

Hello, I have been working on a project that involves processing weather data using the Pygrib library in Python. During my implementation, I encountered an issue with replacing a new variable value within a GRIB file using pygrib. I am using the Pygrib library to read FNL data files, and attempting to replace a group of values of the same size into the corresponding positions in the file.I have successfully accomplished the replacement, but I have noticed that the values I replace with have some slight variations. Here is my program：

out_var = f[grid_index] out_var.values = new_data print(out_var.values)

then I discover out_var.values is not equal new_data

I checked the data type, shape, and reviewed the documentation and various online resources, but I am having difficulties understanding the correct approach to replace the variable value.

jswhit commented 5 months ago

are you accounting for the fact that GRIB uses lossy compression (so the data files you write into the file will not be exactly the same as the values you read back)?

terence0o commented 5 months ago

are you accounting for the fact that GRIB uses lossy compression (so the data files you write into the file will not be exactly the same as the values you read back)?

I really appreciate your response. Yes, after replacing certain variables (RH) with partial levels, the data is not exactly consistent with the original. Does this mean that lossy compression is an inherent characteristic of GRIB2, and unavoidable?

stephane-21 commented 1 month ago

Yes the grib2 format is a lossy compression format, I believe that internally it will store your data on a smaller number of bits. Something to investigate is if the number of bits storage size can be modified.