Closed MRibberink closed 12 months ago
Thanks for opening your first issue here at xarray! Be sure to follow the issue template! If you have an idea for a solution, we would really welcome a Pull Request with proposed changes. See the Contributing Guide for more. It may take us a while to respond here, but we really value your contribution. Contributors like you help make xarray better. Thank you!
@MRibberink Thanks for the report.
Most likely issue is different cf-coding attributes on that variable (maybe packed-data, scale_factor
/add_offset
).
You can check this with:
print(Lor1.encoding, Lor1["msl"].encoding)
print(Lor2.encoding, Lor2["msl"].encoding)
During concatenating only the encoding-dict of the first object survives (as you already assessed), leading to a wrong/different encoding into netcdf.
If different cf-coding attributes is the issue, you have several options:
encoding
-kwarg in to_netcdf
for the variable in question.It looks like my first guess was correct, we have packed data for the msl
variable. Thanks for providing the data @MRibberink.
short msl(time, latitude, longitude) ;
msl:scale_factor = 0.0755888254772405 ;
msl:add_offset = 100995.993455587 ;
msl:_FillValue = -32767s ;
msl:missing_value = -32767s ;
msl:units = "Pa" ;
msl:long_name = "Mean sea level pressure" ;
msl:standard_name = "air_pressure_at_mean_sea_level" ;
To answer your question: Adding 0.0
to the variable turns it into float and the encoding-dict will be cleared. This will in turn write floating point data to netcdf, which might result in a somewhat larger file.
Please see also the discussion on future of encoding-dict here: https://github.com/pydata/xarray/issues/6323
Wonderful. Thank you so much!
@MRibberink You are very welcome. So assuming you've got the answers you need, we can close this. Please do not hesitate to open follow-up issues whenever necessary.
What happened?
I'm using xarray to concatenate several netcdf files in time (ERA5 mean sea level data that needs to be downloaded in separate months), but because I have a lot of them I save the concatenated files to netcdf after using them so I can later do my data analysis in a loop rather than each by hand. Doing so unfortunately seems to set a min/max on the whole timeseries based on the first file, though this doesn't exist when the data is plotted before being processed by to_netcdf.
Data can be found here: https://www.dropbox.com/sh/xb1c2018ey83poc/AABDpQnjbWcGdtq_MwJeEEc-a?dl=0
The code below generates several diagnostic images, primary among them showing the problem is this one:![min_pres_error](https://github.com/pydata/xarray/assets/61438236/9299bd57-73fb-4a2b-8acf-9bfd96e8ec11)
Where the blue line is just the concatenated original datasets, and the orange line is that data saved by to_netcdf and then called up again. The first file ends at the end of August.
The same problem causes weird phenomena when looking at the sea level pressure fields, as shown in the other created plots. The first is how it should look (just concatted together), but the second is what it actually looks like (to_netcdf). Here's how they look on my end:
![image](https://github.com/pydata/xarray/assets/61438236/e9dec888-a5ba-46c5-a5f9-e64b191aa36e)
I also included a cross-section through the phenomenon off the coast of Canada to show what it's doing when it encounters values below the minimum - it brings them so the first value below is at the level of the maximum defined earlier:![image](https://github.com/pydata/xarray/assets/61438236/904bc998-a94d-48a3-bdfc-2f7de5ef86fd)
I found a weird workaround, the problem is fixed by adding 0 to the dataset before sending it to to_netcdf, but this still shouldn't be happening (and I have NO idea why that fixes it).
What did you expect to happen?
I expected to_netcdf to accurately save the dataset the way it is, rather than setting a max/min based on the first file. In essence, to reproduce the minimum pressure line that was plotted before saving and reloading the data.
Minimal Complete Verifiable Example
MVCE confirmation
Relevant log output
No response
Anything else we need to know?
No response
Environment