Closed eivindjahren closed 4 months ago
@eivindjahren Thanks for bringing this to attention.
From the description it's a bit unclear which engine you want/need to use. You mentioned engine=netcdf
(should that be netcdf4
?) and in the code example you use engine="scipy"
. From what I can tell engine scipy
uses NETCDF3 data model which has some restrictions on the dimensions of variables. So it understands only 1 dimension as unlimited which need to be the first dimension of the variable.
If you do
$ ncdump file.nc
ncdump: file.nc: NetCDF: NC_UNLIMITED in the wrong index
But if we move the zero dimension to the front before saving:
ds = ds.transpose("name", "index", "time")
This isn't even recognized by ncdump:
$ ncdump file.nc
ncdump: file.nc: NetCDF: Unknown file format
Whereas it can be read perfectly fine with engine="scipy"
.
I did not explore further, but there is something weird going on with engine scipy
here.
From the description it's a bit unclear which engine you want/need to use. You mentioned
engine=netcdf
(should that benetcdf4
?)
Sorry, I ment engine=scipy
, that was a typo. We have decided to use that in our application for performance reasons.
Sorry, I ment
engine=scipy
, that was a typo. We have decided to use that in our application for performance reasons.
A bit offtopic now, but can you elaborate a bit what performance benefits you have with NETCDF3 format in your use case? What is preventing you from using netcdf4 backend?
For the scipy backend issue I'd appreciate if someone with more knowledge in that part could chime in here.
Sorry, I ment
engine=scipy
, that was a typo. We have decided to use that in our application for performance reasons.A bit offtopic now, but can you elaborate a bit what performance benefits you have with NETCDF3 format in your use case? What is preventing you from using netcdf4 backend?
I don't have the specifics about the benchmarks that were performed, but I will see what I can find. We have planned to change to netcdf4 because we want to use datetime[64]
.
Closing for now. If this is still an issue please reopen with updated information. Thanks!
What happened?
Saving and loading from netcdf with engine=scipy produces an unexpected valueerror on read. The file seems to be corrupted.
What did you expect to happen?
reading works just fine.
Minimal Complete Verifiable Example
MVCE confirmation
Relevant log output
Anything else we need to know?
No response
Environment