jhardenberg / smmregrid

A compact regridder using sparse matrix multiplication
Apache License 2.0
8 stars 0 forks source link

tempfiles do not live long #9

Open jhardenberg opened 1 year ago

jhardenberg commented 1 year ago

the function cdo_generate_weights uses the tempfile to generate a tempfile weight_file for the weights, which is then read to an xarray with

 weights = xarray.open_dataset(weight_file.name, engine="netcdf4")

The problem is that this tempfile at this point, when exiting the function appears to the system not to be needed anymore, so that it will be erased at the first best occasion (at best at the next garbage collection). If we open it with xarray using open_dataset though, this is a lazy opening and relies on the file still existing.

The solution would be to open the xarray with xarray.load_dataset but this will load everything in memory. Might this be an issue for very large weight files?

oloapinivad commented 1 month ago

there is a delete=False option that can be used, but it will require that the removal is applied in a later stage