jhardenberg / smmregrid

A compact regridder using sparse matrix multiplication
Apache License 2.0
8 stars 0 forks source link

Devel/extend #1

Closed oloapinivad closed 1 year ago

oloapinivad commented 1 year ago

This branch addresses a few issues of smmregrid so far

oloapinivad commented 1 year ago

I was able to run it on both DataArray and Dataset, create a series of tests for multiple grids and find a draft way to deal with masks. Pressure levels seems supported by default.

oloapinivad commented 1 year ago

I had a bit of fights with the different versions, but tests are now in place from 3.7 to 3.10. Environment creation is now safe.

oloapinivad commented 1 year ago

Last commit introduced a few fixes to work with Datasets, especially with masks andtime_bnds. However, there is slowdown in vertical levels that should be investigated.

jhardenberg commented 1 year ago

Hi, what is the "remove dobule mask computation" commit? Did you find a way to avoid computing the mask itself or was it duplicated?

oloapinivad commented 1 year ago

Ahaha no I did it twice :-)

oloapinivad commented 1 year ago

Last commit includes a new treatment of the masks - computed once when the weights are loaded and initialised - and significantly reduces the overhead when working with xarray.Dataset() to about 10%. Speedup figures are discussed in #2, and mostly suggest that with a full loading of the files the speed up compared to CDO is about ~1.5-3x. Slower performance are achieved for very big files. However, the good news is that with the current configuration it seems that writing to the disk is not a bottleneck, so that we are faster than CDO also in this case.

I will consider this as done and ask for review from @jhardenberg We still need to figure out how to better exploit dask for parallelisation.

oloapinivad commented 1 year ago

Last commit introduces a very small fix which allows for automatically transferring attributes from the original dataset to the regridded one.

oloapinivad commented 1 year ago

Proceeding to the merge!