oloapinivad
commented
1 year ago Commit b08b045f6fbb6cf63d439676b207e702d182181a establish a good starting point:
| NVars | NRecords | CDO | SMM (Dataset) | SMM (DataArray) | SMM (DataArray+NoMask) | SMM (Dataset+Write) | SMM (DataArray+Write) | |
|---|---|---|---|---|---|---|---|---|
| onlytos-ipsl.nc | 1 | (12, 332, 362) | 1 | 0.216799 | 0.0778903 | 0.0504366 | 0.997036 | 0.77708 |
| tas-ecearth.nc | 1 | (12, 256, 512) | 1 | 0.226347 | 0.0900398 | 0.0631857 | 1.14144 | 0.958676 |
| 2t-era5.nc | 1 | (12, 73, 144) | 1 | 0.170659 | 0.094557 | 0.0610341 | 0.845712 | 0.765937 |
| tos-fesom.nc | 1 | (12, 126859) | 1 | 0.113976 | 0.0399258 | 0.0256824 | 0.755877 | 0.623671 |
| ua-ecearth.nc | 1 | (2, 19, 256, 512) | 1 | 0.398825 | 0.0677817 | 0.0441846 | 1.61761 | 1.35922 |
| mix-cesm.nc | 4 | (12, 192, 288) | 1 | 0.549228 | 0.0688021 | 0.0452126 | 1.73817 | 0.670783 |
| era5-mon.nc | 1 | (864, 721, 1440) | 1 | 0.825034 | 0.000651573 | 0.000439713 | 1.883 | 1.0852 |
Few points:
- There is still too much overhead in the Dataset call. This is likely due to some safecheck we are using for the masks.
- Masks count for about 50% more computational increase. So far it is unsafe to run without masks.
- Performance are very good for big files (see the last era5-montly interpolation) which suggest great scaling
- Writing files bring we back to square one, putting on the same line as CDO (but this is expected).
IMPORTANT: this numbers does not take into account the loading of the data
This issue is to keep track of the speed tests that I have been to see what is the optimal configuration for the regridder based on #1
The tests are based on files on different grids (curvilinear, gaussian, gaussian reduced, lonlat and unstructured) to cover all the possibilities, with 2D files, files with mask (i.e. ocean files) and files with pressure levels. We also tested the access of the entire xarray.Dataset versus working on the single xarray.DataArray. The writing of the NetCDF output is also assessed. All tests are run with conservative remapping.
The tests can be found in the playground notebook, and are based on multiple repetition (usually 20 fo each operation). https://github.com/jhardenberg/smmregrid/blob/devel/extend/playground.ipynb