CDAT / cdms

8 stars 10 forks source link

ESMF regridder slow... #112

Open dnadeau4 opened 7 years ago

dnadeau4 commented 7 years ago

I’m testing thre pmp code with the newer esmf regridder.

It has gotten soooo slow now that it is pretty much rendered unusable…

The test suite went from running a trst in a few seconds to me killing it after 10mn because I thought it was dead. We definitely cannot release cdms2 in that state.

I think the “regrid” fix needs to be revisited as soon as possible!

dnadeau4 commented 7 years ago

I re-implemented the mask in following this paragraph:

One trick you can do right now with missing values and a weight file is to do two interpolations to tell you where the missing values will spread in the destination field. First interpolate a field containing all 0's except for where the missing values are set the missing value locations to 1, the result of this interpolation tells you which destination locations (the ones whose values >0.0) will be affected by the missing values, so you can ignore them after interpolating the data. (You could also do something like this in one step during the data interpolation if you have a restricted range for your data and you use a value way outside that range for your missing value.) This, of course, will give you a coarser result than using the masking, but will probably be more efficient than recalculating the weights each time.

https://www.ncl.ucar.edu/Applications/ESMF.shtml

jypeter commented 7 years ago

Did you monitor the memory usage when you got the slow regridding? Can you check #113 and tell me if things work smoothly now?