Parallel weight generation with Dask

[charlesgauthier-udm]

Implemented parallel weight generation using Dask and xarray's map_blocks. Here is a quick summary: User can pass parallel=True to the Regridder and the weights will be computed in parallel.

Key points:

• Parallel weight generation uses the chunks on the output dataset or dataarray given to Regridder.
• There is overhead associated with map_blocks and dask, especially with the creation of a template for map_blocks, so for small grids serial weight generation is prefered. Therefore, the default is parallel=False
• When parallel=True, an identical Regridder object to the serial case is returned. Could possibly add a self.parallel in the Regridder to keep knowledge of if it was generated in parallel.

Examples

Using dask to compute the weights allows for larger-than-memory dataset to be used. Using subsets of the Gridded Population of the World (gpw) and the CORDEX WRF in lambert conformal with a 0.22° resolution (y:281, x:297), we get the following examples:

• WRF (y:281, x:297) --> GPW_subset(lat:5000, lon:5000); parallel=False: memory overflows, parallel=True: Regridder created in ~86s on my 4-core machine.
• Using parallel=True I can tackle even bigger datasets: WRF (y:281, x:297) --> GPW_subset(lat:7000, lon:7000): Regridder created in ~2mins

Comparing serial vs. parallel, the overhead related to dask and map_blocks makes it slower for small datasets, but for bigger datasets we can compare both:

• WRF (y:281, x:297) --> GPW_subset(lat:5000, lon:4000); parallel=False: Regridder created in ~100s, parallel=True: Regridder created in ~ 50s. Roughly 2x faster.

Execution time and memory usage is highly dependent on chunk sizes and the number of cores available. However, by chunking the output dataset, the user can adjust it to a specific problem.

[review-notebook-app[bot]]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[charlesgauthier-udm]

@huard @aulemahal Looks like moving the para_regrid code outside of __init__ to its own method does not solve the issue of __init__ being too complex..

[huard]

I can live with that.