search

earth-system-radiation / pyRTE-RRTMGP

pyRTE-RRTMGP provides a Python interface to the RTE+RRTMGP Fortran software package
https://pyrte-rrtmgp.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
7 stars 0 forks source link

Add horizontal (CPU) scaling via dask #54

Open tcmetzger opened 1 month ago

tcmetzger commented 1 month ago

The Python user interface will take advantage of horizontal (CPU) scaling options provided by xarray. Specifically, when a user supplies the chunks argument when loading data in xarray, the xarray.DataArray is backed by a dask.Array which requires consideration by pyRTE-RRTMPG.

This task also includes creating a notebook of a global example (roughly 1 degree resolution at one point in time) using the existing clear_sky problem with a snapshot of the globe (Robert to obtain data).

tcmetzger commented 4 weeks ago

The wrinkle here is that the underlying kernel expects chunks - how big should they be? we need a reasonable default, depending on the platform or hardware - something that the user can override.

Potentially: chunking in Python, then call multiple instances of Fortran without chunks? However, this might have severe performance drawbacks.

Potentially: chunk in python by default, but also allow user to pass on a param to Fortran.