search

NCAR / ADF

A unified collection of python scripts used to generate standard plots from CAM outputs.
Creative Commons Attribution 4.0 International
37 stars 31 forks source link

Need to implement DASK class/framework in ADF #47

Open nusbaume opened 3 years ago

nusbaume commented 3 years ago

As new DASK-enabled tools such as GeoCAT and Intake-ESM are brought into the package, there is a potential to use DASK to significantly speed-up computation.

However, to allow for user control, and to optimize the use of DASK across the entire package, an over-arching DASK framework, possibly in the form of a new ADF class or object, should be implemented.

andrewgettelman commented 3 years ago

Not in favor of dealing with DASK at this point. Have not had good luck with it. I expect that the second we change resolution in the model for example some of the chunking would break. IMHO we have other things we need to do to get this package usable. If we use it, we worry about DASK.

nusbaume commented 3 years ago

@andrewgettelman I agree, I am just adding this issue now while I remembered to do so (as we will likely have to deal with it at some point due to its use in other packages). I have added a "low priority" label to indicate that this is not particularly high on our to-do list at the moment.