search

NCAR / esds

Repository for hosting material related to the Earth System Data Science (ESDS) initiative efforts
https://ncar.github.io/esds
Apache License 2.0
24 stars 20 forks source link

pangeo stack debugging cookbook #152

Open dcherian opened 3 years ago

dcherian commented 3 years ago

Moving zulip convo here for better tracking...

It'd be nice if we could start some kind of debugging cookbook with known inefficiencies and suggested workarounds

Possible topics:

  1. [ ] fixing your dask dashboard
  2. [ ] calculating climatologies and anomalies .... talks about groupby issues
  3. [ ] reading in netCDF file datasets with open_mfdataset
  4. [ ] mpi4py vs dask approach to the same problem
  5. [ ] UCX on cheyenne. when do we expect benefits?
  6. [ ] writing datasets in parallel:
    1. [ ] distributed writes to netCDF
    2. [ ] save_mfdataset

Please add commonly encountered problems to this list.

Should these go on the xdev blog; should we start another one?

kmpaul commented 3 years ago

I'm wondering if this should go to the ESDS blog. ...and whether the Xdev blog should go there, too.