search

pangeo-data / pangeo

Pangeo website + discussion of general issues related to the project.
http://pangeo.io
703 stars 189 forks source link

Update `Getting-Started-with-Dask-on-Cheyenne` with proper memory usage #217

Closed tjmills closed 6 years ago

tjmills commented 6 years ago

The current example shows 24GB of RAM for workers, this has been shown to be an issue on Cheyenne PBS jobs per PR https://github.com/pangeo-data/pangeo/pull/27

Please update https://github.com/pangeo-data/pangeo/wiki/Getting-Started-with-Dask-on-Cheyenne with the proper memory usage from the linked PR above.

Once I get these bumps ironed out, what you all have put together will greatly help our work. Thank you!

jhamman commented 6 years ago

@jmills-ncar - you may want to checkout dask-jobqueue: http://dask-jobqueue.readthedocs.io/en/latest/

This is currently probably the easiest way to manage dask distributed on a computer like cheyenne.

jhamman commented 6 years ago

@rabernat is working on an update to the pangeo documentation and I image when that effort is done, we can remove the wiki page you linked to all together.

tjmills commented 6 years ago

Fantastic, thank you! I will check it out

jhamman commented 6 years ago

@jmills-ncar - make sure to checkout https://pangeo-data.github.io/pangeo/setup_guides/hpc.html