Remote GPU Usage - Githubissues

quasiben commented 4 years ago

Lately, I've been using dask/nvdashboard in multinode setups. Can nvdashboard be extended to report GPU memory usage on remote workers ?

cc @jacobtomlinson

jacobtomlinson commented 4 years ago

This has been raised before in #34 and #45.

I feel like we have a few options here, ordered from least complex to most:

Run nvdashboard manually on each node and open each dashboard in a new window
Use Dask's built in GPU dashboards (may not make sense if you aren't using Dask)
Extend the existing dashboards to be able to collate data from multiple instances of nvdashboard. This would probably still require the user to run nvdashboard on each node.

This is a highly requested feature so perhaps we should give it some more time and thought.

quasiben commented 4 years ago

Thanks for the response @jacobtomlinson. For others, the "secret" dask dashboard is at:

http://URL:8787/individual-gpu-memory

rapidsai / jupyterlab-nvdashboard