search

justanhduc / task-spooler

A scheduler for GPU/CPU tasks
https://justanhduc.github.io/2021/02/03/Task-Spooler.html
GNU General Public License v2.0
273 stars 24 forks source link

can I query the GPU ids allocated for a job? #10

Closed ShuyangCao closed 2 years ago

ShuyangCao commented 2 years ago

Thanks for developing the tool! Right now ts -i only returns the number of GPUs allocated and ts -p returns the pid of the main process. To know which GPUs the process and its child processes actually uses, I am using pgrep and crossing ref with nvidia-smi. Is there an easy way to do so?

justanhduc commented 2 years ago

Hi @ShuyangCao. Thanks for your interest. The feature you asked is already available in this branch (via ts -i. It would take a bit more time to merge into master but it is stable enough to be used now.

ShuyangCao commented 2 years ago

Thanks. Can I reinstall ts using this branch without killing the server?

justanhduc commented 2 years ago

Unfortunately it's impossible 🙁

ShuyangCao commented 2 years ago

Thanks @justanhduc. It works great! Just wondering, does this branch support 0 wait time? As the scheduler has recorded the GPU ids, I think it probably can avoid assign jobs to the same GPU, right?

justanhduc commented 2 years ago

Hi @ShuyangCao. Yes it does! Now the server will assign GPUs to client instead of letting client choose as before, so queued jobs are executed immediately if there are available slots and GPUs.