Open davidmatthews1uvm opened 3 years ago
It sounds like this could be due to the exclusive-process mode E. Process
settings in your cluster: https://discuss.pytorch.org/t/distributeddataparallel-runtimeerror-cuda-error-all-cuda-capable-devices-are-busy-or-unavailable/102763/7 ?
Thanks for the advice. I checked and you are right -- the cluster is in exclusive-process mode. Do you know of any workarounds for this issue?
There is a suggestion of how to turn it off: https://discuss.pytorch.org/t/distributeddataparallel-runtimeerror-cuda-error-all-cuda-capable-devices-are-busy-or-unavailable/102763/5. But I'm not sure if you are authorized to just change that on your cluster.
Also could you try printing out CUDA_VISIBLE_DEVICES
? You may be able to widen the devices you can use, see https://stackoverflow.com/a/39661999/12003165. (Of course this depends on how many physical GPU devices your cluster machine has)
Last but not the least, your stack trace has implied that the pytorch tensor is on a CPU device (the else
branch means CPU):
Could you try directly passing in a tensor on GPU?
Thanks for the suggestions. I was able to get taichi-pytorch to pass data between GPU memory directly however with some caveats: this only works when pytorch is using a different GPU than taichi. In that case: x.to_torch(device=device) and x.from_torch(x_torch) work as expected.
Do you think it would be feasible to expose the taichi CUDA context and directly pass it to pytorch or vice versa? I can use 2 GPUs if needed but ideally I would only need to allocate 1.
Do you think it would be feasible to expose the taichi CUDA context and directly pass it to pytorch or vice versa? I can use 2 GPUs if needed but ideally I would only need to allocate 1.
I see. While it is possible for Taichi to borrow the existing CUDA context created by pytorch (or any other CUDA-enabled library) in the same process, this assumes that the borrowed context is not destroyed or changed during the lifetime of Taichi. I couldn't find any reference in Pytorch saying whether this holds or not.
Would it be a bit more easier if, while you still need to use two GPUs, Taichi would allow you to specify which GPU device ID to use?
Filed https://discuss.pytorch.org/t/share-the-cuda-context-created-by-pytorch/113359
Describe the bug Taichi is crashing when I call .to_torch() on a taichi field from a cluster (SLURM) environment.
To Reproduce
Log/Screenshots
Additional comments The following versions of torch and torch vision were installed with pip
ti diagnose output below: