search

getkeops / keops

KErnel OPerationS, on CPUs and GPUs, with autodiff and without memory overflows
https://www.kernel-operations.io
MIT License
1.04k stars 64 forks source link

Multi-GPU support #218

Open mbanani opened 2 years ago

mbanani commented 2 years ago

Thank you for the great library. I tried using this library within a multi-gpu training pipeline, but got the following error regarding locks (see below). I was curious if there's anyway for me to use this within a multigpu setting? Thank you! 

  File "/home/mbanani/miniconda3/envs/project/lib/python3.9/site-packages/pykeops/common/lazy_tensor.py", line 2131, in argKmin
    return self.reduction("ArgKMin", opt_arg=K, axis=axis, dim=dim, **kwargs)
  File "/home/mbanani/miniconda3/envs/project/lib/python3.9/site-packages/pykeops/common/lazy_tensor.py", line 744, in reduction
    return res()
  File "/home/mbanani/miniconda3/envs/project/lib/python3.9/site-packages/pykeops/common/lazy_tensor.py", line 929, in __call__
    return self.callfun(*args, *self.variables, **self.kwargs)
  File "/home/mbanani/miniconda3/envs/project/lib/python3.9/site-packages/pykeops/torch/generic/generic_red.py", line 568, in __call__
    out = GenredAutograd.apply(
  File "/home/mbanani/miniconda3/envs/project/lib/python3.9/site-packages/pykeops/torch/generic/generic_red.py", line 47, in forward
    myconv = LoadKeOps(
  File "/home/mbanani/miniconda3/envs/project/lib/python3.9/site-packages/pykeops/common/keops_io.py", line 48, in __init__
    self._safe_compile()
  File "/home/mbanani/miniconda3/envs/project/lib/python3.9/site-packages/pykeops/common/utils.py", line 74, in wrapper_filelock
    with FileLock(f):
  File "/home/mbanani/miniconda3/envs/project/lib/python3.9/site-packages/pykeops/common/utils.py", line 51, in __enter__
    fcntl.flock(self.fd, self.op)
OSError: [Errno 37] No locks available