ErikKrauter
commented
8 months ago After tinkering around with that issue some more I have noticed that this behavior is not caused by the ReplayMemory class. The behavior also occurs in other scenarios. When executing the training on a GPU that does not support CUDA the script exits with an error, but the SLURM job hangs. When terminating the SLURM job with Control+C the following is printed. This again shows that the process is stuck waiting for communication. It seems the parallel_runner.py script does not handle errors correctly leading to endless loops.
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Error in atexit._run_exitfuncs:
Process Worker-1:
Traceback (most recent call last):
File "/scratch_net/got/ekrauter/conda_envs/lab_mani_skill2/lib/python3.8/multiprocessing/popen_fork.py", line 27, in poll
pid, sts = os.waitpid(self.pid, flag)
KeyboardInterrupt
Traceback (most recent call last):
File "/scratch_net/got/ekrauter/conda_envs/lab_mani_skill2/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/scratch_net/got/ekrauter/Masterthesis/ManiSkill2-Learn/maniskill2_learn/utils/meta/parallel_runner.py", line 113, in run
op, args, kwargs = self.worker_pipe.recv()
File "/scratch_net/got/ekrauter/conda_envs/lab_mani_skill2/lib/python3.8/multiprocessing/connection.py", line 250, in recv
buf = self._recv_bytes()
File "/scratch_net/got/ekrauter/conda_envs/lab_mani_skill2/lib/python3.8/multiprocessing/connection.py", line 414, in _recv_bytes
buf = self._recv(4)
File "/scratch_net/got/ekrauter/conda_envs/lab_mani_skill2/lib/python3.8/multiprocessing/connection.py", line 379, in _recv
chunk = read(handle, remaining)
KeyboardInterrupt
xuanlinli17
lz1oceani
I train a PPO+DAPG agent on 5GPUs in prallel, with 3 processes per GPU for rollouts. I use SLURM as the cluster management system to schedule sbatch jobs.
The issue I encounter is that the sbatch job never terminates even if the training is done. The error log contains the following messages:
To debug the issue I tried running the same training with only one GPU (still 3 processes for rollout) in SLURM interactive debug mode. The issue persists (same error message as above). When I manually terminate the job with Control+C the following is printed to the terminal:
Even if I only use one GPU with only one process for rollouts I observer the exact same behavior.
It seems to me that the problem is likely related to the ReplayMemory class and its interaction with Worker class in parallel_runner.py. This interaction is failing during cleanup, potentially due to resource management or IPC handling within these classes. The fact that the process hangs waiting for pipe communication (recv()) suggests a potential deadlock or synchronization issue. It's possible that the cleanup process is not being executed correctly, leading to a state where a worker process is indefinitely waiting for a signal or data.
The training still works, just the termination does not work properly. Do you have any advice on how to fix the issue or can you propose a workaround of how to make the SLURM job terminate after completion of the training?