nsaadati
commented
1 year ago nevermind this is my system problem
Closed nsaadati closed 1 year ago
nsaadati
commented
1 year ago nevermind this is my system problem
hi thanks for your well written paper I had some questions toward your code: when I try to run the usage part (sh butd_detr/scripts/train_test_det.sh) I get this error message, do you know what can be the reason? I have already completed the other steps. best Nastaran
File "/home/exouser/.conda/envs/bdetr3d/lib/python3.7/site-packages/torch/nn/functional.py", line 5101, in multi_head_attention_forward attn_output, attn_output_weights = _scaled_dot_product_attention(q, k, v, attn_mask, dropout_p) File "/home/exouser/.conda/envs/bdetr3d/lib/python3.7/site-packages/torch/nn/functional.py", line 4844, in _scaled_dot_product_attention attn = torch.bmm(q, k.transpose(-2, -1)) RuntimeError: CUDA out of memory. Tried to allocate 768.00 MiB (GPU 0; 20.00 GiB total capacity; 16.10 GiB already allocated; 309.69 MiB free; 16.12 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 7908) of binary: /home/exouser/.conda/envs/bdetr3d/bin/python Traceback (most recent call last): File "/home/exouser/.conda/envs/bdetr3d/lib/python3.7/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/home/exouser/.conda/envs/bdetr3d/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/exouser/.conda/envs/bdetr3d/lib/python3.7/site-packages/torch/distributed/launch.py", line 193, in
main()
File "/home/exouser/.conda/envs/bdetr3d/lib/python3.7/site-packages/torch/distributed/launch.py", line 189, in main
launch(args)
File "/home/exouser/.conda/envs/bdetr3d/lib/python3.7/site-packages/torch/distributed/launch.py", line 174, in launch
run(args)
File "/home/exouser/.conda/envs/bdetr3d/lib/python3.7/site-packages/torch/distributed/run.py", line 713, in run
)(*cmd_args)
File "/home/exouser/.conda/envs/bdetr3d/lib/python3.7/site-packages/torch/distributed/launcher/api.py", line 131, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/home/exouser/.conda/envs/bdetr3d/lib/python3.7/site-packages/torch/distributed/launcher/api.py", line 261, in launch_agent
failures=result.failures,
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
butd_detr/train_dist_mod.py FAILED
Failures: