Open caojiehui opened 11 months ago
terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -11) local_rank: 0 (pid: 2346527) of binary: /home/yingjie/miniconda3/envs/point_uv_diff/bin/python Traceback (most recent call last): File "/home/yingjie/miniconda3/envs/point_uv_diff/lib/python3.8/runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None, File "/home/yingjie/miniconda3/envs/point_uv_diff/lib/python3.8/runpy.py", line 87, in _run_code exec(code, run_globals) File "/home/yingjie/miniconda3/envs/point_uv_diff/lib/python3.8/site-packages/torch/distributed/launch.py", line 193, in <module> main() File "/home/yingjie/miniconda3/envs/point_uv_diff/lib/python3.8/site-packages/torch/distributed/launch.py", line 189, in main launch(args) File "/home/yingjie/miniconda3/envs/point_uv_diff/lib/python3.8/site-packages/torch/distributed/launch.py", line 174, in launch run(args) File "/home/yingjie/miniconda3/envs/point_uv_diff/lib/python3.8/site-packages/torch/distributed/run.py", line 715, in run elastic_launch( File "/home/yingjie/miniconda3/envs/point_uv_diff/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 131, in __call__ return launch_agent(self._config, self._entrypoint, list(args)) File "/home/yingjie/miniconda3/envs/point_uv_diff/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 245, in ### launch_agent raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError: ========================================================= test.py FAILED****_**
Looks like you're hitting a memory issue, either not enough memory or maybe a memory leak. Check your system's free memory, and maybe keep an eye on GPU memory usage to spot any leaks.
what is the recommended GPU vram ? Will it run on a 6gb rtx 3060 ?