C4ppucc1n0
commented
1 week ago RuntimeError: CUDA out of memory. Tried to allocate 512.00 MiB (GPU 1; 23.64 GiB total capacity; 20.90 GiB already allocated; 211.69 MiB free; 21.16 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF RuntimeError: CUDA out of memory. Tried to allocate 512.00 MiB (GPU 0; 23.64 GiB total capacity; 20.90 GiB already allocated; 213.69 MiB free; 21.16 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
你好 ，我在用两个4090复现small版本的时候，会遇到out of memory，是为啥呢，使用的命令行是python -m torch.distributed.launch --nproc_per_node=2 train.py configs/abcnet/abcnet_clft-s_512x512_500e_irstd1k.py