search

thuml / SimMTM

About Code release for "SimMTM: A Simple Pre-Training Framework for Masked Time-Series Modeling" (NeurIPS 2023 Spotlight), https://arxiv.org/abs/2302.00861
107 stars 13 forks source link

GPU out of memory error #5

Open ajati opened 8 months ago

ajati commented 8 months ago

https://github.com/thuml/SimMTM/blob/main/SimMTM_Forecasting/scripts/pretrain/ECL_script/ECL.sh fails with GPU out of memory error in a A100 80GB.

RuntimeError: CUDA out of memory. Tried to allocate 13.26 GiB (GPU 0; 79.15 GiB total capacity; 60.37 GiB already allocated; 7.43 GiB free; 70.33 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

linxi20 commented 3 months ago

Hello, I have also encountered this issue. Have you resolved it now?

iDestro commented 2 months ago

I also have this problem, why?

Hardware: 3090

Script: bash ./scripts/pretrain/ECL_script/ECL.sh


torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 207.37 GiB (GPU 0; 23.70 GiB total capacity; 8.14 GiB already allocated; 13.37 GiB free; 9.09 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF