search

paTRICK-swk / D3DP

[ICCV2023] The PyTorch implementation for "Diffusion-Based 3D Human Pose Estimation with Multi-Hypothesis Aggregation"
MIT License
158 stars 8 forks source link

training evaluation CUDA out of memory #9

Closed HarrytheOrange closed 1 year ago

HarrytheOrange commented 1 year ago

Hi, when I am training from scratch on one 3090 GPU, it occurs:

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 23.11 GiB (GPU 0; 23.70 GiB total capacity; 1.54 GiB already allocated; 20.83 GiB free; 1.87 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Train process is totally OK, but after the training and in the evaluation step, it occurs CUDA memory problem.

paTRICK-swk commented 1 year ago

Hi, During the testing phase, sampling and iterating will increase the GPU memory footprint. You can try to reduce the value of H and K.