search

fudan-generative-vision / champ

Champ: Controllable and Consistent Human Image Animation with 3D Parametric Guidance
https://fudan-generative-vision.github.io/champ/
MIT License
4.29k stars 533 forks source link

Hardware and data requirements for model training. #115

Open siyuzhu-fudan opened 4 months ago

siyuzhu-fudan commented 4 months ago

What are the GPU requirements to run the training and approximately how many input videos should be used for training?

pearbender commented 4 months ago

I would also like to know. I attempted training with an RTX 3060 and batch size of 1 and ran out of memory. I have 12 GB of dedicated GPU memory and 16 GB of shared GPU memory.

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 114.00 MiB. GPU 0 has a total capacity of 12.00 GiB of which 0 bytes is free. Of the allocated memory 24.51 GiB is allocated 
by PyTorch, and 766.31 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
pearbender commented 4 months ago

I have stage 1 training running with my RTX 3060 and 64 GB of RAM. That's 12 GB of GPU memory and 32 GB of shared GPU memory totaling 44 GB of GPU memory. It seems to be just barely enough.

pearbender commented 4 months ago

I've given up on training after reading the paper which said training was conducted with 8 NVIDIA A100 GPUs. My training time for stage 1 is approximately 1800 hours.

image

image