Problem about cuda-out-of-memory

[Chelsea-abab]

Hi, I want to reproduce your results via your provided codes. But I was stuck in the fine-tuning section. No matter how I reduce the batch size and input image size, it still says cuda out of memory. I run the codes by your instructions on a nvidia3090 gpu with 24g memory. But it seems that before the program loads the images, all the memory has already been allocated. So although I reduce the batch size to 1 and input size to 64, the cuda out of memory problem is still there. Does the memory be allocated to the models? Do you have any idea on how to solve this problem? I guess 24g is enough to tune a diffusion model.

[yu-rp]

Hi @Chelsea-abab ,

Reducing the GPU consumption is a quite general problem. Here are some ideas to achieve that:

1. Use Mixed Precision Training: This involves using half-precision floating points (FP16) instead of single precision (FP32). Libraries such as NVIDIA's Apex make this easier.

2. Gradient Accumulation: Instead of updating weights every batch, accumulate gradients over multiple batches and then make a single update. This effectively simulates a larger batch size without the memory requirements.

3. Use Gradient Checkpointing: Trade computation for memory by re-computing intermediate activations during the backward pass.

Or, for the finetuning of the diffusion model, you may also try to find other repositories that include much more efficient implementation.

Hope these ideas may help you to address the problem.

[Chelsea-abab]

OK, thanks for your reply! I thought there are some hyperparameters could be adjusted to reduce the GPU consumption. Then I will try these general methods to address this problem. Thanks again!

[jxthyatt]

@Chelsea-abab did u solve this problem?