search

zwx8981 / LIQE

[CVPR2023] Blind Image Quality Assessment via Vision-Language Correspondence: A Multitask Learning Perspective
MIT License
197 stars 11 forks source link

Training OOM #27

Open rexainn opened 3 months ago

rexainn commented 3 months ago

Hi, I want to know what GPU do you use for training? I use a V100, but it kept reporting out of memory. I have turned off the 'convert_models_to_fp32'.

rexainn commented 3 months ago

I notice that in your paper you mentioned experiments using a single 3090. So is it because I train it on my own dataset, and there exists 7 tasks?

zwx8981 commented 3 months ago

Maybe, try using smaller batch size

zwx8981 commented 3 months ago

You may also try setting opt = 1 in Line127, which freezes the weights of text encoder. Empirically, this would not affect the final performance very much, but can significantly reduce the memory cost.

rexainn commented 3 months ago

Maybe, try using smaller batch size

set batchsize = 1 still cause OOM, quite strange....

rexainn commented 3 months ago

You may also try setting opt = 1 in Line127, which freezes the weights of text encoder. Empirically, this would not affect the final performance very much, but can significantly reduce the memory cost.

This works, thanks! Meanwhile, I will still try to find the way to not freeze the text encoder