Lora training is OOM on RTX4090 GPU？

Tencent / HunyuanDiT

Hunyuan-DiT : A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understanding

https://dit.hunyuan.tencent.com/

Other

2.68k stars 197 forks source link

Lora training is OOM on RTX4090 GPU？ #130

Closed frankchieng closed 2 days ago

frankchieng commented 3 days ago

how much VRAM required during lora training?24G is not enough so far,can you support bf16 in deepspeed?currently only zero stage 2 with fp32 or fp16 supported,plz figure it out with lower VRAM running thx