Train with fp16

[yjhong89]

I am trying to fine-tune HumanSD ckpt with use_fp16=True option. (I installed xformers) But it still RuntimeError occurs like below

• I think there needs some codes converting model weight to fp16 but there isn't any convertion or autocast

• https://github.com/IDEA-Research/HumanSD/blob/fe480f723f80bc62b158503e858c31174fa629c1/ldm/modules/diffusionmodules/openaimodel.py#L23

• Is it possible training stable diffusion with fp16 ??

• If I set use_fp16=False in configs, it works

[juxuan27]

Hi, @yjhong89 ! Thank you for your focus. I have tried to train with use_fp16=True but it just works fine. Maybe you can try to force the input to FloatTensor in your case? To do so, you can go to comparison_models/ControlNet/ldm/models/diffusion/ddpm.py and find get_input function in your latent diffusion class, then use .to(your expected data type).

[yjhong89]

Thanks!

[xiao2mo]

你能自己复现下吗。把环境发一下，fp16 训练有问题啊。

[xiao2mo]

@yjhong89

[chaunceywx]

Thanks!

Have you successfully trained model with fp16 now? I encountered the same issue. How should I modify the code to train the model with fp16?

[chaunceywx]

Thanks!

@yjhong89