Open AI-Casanova opened 9 months ago
huh weird - but is bfloat16 working for you?
T4 does not support bfloat16 natively
Will try float32 when I get back to a work station.
It is quite odd, I get loss readings on the first iteration, but as soon as the Network kicks in, both loss and latents are all NaN
https://huggingface.co/stabilityai/sdxl-vae/discussions/6
Found this issue on stabilityAI repo - it seems like some issue with fp16 support on diffusers. even I could not get it working
I tried side loading the fp16-fix VAE (which I also use for Koyha LoRA training) but even that was failing.
sdxl in regular inference seems to be doing it for me (simple inference, no sliders). I am not sure what the reason is.
Essentially the first step looks good, but after that, I am just seeing black images. I looked at the tensors they are all NaN values
The VAE that came packaged with sdxl had activations that were too large for fp16.
People who do fp16 inference use the sdxl0.9 VAE or this one that has been rescaled. https://huggingface.co/madebyollin/sdxl-vae-fp16-fix
I am trying to train in colab t4 but I am getting this error I also have this problem but I'm trying to train in sd1.4, did anyone solve it?
It is working when running with float32
precision
Attempting to train on a Colab T4, which requires fp16 precision.
All scripts report NaN loss after the first Network update, no matter the hyperparameters I choose.
Any advice for future experimentation?