while training a ipadapter model over my finetuned stable diffusion 1.5 model, i am observing training loss not converging properly, i have not altered the default learning rates or the optimizers they are same as in repo. What could be the problem, the reference image is not able to condition well the ip adapter or i need to modify the dataset ? . Or is it common for ip adapter loss to remain very steady and not change much
while training a ipadapter model over my finetuned stable diffusion 1.5 model, i am observing training loss not converging properly, i have not altered the default learning rates or the optimizers they are same as in repo. What could be the problem, the reference image is not able to condition well the ip adapter or i need to modify the dataset ? . Or is it common for ip adapter loss to remain very steady and not change much