Open uYouUs opened 2 months ago
hollowstrawberry
commented
2 months ago Okay so basically this: https://github.com/hollowstrawberry/kohya-colab/issues/134#issuecomment-2054203949
I'm surprised to hear it worked well for you in non-XL. I recognize it would be very useful for XL, but I don't think it would be usable.
uYouUs
commented
2 months ago Yes. I first tried it after the 10th epoch was still not very well trained. At the default training rate it was taking nearly 30 epochs. Increasing the rate did help but I couldn't get a sweet spot. But I did find that training at a high rate until overfitting, and continuing training from 2 previous epoch at a lower rate gave me pretty good results.
As for the XL training, 5e-4 for 15 epochs is the best setting I have found for great results. However 4/5 times with the same dataset and settings it will overfit. So 4e-4 is what I've been using. Much less early overfitting but almost always undertrained.
I think using the lower training rate and continuing from there should work better in the case of the XL training.
uYouUs
commented
2 months ago I copied the relevant parts from the non XL version over to the XL version and I can confirm it works wonderfully. Here's a blank copy with the settings I use in case you would like to test. https://colab.research.google.com/drive/1ggqQp-nvqFT0ZJNS9m_l9fhTc2taYA7c?usp=sharing
From my testing in the past, and it still seems true now. If you continue with the original training settings,the first 2 epochs are no good. The 3rd epoch is the first real result from continuing the training in this way. From what I've seen, those first two have a chance of becoming useless along with the ones after, which I assume is why most people don't get good results. For that, the "secret" i've found is to lower the strength of the lora. 0.6-0.7 of the original strength before resuming training does wonders in that aspect. Here's how the training went:
And the lora came out wonderful.
I was wondering if it would be possible to add the option to continue from a previous lora in the XL trainer like you can with the non XL trainer. I know its not the same as continuing from where it left off but It worked great for me in the non XL, and with XL taking longer, It would be of great help.