search

ostris / ai-toolkit

Various AI scripts. Mostly Stable Diffusion stuff.
MIT License
220 stars 25 forks source link

lora slider, is there a way to stop and resume? #11

Closed igobymatthew closed 10 months ago

igobymatthew commented 10 months ago

lora slider, is there a way to stop and resume for people that have slow gpus?

FoundSol commented 10 months ago

Using powershell terminal(assuming that you're using windows), press ctrl+c to stop the training, then when you want to resume, just re-type the code used initially(python run.py YourTemplate.yml). The toolkit save a metadata from training then you can start from where you stop. I've not tested this function deeply, but seems to work pretty fine. Remember to try this function before letting your concepts training for hours. A message like '''Found step in metadata, starting from there''' must be shown if its resuming the training.

jaretburkett commented 10 months ago

Yes as @FoundSol mentioned, it it will auto resume if it finds a a saving step in the training folder. I do not save the state of the optimizer currently, but for general LoRA training, it is not really necessary (in my opinion)