restarting a finetune hangs sometimes

helixml / helix

Multi-node production GenAI stack. Run the best of open source AI easily on your own servers. Easily add knowledge from documents and scrape websites. Create your own AI by fine-tuning open source models. Integrate LLMs with APIs. Run gptscript securely on the server

https://tryhelix.ai

Other

339 stars 29 forks source link

restarting a finetune hangs sometimes #143

Open lukemarsden opened 9 months ago

lukemarsden commented 9 months ago

runner segfaulted while finetuning
when it came back, clicking restart did nothing
nothing is running on the runner now

lukemarsden commented 9 months ago

lower priority because things are more reliable now so this is hurting less