Auto resume from checkpoint looks for "trainer_state.json", but no file is generated - Githubissues

axolotl-ai-cloud / axolotl

Go ahead and axolotl questions

https://axolotl-ai-cloud.github.io/axolotl/

Apache License 2.0

7.87k stars 866 forks source link

Auto resume from checkpoint looks for "trainer_state.json", but no file is generated #1631

Open l3utterfly opened 5 months ago

l3utterfly commented 5 months ago

Please check that this issue hasn't been reported before.

[X] I searched previous Bug Reports didn't find any similar reports.

Expected Behavior

Auto resume from checkpoint should work

Current behaviour

It is looking for: FileNotFoundError: [Errno 2] No such file or directory: 'outputs/out/checkpoint-9/trainer_state.json'

However this file is not generated

Steps to reproduce

Start training for Llama3
Wait for one checkpoint
Stop and resume

Config yaml

No response

Possible solution

No response

Which Operating Systems are you using?

[ ] Linux
[ ] macOS
[ ] Windows

Python Version

3.10

axolotl branch-commit

main-latest

Acknowledgements

[X] My issue title is concise, descriptive, and in title casing.
[X] I have searched the existing issues to make sure this bug has not been reported yet.
[X] I am using the latest version of axolotl.
[X] I have provided enough information for the maintainers to reproduce and diagnose the issue.

winglian commented 5 months ago

@l3utterfly Can you paste the ls output for the contents of that directory?