google colab kicked me out due to gpu usage

[DrakeFruit]

after running for 4 hours it kicked me out and wont let me run the colab, this also deleted all my progress in the ckpt file... any way to recover that?

[nicolai256]

no if your colab stops it stops, one of the downsides of colab

[DrakeFruit]

how do I save the files to my drive and run it from there

[nicolai256]

put the repo in ur drive and modify the commands to use the repo in the drive

[DrakeFruit]

its been running for an hour on my drive but im not seeing any progress images in logs/imagesfortraining/images/train

[affanmehmood]

How often does the code save the checkpoint files?

[nicolai256]

it saves when u cancel the training

[nicolai256]

intermediate saving still has to be fixed, for now it saves every epoch tho

[affanmehmood]

This is the issue I'm facing when I try to resume the training from last.ckpt. Why is this happening?

Global seed set to 23 Running on GPUs 1 Loading model from /content/drive/MyDrive/---/checkpoints/last.ckpt Traceback (most recent call last): File "/content/Stable-textual-inversion_win/main.py", line 604, in model = load_model_from_config(config, opt.actual_resume) File "/content/Stable-textual-inversion_win/main.py", line 37, in load_model_from_config sd = pl_sd["state_dict"] KeyError: 'state_dict'

[nicolai256]

Resume doesn't work yet