search

PDillis / stylegan3-fun

Modifications of the official PyTorch implementation of StyleGAN3. Let's easily generate images and videos with StyleGAN2/2-ADA/3!
Other
237 stars 37 forks source link

is it possible to resume tick and augment too? #5

Closed nicolai256 closed 2 years ago

nicolai256 commented 2 years ago

Describe the bug the storage of the remote pc was full so it stopped training but it's already pretty far into training i'd like to resume without it blurring the new gens maybe it's possible with the log file?

tick 1587 kimg 6348.0 time 7d 08h 07m sec/tick 396.5 sec/kimg 99.12 maintenance 0.3 cpumem 6.35 gpumem 30.74 reserved 41.21 augment 34.266

PDillis commented 2 years ago

Yes, this can be done. I will add the --initstrength that @dvschultz has in his repo. The tick resume can be done, but it's just the number of iterations on the current run. In the end, what really matters is the kimg that the model has been trained for, which is already solved.

PDillis commented 2 years ago

It should be fixed now, you can set --initstrength=34.266 or whatever you want. I think a more careful analysis should be done here w.r.t. rampup/ema, but let me know if this works for you.