Is it possible to display an intermediate image of the training process, or to restart training again from the intermediate process?

levihsu / OOTDiffusion

Official implementation of OOTDiffusion: Outfitting Fusion based Latent Diffusion for Controllable Virtual Try-on

Other

5.45k stars 801 forks source link

Is it possible to display an intermediate image of the training process, or to restart training again from the intermediate process? #192

Open gaoming714 opened 3 months ago

gaoming714 commented 3 months ago

Excuse me, is it possible to display an intermediate image of the training process, or to restart training again from the intermediate process? I'd like to be able to pause the pipe so that I can do other things in the middle of the process (like getting a new coat) and thus reduce the overall build time.

levihsu commented 3 months ago

Yep. Just save checkpoints and set resume_from_checkpoint in your training script to restart training.