NVlabs / imaginaire

NVIDIA's Deep Imagination Team's PyTorch Library
Other
4.02k stars 449 forks source link

Pix2PixHD - Train second batch on top of previous #102

Closed alelordelo closed 3 years ago

alelordelo commented 3 years ago

Hello Nvidia team,

My goal is to train a pix2pixHD model for the second time, with Dataset 2.

For dataset 1, I trained with: python train.py --label_nc 0 --no_instance --name 1 --dataroot ./datasets/1--save_epoch_freq 50

And on Dataset 2, I copied epochs from Train/checkpoint 1 (200_net_D.pth 200net_G.pth) and placed on Checkpoint 2 folder: python train.py --label_nc 0 --no_instance --name 2 --dataroot ./datasets/2 --save_epoch_freq 50 --which_epoch 200

I want to make sure that train 2 will learn on top of train 1. Is this the correct workflow?

I looked into --continue training but looks like its supposed to resume training, not train a second dataset batch on top of a previous model.

Is that correct?

Appreciate any inputs! : )

arunmallya commented 3 years ago

If you want to restart training (not resume) from a previous checkpoint, you need to use a command like python train.py --checkpoint <path-to-checkpoint>. if you want to resume, then python train.py --checkpoint <path-to-checkpoint> --resume 1.