search

NVlabs / imaginaire

NVIDIA's Deep Imagination Team's PyTorch Library
Other
4.02k stars 449 forks source link

Where SPADE/GauGAN saves checkpoints? how to load them into inference.py? #99

Closed Kitty-sunray closed 3 years ago

Kitty-sunray commented 3 years ago

So far many epochs gone and no checkpoint was saved in my training, despite "snapshot_save_epoch: 1" in the config. train.py has parameter "checkpoint" but it is not for saving path of checkpoints What is 'val' part of dataset? is it necessary for training to train properly or is it just for user to see how good model performs at each epoch?

arunmallya commented 3 years ago

Val is used to determine the quality of the model before saving. The checkpoints should be saved in the logs/ directory.

Kitty-sunray commented 3 years ago

I mean, is "val" folders affects training in any way? or is it just for human to see? Can I have empty "val" folder and successfully train?

arunmallya commented 3 years ago

val does not affect training, but it might break the training loop if no val folder is found. You can provide a subset of train as val set in that case.