Closed Quasimondo closed 7 years ago
saver=tf.train.Saver(max_to_keep=1000)
assumes the variable list to be all the variables. It is not a blank Saver. I double check the code by running it again.
Unfortunately I am not too experienced with the way tensorflow handles checkpoints, but it looks to me like there is something strange happening in demo512 (vs. demo256 which works as expected): When I try to continue training from a result_512p checkpoint of a model that has been trained for 20 epochs and has progressed to a decent quality, the results I get when continuing training from the saved checkpoint look like it went back to epoch 1, looking very blurry and crude.
Oh, never mind - I found the error and it was my fault. I had renamed my model path but missed an identifier, so it was loading the wrong checkpoint.
It seems like continuing a training session of demo_512 and demo_1024 does not work since after restoring a previously trained model it gets immediately overwritten by a blank Saver. I think that last line should be moved before the ckpt check, like in demo_256: