TachibanaYoshino / AnimeGANv2

[Open Source]. The improved version of AnimeGAN. Landscape photos/videos to anime
5.08k stars 714 forks source link

issue saving checkpoints of model #41

Closed wooae closed 2 years ago

wooae commented 2 years ago

hello! when I try to train the model, I get the following error when the code tries to save the checkpoint:

Traceback (most recent call last):
  File "main.py", line 115, in <module>
    main()
  File "main.py", line 107, in main
    gan.train()
  File "/content/drive/.shortcut-targets-by-id/1X8hfrOWE2KxmaJG4LFKH9ydVQ4BA7oyZ/cs7643-final-project/AnimeGANv2.py", line 302, in train
    self.save(self.checkpoint_dir, epoch)
  File "/content/drive/.shortcut-targets-by-id/1X8hfrOWE2KxmaJG4LFKH9ydVQ4BA7oyZ/cs7643-final-project/AnimeGANv2.py", line 341, in save
    self.saver.save(self.sess, os.path.join(checkpoint_dir, self.model_name + '.model'), global_step=step)
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/training/saver.py", line 1186, in save
    save_relative_paths=self._save_relative_paths)
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/training/checkpoint_management.py", line 231, in update_checkpoint_state_internal
    last_preserved_timestamp=last_preserved_timestamp)
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/training/checkpoint_management.py", line 110, in generate_checkpoint_state_proto
    model_checkpoint_path = os.path.relpath(model_checkpoint_path, save_dir)
  File "/usr/lib/python3.7/posixpath.py", line 475, in relpath
    start_list = [x for x in abspath(start).split(sep) if x]
  File "/usr/lib/python3.7/posixpath.py", line 383, in abspath
    cwd = os.getcwd()
FileNotFoundError: [Errno 2] No such file or directory

I mounted my google drive into colab and am using colab to train the model. When I check my checkpoint folder, I have two files there but it appears that I am missing the checkpoint binary file and the .meta file. Any idea why this could be happening? image

wooae commented 2 years ago

it seems that saving the checkpoints did not work consistently if I tried saving to a shared google drive folder, but worked if I saved to my own folder. closing the issue.