Closed zjxgithub closed 3 years ago
Hi! According to the paper, they trained 500K steps for AEI-Net. In our case, the training loss seems to be little bit random and unstable, but in the long term the loss was going down. We trained for over 500K with 32 batch size. For conclusion, there are no absolute training epoch, you should observe the loss graph and validation results. Then stop when you satisfy. I know it sounds little irresponsible, but this is a behavior of GAN.
Thanks for your response! That is helpful to know! May I ask how long it took to finish 500K steps on single GPU? Several days? Cuz I am trying to estimate when I can finish the training.
On Fri, Oct 30, 2020 at 3:20 AM Changho Choi 최창호 notifications@github.com wrote:
Hi! According to the paper, they trained 500K steps for AEI-Net. In our case, the training loss seems to be little bit random and unstable, but in the long term the loss was going down. We trained for over 500K with 32 batch size. For conclusion, there are no absolute training epoch, you should observe the loss graph and validation results. Then stop when you satisfy. I know it sounds little irresponsible, but this is a behavior of GAN.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/mindslab-ai/faceshifter/issues/4#issuecomment-719402228, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFMTSLPVD4CD7P44ENFNH3DSNJZMNANCNFSM4TBN6P7Q .
Depends on your environment. Two V100 32G with 32 batch size took more than a week.
I see... Thanks!
On Fri, Oct 30, 2020 at 3:31 AM Changho Choi 최창호 notifications@github.com wrote:
Depends on your environment. Two V100 32G with 32 batch size took more than a week.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/mindslab-ai/faceshifter/issues/4#issuecomment-719411146, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFMTSLLZZAGR65DKBWPD5YTSNJ2VXANCNFSM4TBN6P7Q .
hi @usingcolor ! I wonder how many images were used in your training/validation? thanks!
hi @usingcolor ! I wonder how many images were used in your training/validation? thanks! please make a new issue next time.
For the Question, I used the dataset that I mentioned in README.md (i.e. FFHQ, CeleabA-HQ and VGG face) You can add the face images as you want.
Thanks for the implementation! Well done! I am about to use your code to train my own model. I am curious how long it took your model to be trained? How many epochs did you use?
Thanks!