search

soumik-kanad / diff2lip

Other
321 stars 38 forks source link

Information about GAN loss #27

Closed agokrani closed 3 months ago

agokrani commented 7 months ago

Hi,

Very interesting work. Thank you so much for open sourcing inference code. Since the training code is missing and I would assume that you guys aren't planning to release it. Can you please give more information on the Patch GAN loss was applied during training? Did you use the pretrained model or was it trained together in an end-to-end fashion?

soumik-kanad commented 7 months ago

The sequence discriminator was trained together from scratch during training.

agokrani commented 7 months ago

Thank you so much. Few more questions regarding this. In the paper, you have mentioned "where we use a PatchGAN [23] discriminator Dψ. This task requires more context than just two frames [4] but no optical flow [52]."

Can you explain why this requires more context than two frames? Also when passing input to the sequence discriminator, are you considering all 5 frames as a single sequence or all the frames are considered different sequence?

soumik-kanad commented 7 months ago

We use all the 5 frames as a sequence. It is fed to the discriminator by concatenating along the channel dimensions.