search

zhengkw18 / face-vid2vid

Unofficial implementation of the paper "One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing" (CVPR 2021 Oral)
155 stars 17 forks source link

There is a problem with the generated video, what is the cause? #6

Closed applech666 closed 3 weeks ago

applech666 commented 3 years ago

Thank you for your shared model! output

I used your pre-trained model. What's the problem?

thanks!

zhengkw18 commented 3 years ago

Which did you use, motion transfer or reconstruction? I guess there is problem with your source image.

applech666 commented 3 years ago

test source image. motion transfer

run command is :python evaluate.py --ckp=100 --source=005.jpg --driving=datasets/vox/test/id10343#TFIZ9vWg6EE#003826#003939.mp4

zhengkw18 commented 3 years ago

This is strange. You can try other source/driving to see which causes the problem. Several reasons are possible:

  1. driving has bad illumination
  2. source/driving face should be resized
  3. source jpg format image is not correctly read
  4. Asian people are out of the training dataset distribution so not properly modeled
applech666 commented 3 years ago

According to your suggestion, I changed the driving video and the original image, but the result is still terrible. generated: output drive video: 0000000 source image: test (2) run command: python evaluate.py --ckp=100 --source=test.png --driving=datasets/vox/test/id10070#FWHkEnS8v-M#001286#001433.mp4

applech666 commented 3 years ago

How do I understand it here?

1637216932(1)

zhengkw18 commented 3 years ago

Now I think this is due to environment problems such as Pytorch version...For some reasons the pretrained model or the image is not correctly loaded. The headpose estimator is used during training so it's irrelevant.

applech666 commented 3 years ago

I'm using torch==1.6. What about you?

zhengkw18 commented 3 years ago

You can update all your packages to latest version. Besides you can re-clone the repo to confirm there is no code change.

applech666 commented 3 years ago

output thanks @zhengkw18 I made a mistake in solving the problem: RuntimeError: Integer division of tensors using div or / is no longer supported, and in a future release div will perform true division as in Python 3. Use true_divide or floor_divide (// in Python) instead.