There seems to be a problem with the effect of MotionTransfer in HumanViewSynthesis

[mioyeah]

The test output generated by the View Synthesis is normal. Source Image Novel View But there is a problem with the perspective of the test output of Motion Transfer. Appearance Motion Output The parameters I entered into motion_transfer.py are
--ckpt experiments/NovelViewSynthesis/Panoptic/256x256/LPIPS/model.ckpt \ --root ./datasets/Panoptic \ --sequence_motion 171026_pose1/Subsequences/0 \ --in_view_motion 00 \ --out_view_motion 24 \ --interval_motion 1000 1500 \ --sequence_appearance 171026_pose1/Subsequences/3 \ --in_view_appearance 00 \ --interval_appearance 3000 3000 \ --output_dir experiments/NovelViewSynthesis/Panoptic/256x256/LPIPS/MotionTransfer What is the cause of this problem?

[GuillaumeRochette]

Hi @mioyeah,

There is no problem with the code actually.

The subject you have used to extract the appearance was never seen during training. Our method can generalize about new poses and view angles but not about unseen appearances, as there were only 28 subjects whose appearances were used during training, and 8 subjects who were left out for experiments like motion transfer. This limited number of subjects for training leads to a strong overfit w.r.t the appearance, and this is why you see a completely different human, which probably relates to a combination of humans whose appearances were seen during training.

I invite you to look at this page, and more specifically KNOWN_SEQUENCES and UNKNOWN_SEQUENCES to know which sequences to use for the motion transfer task, e.g. pick a known appearance, but an unknown motion, and the results should look much better.

Cheers,

Guillaume

PS: If you have other issues unrelated to the pre-processing code, but rather the training/inference code itself, please open them in the main repository

[mioyeah]

The problem is solved. The reason for the angle of view error is that I modified a parameter. Now the problem has been solved. Thank you very much for your answer.