How to process two canonical images

[YinzhenWang]

Thank you for doing such a great job!

I want to use controlnet for video style migration. For your example beauty_1, the step of reconstruction generates two canonical images. If I simply use the controlnet to transfer these tow images, the generated video is bad even with a same controlnet seed. May I ask which step is wrong with me？

Thank you!

[ken-ouyang]

Can you show the canonical image, translated canonical image and the translated video? I cannot decide which step is wrong as the information is too limited.

[YinzhenWang]

Thank you for your reply.

I follow the reconstruction step and get two canonical images like below. And then I use controlnet with the same seed to generate images like below. The dual video is

https://github.com/qiuyu96/CoDeF/assets/43692893/26d5c5f2-cc3b-4f66-8790-5895f68106a3

I think it is due to the inconsistency between two generated images

[ken-ouyang]

The issue you're encountering is likely due to the translated image not being perfectly aligned with the original one. To address this, you might want to try the following solutions:

1. Consider using a model that promotes better alignment, such as a lineart or animelineart model. These models are explicitly designed to preserve the structure of the original image in the translated one and may offer improved alignment.
2. Another approach could be to focus solely on the foreground for inference. This can be achieved by setting the masks to null during the inference process.

[YinzhenWang]

Thank you for your advice. I will try it.

[cslvjt]

Hello, may I know which model of ControlNet you are referring to？ Thank you.

[YinzhenWang]

openpose