williamyang1991 / VToonify

[SIGGRAPH Asia 2022] VToonify: Controllable High-Resolution Portrait Video Style Transfer
Other
3.53k stars 446 forks source link

kee background area while stylizing portrait area #56

Closed leedewdew closed 1 year ago

leedewdew commented 1 year ago

Thanks for your great job. I have a question about how to keep background area while stylizing portrait area? Looking for your reply.

williamyang1991 commented 1 year ago

You can use segmentation prediction networks and blend the results with the originla frame,

leedewdew commented 1 year ago

Good idea. Bu it works when the mask of generated image is larger than that of original image, while in the other case(the former is smaller), it maybe won't.

williamyang1991 commented 1 year ago

I cannot understand you question. The output image (and the mask) should be always no larger than the whole image. Our method does not create new content, but only relies on the input image.

leedewdew commented 1 year ago

Sorry, I means when the portrait mask of stylized image is smaller than the human mask of the original image, directly use stylized segmentation mask doesn't match the size of the mask of the original image.

williamyang1991 commented 1 year ago

Can you give some image example of what the portrait mask/human mask/stylized segmentation mask/mask of the original image look like? There are many masks in your description.

leedewdew commented 1 year ago

You can use segmentation prediction networks and blend the results with the originla frame,

Human mask of original image and stylized image don't match in size.

williamyang1991 commented 1 year ago

Maybe you can use a dilated stylized mask, and smooth the mask boundary to alliviate the mismatches of the two masks

leedewdew commented 1 year ago

Ok, Thank you.