search

TIGER-AI-Lab / AnyV2V

Code and data for "AnyV2V: A Tuning-Free Framework For Any Video-to-Video Editing Tasks" (TMLR 2024)
https://tiger-ai-lab.github.io/AnyV2V/
MIT License
508 stars 38 forks source link

Adaptation for inpainting #4

Closed alasokolova closed 7 months ago

alasokolova commented 7 months ago

Hey, Can AnyV2V be adapted for video inpainting task?

wren93 commented 7 months ago

Hi, thanks for your interest. We haven't thoroughly tested the framework for video inpainting, but I guess the method generally makes sense for inpainting tasks such that given a masked video, you can first use some image inpainting method to generate the first inpainted frame and let the I2V model propagate the frame to the entire video. We do have some examples for object swapping which should also be related to inpainting.

vinesmsuic commented 7 months ago

We did try some tasks similar to inpainting (e.g. Adding objects and Removing objects) and found that such modifications are even better with such tasks compared to frame-based methods due to the inherent knowledge from I2V models.

alasokolova commented 7 months ago

Thanks for the answers! @vinesmsuic did you try it the way @wren93 described? Applying image-based model to the first frame and then propagating to the whole video? I will try it :)

vinesmsuic commented 7 months ago

We tried using instructpix2pix to add/remove objects in the first frame. Of coz we were talking about the AnyV2V method.