search

advimman / lama

🦙 LaMa Image Inpainting, Resolution-robust Large Mask Inpainting with Fourier Convolutions, WACV 2022
https://advimman.github.io/lama-project/
Apache License 2.0
8.06k stars 855 forks source link

Batch/Video Processing #97

Closed Echolink50 closed 2 years ago

Echolink50 commented 2 years ago

I was looking to install this. I was wondering if it supports some way to do a batch of images from a sequence or even do a video? Or do I have to manual highlight the object i want removed in each image? Thanks everyone.

windj007 commented 2 years ago

LaMa was designed for image inpainting. Technologies for video inpainting significantly differ from those for separate images, because in videos on the one hand you need temporal consistency and on the other you have more information - regions of different frames can be reused.

LaMa can be used for videos, but the quality will probably be inferior compared to designated video inpainting methods. The most straightforward way to use existing LaMa codebase to process video is:

  1. Split video to separate frames and store frames to a folder, with something like ffmpeg -i video.mp4 video_frames/%05d.jpg. Somehow add masks for each frame to the same folders.
  2. Run prediction with LaMa - its current codebase can process all images in a folder in a single run.
  3. Merge inpainted images into a video using e.g. ffmpeg.

Note that you need to specify a mask for each separate frame. There is no functionality for object segmentation and tracking in videos in LaMa.

There is a third-party colab with CLIP-based segmentation and LaMa for video inpainting, which is based on the similar idea (twitter post and video, colab)

There are designated video completion methods, e.g. Flow-edge guided video completion.

Echolink50 commented 2 years ago

Thank you so much for a very clear and informative answer.