Apply part swapping on selected part of full size video, like DeepFake work?

AliaksandrSiarohin / motion-cosegmentation

Reference code for "Motion-supervised Co-Part Segmentation" paper

Other

653 stars 142 forks source link

Apply part swapping on selected part of full size video, like DeepFake work? #34

Open urmit-space opened 3 years ago

urmit-space commented 3 years ago

How can apply part swapping in this video on a particular portion.? I get a very unwanted result in the whole video. I know in this video we should detect the face in the full-size video then apply the mask on a particular part. maybe, I know the process but don't know where to start. or is that use different method can you please tell that. can you please take time and help with that!!!?

https://user-images.githubusercontent.com/68328018/104272873-148c6300-54c4-11eb-8e5a-9f66f4081d78.mp4

Iron man replace with this vin image

AliaksandrSiarohin commented 3 years ago

This is rather complicated, need to write code for frame detection and back Inpainting.

AliaksandrSiarohin commented 3 years ago

I guess it will be easier to try some exiting solution for this.

urmit-space commented 3 years ago

This is rather complicated, need to write code for frame detection and back Inpainting.

yes, it takes more time like 4-5 min to create a face-swapping. but get a good result. can you please help me out as you say the existing solution is easier? I don't know where to change it. if you tried before, let me give some help in the solution.

AliaksandrSiarohin commented 3 years ago

Slice the video into chunks using https://github.com/AliaksandrSiarohin/first-order-model/blob/master/crop-video.py
Also crop image Win Diesel using paint to make it a square image.
Run a face swap for all chunks from 1.
Write a script that take modified chunks and place them back to the original positions (positions is a part of the ffmpeg command).

urmit-space commented 3 years ago

OK let me try it. in this, we have to first detect the face in the video. divided video in chunks where a face appeared then crop the face portion. after these things applied by reverse sequence. is that correct? crop-video.py has face crop logic or have to do it myself.

AliaksandrSiarohin commented 3 years ago

Yes, crop video will produce you ffmpeg commands for cropping. Also you may need to tweek default parameters of crop-video, if you want to include all the chunks.

urmit-space commented 3 years ago

Ok, let me try once. then get to know what challenges are made from my side. will update on this soon.

urmit-space commented 3 years ago

I get success in this now all things work. here I attach a video. greatee bro!!

https://user-images.githubusercontent.com/68328018/105016113-44111180-5a68-11eb-91ce-558f94d9404b.mp4

but two things I want to change. how can make all these things automatic just one click and want to improve the result? and other I want to change crop box size like want exectly face crop so may result will improve. The last one how can we achieve this type of face swapping. first-order-model-Face-swap GiF I tried many times but not get exactly this type.

Hpjhpjhs commented 3 years ago

@urmit-space Hi, could you share the code about how to place the croped video back to the original positions (positions is a part of the ffmpeg command). It is appreciated If I can received your reply.

urmit-space commented 3 years ago

@Hpjhpjhs you get x, y, and size from the crop frame. you need to add an overlay video on the current video with respect frame.

!ffmpeg -i /content/Data/4.1_original_conan_obrien.mp4 -vf "movie=/content/Data/predictionscrop.mp4, scale=429: -1 [inner]; [in][inner] overlay =736: 130 [out]" /content/Data/completed.mp4

Maybe this will help.

Hpjhpjhs commented 3 years ago

@urmit-space Thank you so much. That's very helpful.