Portrait video editing with stitching option ?

[3blackbar]

In their example you can load input video, and just drive the mouth and eyes in the video, so it works with full video not just one image https://liveportrait.github.io/ https://liveportrait.github.io/src/video/video_animation_0_opt.mp4

[kijai]

Either I'm missing something or they didn't include the code for this. Unless it's the template option which I don't yet understand.

[3blackbar]

ok, heres how it can be done, - in relative mode wth source video input under source image input, so instead of feeding the node the same image, how about you feed frame from another video per each new frame ? So you have video feed as input frames too, face aligner should detect where the face is per new frame. Turn off retageting and enable relative mode ,cause in "relative" mode its using whole input image and its using one image for entire video, im sure this can be changed so its using consecutive video frames instead of one image , so in theory it should be able to use different image per every frame and not just one image ?

With this working we will be able to change what people say on the videos.

[JZZZ1314]

I was able to perform a simple test, and by inferring about each frame of the edited video by talking about each frame of the driving video, I was able to get an effect close to that of the paper，Of course, this needs to wait for the official release of the full code, which is only experimental

[ozworkz]

ok, heres how it can be done, - in relative mode wth source video input under source image input, so instead of feeding the node the same image, how about you feed frame from another video per each new frame ? So you have video feed as input frames too, face aligner should detect where the face is per new frame. Turn off retageting and enable relative mode ,cause in "relative" mode its using whole input image and its using one image for entire video, im sure this can be changed so its using consecutive video frames instead of one image , so in theory it should be able to use different image per every frame and not just one image ?

With this working we will be able to change what people say on the videos.

Yes, with some tweaks in the code, reading each frame from the video input as a source image seems to work. Thanks!

[3blackbar]

Can You guys do pull request if You succesfully did it ?

[Trimad]

ok, heres how it can be done, - in relative mode wth source video input under source image input, so instead of feeding the node the same image, how about you feed frame from another video per each new frame ? So you have video feed as input frames too, face aligner should detect where the face is per new frame. Turn off retageting and enable relative mode ,cause in "relative" mode its using whole input image and its using one image for entire video, im sure this can be changed so its using consecutive video frames instead of one image , so in theory it should be able to use different image per every frame and not just one image ? With this working we will be able to change what people say on the videos.

Yes, with some tweaks in the code, reading each frame from the video input as a source image seems to work. Thanks!

Can you post your ComfyUI workflow please?

[serkandyck]

ok, heres how it can be done, - in relative mode wth source video input under source image input, so instead of feeding the node the same image, how about you feed frame from another video per each new frame ? So you have video feed as input frames too, face aligner should detect where the face is per new frame. Turn off retageting and enable relative mode ,cause in "relative" mode its using whole input image and its using one image for entire video, im sure this can be changed so its using consecutive video frames instead of one image , so in theory it should be able to use different image per every frame and not just one image ? With this working we will be able to change what people say on the videos.

Yes, with some tweaks in the code, reading each frame from the video input as a source image seems to work. Thanks!

Can you post your ComfyUI workflow please?

not perfect only 32 frame

https://gist.github.com/serkandyck/f937a540ca60fce8843d6c4242bfd869

[cheezecrisp]

There is a dumb way to do this. Put the node 'image batch to image list'(Impack Pack) before source_image and put the node 'image list to image batch' after output. With this setting the workflow will produce a matrix of images, then pick up the images on the diagonal of the matrix, those images are exactly the frames what we want. But it's very inefficient, it will take much time to produce the matrix. Wanna know if there is a better way.

[kijai]

The video editing is mostly working in the develop branch already, just some bugs to squash and optimizations to do.

[skydam]

I can only see Source Image as input for the node (in Develop branch), no video option yet. What am I missing?

[kijai]

I can only see Source Image as input for the node (in Develop branch), no video option yet. What am I missing?

The version in the develop branch takes both single image or a batch of images.

[skydam]

Thanks! I'm looking at the new node, but a. what's the input node for crop_info? That's a new one, and b. which input node would one use as batch input for source image? BTW love your work!

[skydam]

Sorry, beginner here. I'm seeing crop_info = self.cropper.crop_single_image(img_rgb), but Cropper seems commented out in #from .utils.cropper import Cropper

[kijai]

Sorry, beginner here. I'm seeing crop_info = self.cropper.crop_single_image(img_rgb), but Cropper seems commented out in #from .utils.cropper import Cropper

It's a whole new structure in that version, there's separate node that does the detection and cropping, it outputs the crop info for the main node. You input your source to the cropper node, it can be single image or batch of images, then you input the driving video to the process node.

[skydam]

Ah I see! Yes, found it: LivePortraitCropper. Thanks!

[skydam]

It's working! The cropped images work fine. The full images is still a mess for me though. Not quite sure if I'm doing it wrong or if it's still work in progress.

[skydam]

https://github.com/kijai/ComfyUI-LivePortraitKJ/assets/73529516/188c4279-8ea7-49a9-9e8b-fba82f1a8109

[Maelstrom2014]

LivePortrait_00008.mp4

Please post example! or screenshot of workflow.

[skydam]

https://github.com/kijai/ComfyUI-LivePortraitKJ/assets/73529516/25f0c47d-c4e9-46d8-91b5-3df0f2829f20

This is what comes out of full_images, whereas the cropped (stabilised) images are as above.

[skydam]

[kijai]

What are you feeding for the source_image there? it should be the input video, not the crop

[skydam]

THAT WAS IT! I fed the cropped image into source.

https://github.com/kijai/ComfyUI-LivePortraitKJ/assets/73529516/10b1ae04-013d-4ab7-a2c5-6ac27b8d301e

[Maelstrom2014]

need also a mask for pasting image.

[Maelstrom2014]

LivePortraitCropper still use CPU, takes 160sec, too long. I does not matter that i put CUDA in menu

[kijai]

LivePortraitCropper still use CPU, takes 160sec, too long. I does not matter that i put CUDA in menu

Possibly you don't have onnx installed for GPU? Do you have onnxruntime-gpu installed? It's sometimes bit of a pain to get to work, for reference on my 4090 it's about 33it/s.

[skydam]

I have 2.75it/s on my 3090/24gb