search

ltdrdata / ComfyUI-Impact-Pack

Custom nodes pack for ComfyUI This custom node helps to conveniently enhance images through Detector, Detailer, Upscaler, Pipe, and more.
GNU General Public License v3.0
1.89k stars 183 forks source link

Possible Enhancement? #137

Closed AIMusicExperiment closed 1 year ago

AIMusicExperiment commented 1 year ago

I use lots of your nodes, and I think that you are nothing short of Genius when it comes to ComfyUI. My main thing is deforum-like Hybrid video animation. I like to use your face detailer and I think it is one of the best nodes on Comfy. Often the characters in my source video will be singing, and making facial expressions. Unfortunately the detailed lip and face movements get "lost in translation." Media pipe is capable of tracking these, but it is tricky and basically useless to apply before the detailer. My quandary, is if someone like you with your brilliant ways could isolate the face from the frame, like you do in the detailer, then apply media pipe to it for detection and finally apply the result to the conditioning for your detailer. To my knowledge no one has figured this out, and it often comes up as a discussion on the deforum discord. It certainly can't be done in A1111, but I believe that with the resources in ComfyUI you could make it happen. If you could figure out how to make your detailer do this, it would be beyond epic.

ltdrdata commented 1 year ago

It seems that utilizing the ControlNet preprocessor instead of Mediapipe to extract facial expression information from the original and then applying ControlNet to SEGS could be a necessary approach.

I will consider an extension that adds ControlNet conditioning to SEGS.

ltdrdata commented 1 year ago

https://github.com/ltdrdata/ComfyUI-extension-tutorials/blob/Main/ComfyUI-Inspire-Pack/tutorial/SEGSControlNet.md https://github.com/ltdrdata/ComfyUI-extension-tutorials/tree/Main/ComfyUI-Inspire-Pack/workflow

ltdrdata commented 1 year ago

I'd added MediaPipeFaceMeshToSEGS into Impact Pack. and MediaPipeFaceMeshDetectorProvider into Inspire Pack. They are depend on ControlNet Auxiliary Preprocessors.

mp

AIMusicExperiment commented 1 year ago

Thank you so much! this should really enhance the ability!

AIMusicExperiment commented 1 year ago

I've been working with trying to get this right for several hours, and I keep failing. I have put a SEGSpreview on the output and I can see that it is not analyzing my input correctly. Regardless of the settings I can't get the face centered, and with some setting it tiles the image in haphazard way. I sure that it is my setting that are wrong, but I can't figure them out. Could it be the lack of a BBOX input that is causing this? I would LOVE to get this working as it would really enhance my work flows. Please help me to figure out how to set this up properly!

ltdrdata commented 1 year ago

I've been working with trying to get this right for several hours, and I keep failing. I have put a SEGSpreview on the output and I can see that it is not analyzing my input correctly. Regardless of the settings I can't get the face centered, and with some setting it tiles the image in haphazard way. I sure that it is my setting that are wrong, but I can't figure them out. Could it be the lack of a BBOX input that is causing this? I would LOVE to get this working as it would really enhance my work flows. Please help me to figure out how to set this up properly!

mediapipe detects not only face it also detects parts of faces. if you want just detect face then turn off all options except face.

mediapipe.png

This is an odd eye workflow.

AIMusicExperiment commented 1 year ago

Turns out that I had my cropping adjustments incorrect and it is now working as prescribed! I cannot overstate how valuable this is to the core of the work that I do! And I am not, "blowing smoke up your ass!" This really is ground breaking. This allows ComfyUi to do what Deforum really can't. Now because I'm a needy little girl. . Ha Ha. . will you make one for hands?

AIMusicExperiment commented 1 year ago

I have been wondering: Can you use one segs detailer and one debug to do bothe face and hands? If so it would really cut down the node count for my workflows. . .

ltdrdata commented 1 year ago

You can use SEGSConcat.

AIMusicExperiment commented 1 year ago

where in the chain do I put that? Before the detailer I'm guessing

ltdrdata commented 1 year ago

where in the chain do I put that? Before the detailer I'm guessing

Sure. It makes multiple SEGS to single SEGS And you can separate then through SEGSFilters.

AIMusicExperiment commented 1 year ago

I'm testing this now, if I can make this work, in conjunction with the contronet that you did on faces, then we have just conquered the universe! Well, a little tiny part of the Universe anyway.

ltdrdata commented 1 year ago

I'm testing this now, if I can make this work, in conjunction with the contronet that you did on faces, then we have just conquered the universe! Well, a little tiny part of the Universe anyway.

Ah.. You should update to latest version both Impact Pack and Inspire Pack.

I'd fixed mediapipe bug last night.

AIMusicExperiment commented 1 year ago

I already did, it is always the first thing I do when designing! Having a little trouble, but I will probably get it all sorted out. . .Do I need both a SEGs Detailer and a Detailer Debug? Or is that redundant? I've been putting the init image into the detailer and the image to be corrected int to debug. . .

ltdrdata commented 1 year ago

I already did, it is always the first thing I do when designing! Having a little trouble, but I will probably get it all sorted out. . .Do I need both a SEGs Detailer and a Detailer Debug? Or is that redundant? I've been putting the init image into the detailer and the image to be corrected int to debug. . .

The output of SEGS Detailer comes in the form of SEGS. Detailer integrates this with the process of pasting SEGS onto the original image. If you're not going to perform separate processing with SEGS, there's no need to use SEGS Detailer.