cubiq / ComfyUI_IPAdapter_plus

GNU General Public License v3.0
3.67k stars 278 forks source link

channel penalty(Fooocus) different results #70

Closed liangkun43 closed 6 months ago

liangkun43 commented 9 months ago

This is my reference image. 273487475-0f81b24d-e55d-4e4d-b82c-336edb194c2f use ip-adapter-plus_sdxl_vit-h model Positive text:photograph dragon, 50mm . cinematic 4k epic detailed 4k epic detailed photograph shot on kodak detailed cinematic hbo dark moody, 35mm photo, grainy, vignette, vintage, Kodachrome, Lomography, stained, highly detailed, found footage Negative text:Brad Pitt, bokeh, depth of field, blurry, cropped, regular face, saturated, contrast, deformed iris, deformed pupils, semi-realistic, cgi, 3d, render, sketch, cartoon, drawing, anime, text, cropped, out of frame, worst quality, low quality, jpeg artifacts, ugly, duplicate, morbid, mutilated, extra fingers, mutated hands, poorly drawn hands, poorly drawn face, mutation, deformed, dehydrated, bad anatomy, bad proportions, extra limbs, cloned face, disfigured, gross proportions, malformed limbs, missing arms, missing legs, extra arms, extra legs, fused fingers, too many fingers, long neck This is the result in fooocus. 微信截图_20231108211713 This is the result of channel penalty.(including workflow) workflow2 This is the result I got from this branch. workflow

cubiq commented 9 months ago

fooocus and the other repo both use an implementation that in my opinion is not ideal and too far from the IPAdapter intended behavior. In fact at a strength 0.6 it makes sense that the bird is still very visible while in the other implementations has basically disappeared.

My theory is confirmed by the fact that the original IPAdapter implementation is closer to the "channel penalty" than it is the "fooocus" one.

I haven't checked but I also believe that the other comfyui implementation doesn't work with multiple images (or at least the result would be very unpredictable).

I'm open to discussion.

liangkun43 commented 9 months ago

Thank you very much for your work.Below are the results of the two reference pictures,The results are indeed unpredictable 微信截图_20231108222014 two picture workflow two picture workflow2 "channel penalty" image are very bad,"linear" and "original"result combines the features of the two images two picture workflow3 two picture workflow4 Maybe some people just need an option to accomplish the same result in other UI

liangkun43 commented 9 months ago

An interesting thing seems to have happened during my testing, I don't know what happened, but got good results two picture workflow5 Cancel the queue when the "apply IPAdapter" is loaded, and then reconnect to another node,queue prompt 微信截图_20231108233828

cubiq commented 9 months ago

if you add a little "noise" it usually helps to stabilize the composition

xiaohu2015 commented 9 months ago

I think Fooocus also stop early (at step 0.4) for image prompt, which is important?

cubiq commented 9 months ago

I think Fooocus also stop early (at step 0.4) for image prompt, which is important?

I haven't checked the full code. The way fooocus merges the embeds is completely different to how I do it. In foooocus implementation the text prompt has a very strong influence but if they stop at 0.4 that might explain why the results are widely different.

Timestepping is something that I wanted to do as well but it can be already simulated stopping the generation at X step and resuming it with a second pipeline. It can be done both with diffusers and comfyui

This is the generation with IPAdapter stopped at 50% of the generation: image

this is stopped at 60% image

and so on

hben35096 commented 9 months ago

Request add support for start _step and end _step.

ttulttul commented 9 months ago

Request add support for start _step and end _step.

Your wish has now been granted.

hben35096 commented 9 months ago

Request add support for start _step and end _step.

Your wish has now been granted. Thank you very much, this is a very useful feature.