ssitu / ComfyUI_fabric

ComfyUI nodes based on the paper "FABRIC: Personalizing Diffusion Models with Iterative Feedback" (Feedback via Attention-Based Reference Image Conditioning)
GNU General Public License v3.0
84 stars 6 forks source link

Can Fabric work with more than one positive or negative latent feedback? #9

Open EricRollei opened 11 months ago

EricRollei commented 11 months ago

ChrisGoringe's Image Picker / Chooser and Use Everywhere (UE) node sets enabled an interesting workflow for use with Fabric. We have been testing this today and I built a 4 round Fabric workflow with his multi-latent chooser node. that allows the workflow to pause after each step until the user selects the positive and negative images from a batch - then those positive and negative latents are sent to the next round as feedback to the Fabric sampler. I got some strange results playing with this today where it was almost like going further away from the feedback - like exactly reversed. But I wonder if it's not possible to send more than one latent to either pos or neg latent feedback?
I'm attaching the workflow in a zip in case you want to check it out, but you'll need the UE and Image picker node sets from ChrisGoringe https://github.com/chrisgoringe/cg-image-picker https://github.com/chrisgoringe/cg-use-everywhere

Fabric-4-round-iterative-UEnodes.zip

Also here are some screen grabs to show you it as well. First is of the basic round where you see the image chooser and multi-latent nodes that allow the user to select the pos and neg, then hit "go" to advance to the next round. You can also see the pos and neg feedback choices. The prompt was a "beautiful landscape of fall trees, a meadow with a stream running and a foot bridge that crosses it. "

image

here's what was generated for Round 2 choices - way crazy saturation and further away from the start

image

and round 3 even further. It's almost like the positive and negative feedback was reversed.

image

But my guess is that this doesn't work with more than one latent feed back and I wondered if that was true and if not why not? Could it be made to take more than one feedback image?

chrisgoringe commented 11 months ago

I think you can send multiple latents to the pos and neg.

I just did a quick test, in which the seeds were all fixed throughout:

I note that in the tips, it says "Pay attention to the pos/neg weights. The default value of 1.0 is probably too high in most cases." so I wonder if that's the issue?

EricRollei commented 11 months ago

Yes, possibly the weights are an issue since in that workflow, I had no easy way to set them.

ssitu commented 11 months ago

The negative weight is better set to a smaller value. I found that negative images are taken very literally. So if you put a decent looking image in the negatives, you will get a very bad result if your weight is 1.0.

lzardy commented 10 months ago

I use batch image -> vae encode -> pos/neg latent.