how to do controlnet reference?

[FemBoxbrawl]

I don't know how to do reference_only or something like that, is there a way to implement it if not done already?

[mcmonkey4eva]

"controlnet reference" isn't actually a controlnet, it's just merged into that extension for auto webui --- to use in swarm you'd need to find a Comfy node for it, which last i looked iirc there's only experimental/hacky ones not great options. See also earlier thread here https://github.com/Stability-AI/StableSwarmUI/issues/176

[Sisimshow]

I too would be interested in this. It does work for Stability Matrix Inference interface (which uses ComfyUI) so it can be done. The results aren't the same as using Revision or IPAdapter, I'd say for replicating, say, the outfit of a character without a Lora, it's much better. This is one of the last things keeping me from jumping from A1111/Forge to StableSwarm because I use it for making variants of my D&D character art.

All that said, I don't want to sound overly assertive, just wanted to put out there that there is some demand for this feature and it's not simply an outdated version of Revision or IPadapter.

[mcmonkey4eva]

It does work for Stability Matrix Inference interface (which uses ComfyUI) so it can be done do you have a reference to the comfy node they used for it?

[Sisimshow]

Yes, Inference_Core_ReferenceOnlySimple. Here is a picture made with Stability Matrix using Reference Only and put into ComfyUI. workflow.json

[Sisimshow]

Sorry for the anime

[mcmonkey4eva]

Inference_Core_ReferenceOnlySimple appears to correspond to https://github.com/LykosAI/ComfyUI-Inference-Core-Nodes/blob/main/src/inference_core_nodes/comfyui_experiments/reference_only.py which is Inference's wonky broken import of https://github.com/comfyanonymous/ComfyUI_experiments/blob/master/reference_only.py which as noted in the prior thread didn't seem to generally actually work as intended last I tested it

I guess I'll have to re-test

[Sisimshow]

I appreciate the effort :) I hope this doesn't waste too much of your time good man

Edit

It at least seems to work similarly to how it does in Forge using the ControlNet extension, though I can't get an exact 1 to 1 output, just very similar.

[mcmonkey4eva]

I'm very confused what happened last year when trying it then, but it indeed works now!

I've enabled ReferenceOnly via comfy's experiment code (slightly adapted), with a new Use Reference Only param under ReVision (ie drag an image to prompt box and it will be included as an option)

[Sisimshow]

Niceness! Thank you so much!