ExponentialML / ComfyUI_VisualStylePrompting

ComfyUI Version of "Visual Style Prompting with Swapping Self-Attention"
Apache License 2.0
278 stars 8 forks source link

Can not get outcome as shown in the examples with the provided workflow #8

Open RudyB24 opened 7 months ago

RudyB24 commented 7 months ago

Hi. I am looking forward to using VSP, the examples shown in the papers look awesome, and look indeed better than the 'competition'. However ... I can not get the expected outcome. I installed as described, via git clone, then opened the provided workflow, in which I chose model realisticvison4, loaded an image, wrote 'purple fur' in the style prompt and 'dog' in the pos prompt, expecting to get a purple fur dog. Please see the image I attached. Can you point me in the proper direction to get it working?

Greetings, Ruud.

VSP problem

ExponentialML commented 7 months ago

Hey @RudyB24! Please refer to the discussion here, thanks!

RudyB24 commented 7 months ago

Thanks for your effort and the work and time you put in.

Unfortunately here with me the results have not changed after the update. I again tried the purple fur dog. Looking at the examples shown in the paper, like with the fire or the white clouds, I would expect to get a purple fur dog. The dog that I do get has only a hint of purple fur and is deformed, with two noses. See the attached image. As a comparison, there's also an IPAdapter output (at 1024 px)

Then there's this other thing. You combined the pos- and the style prompt into one new node. That is a pity. Before I was able to implement the VSP into my existing workflows, that may contain a Prompt Styler and may use Integrated Nodes. With this new combined node I can not simply add VSP, I have to do a lot more rework of the workflows.

I hope you'll return to the separate pos, neg, style prompts again in a next update.

Best regards, Rudy.

purple_fur_dog purple_fur_dog_IPAdapter

ExponentialML commented 7 months ago

Another PR has been pushed that updates the functionality. Could you check to see if it works for you?

RudyB24 commented 7 months ago

First, thanks for having separate prompts for style/reference and positive. It's also useful to have the style prompt separate when using an image analyzer > prompt generator like WD14 Tagger for the style/reference image.

The output images are getting better, but we're still nowhere near the images that are shown in the paper https://curryjung.github.io/VisualStylePrompt/

Images: VSP workflow used, 8 output images of purple fur dog, Japanese Geisha with VSP, Japanese Geisha with IPAdapter.

Best regards, Rudy (from https://www.youtube.com/playlist?list=PLyC6aoYnRBZbU7RDv3bvnXjDHTn1zoyuY)

Version20240323 Purple_Fur_Dogs Geisha_VSP Geisha_IPA

jesenzhang commented 7 months ago

After today's update, can not get "a dog made out of cloud" workflow (5)

jesenzhang commented 7 months ago

Please give more workflows with the images shown in the paper including thonse with controlnets.