Support for AUTOMATIC1111/stable-diffusion-webui and InvokeAI/InvokeAI via proxy engine mode

[remixer-dec]

Greetings! This makes the photoshop plugin compatible with two popular stable diffusion backends. They are optimized differently and have some features, that raw diffusers module does not have + they are easier to install and can handle ckpt files.

In this PR I moved utils and engines into separate files, divided the workflow for each engine into run, prepare_args and process methods, but tried to keep most of the original non-proxy code untouched. I only added "universal" task engine support, since separating tasks does nothing in proxy mode.

Everything seems to be working correctly, but some features are missing in the plugin. For example sampler selection and negative prompts. In InvokeAI negative prompt can be used right in the regular prompt with square brackets syntax, but in A1111 they mean a different thing (word attention reduction), so there is currently no way to use negative prompt. Maybe some other syntax like splitting two linebreaks ('\n\n') can be used as alternative? Sampler names are also different, but If you decide to add sampler selection into the plugin, it can be somehow mapped later.

P.S. Currently in this branch proxy engines output nsfw: False hardcoded, since there is no data from the base server about the output. InvokeAI is compatible in legacy mode because they moved to websocket/session protocols, but later brought back legacy api support. Also I saw "masking" task in the code, but I have no idea how to test it in Photoshop.

Thank you for making the original plugin and hopefully this can help to make it better :)

[cantrell]

This is awesome. Thanks for the PR. Apologies for the slow response. I'll dig in over the holiday weekend.

[benjamind]

Any chance of reviving this @cantrell?

It was previously working for me, but I updated the PS plugin and it seems to have broken something, getting a Runtime Error now. Tried a quick rebase of this PR against latest but no joy. Would be really good to get this in, having to have diffusers copies of all my models as well as the ckpt files in Automatic1111 really eats some disk space!

Not to mention its pretty handy to be able to have the automatic1111 gui running and jump between using the PS plugin and Automatic depending on what I'm doing. Having to restart one server or the ui between each switch is annoying.

[keturn]

Notes from the InvokeAI side:

• InvokeAI is migrating to use the diffusers library, so you'll be able to use the same set of models without converting anything to the CompVis/ldm format.
• Relying on anything called legacy (like scripts/legacy_api.py) is probably not a good plan for forward compatibility.

[remixer-dec]

@keturn 1) Currently most custom models are distributed as a single ckpt/safetensors file, does this migration mean that convertion to diffusers format will be required for use of custom models with InvokeAI? 2) Is the new websocket/session-based API documented somewhere? I understand that it is convenient for InvokeAI's interactivity, but for third party integrations REST is a well established industry standard which can be integrated anywhere without much pain.

[DrMacabre]

im having some trouble running this, it keeps asking for the huggingface token. why is it even asking for it if its working directly with a1111, did i miss something?

[remixer-dec]

@DrMacabre make sure you have "mode": "proxy" in config.json

[DrMacabre]

@DrMacabre make sure you have "mode": "proxy" in config.json

you mean like this ? without hf_token ?

[remixer-dec]

@DrMacabre like this (without ...), no token required

[isben]

@cantrell I understand things are evolving quite fast over the last few months and it may be difficult to predict where everything is heading. With the recent addition of Adobe's "Generative fill" and other recent features, you may have considered that there is no more need for an interface with Stable Diffusion, but I beg to differ and I'd like to assure you that the Photoshop plugin is definitely something that many people would love to use. Any chance to get a direct interfcae with Automatic1111 in the near future?

[cantrell]

We're trying to work out whether we should interface with A1111 or emulate the Stable Diffusion REST API.

[remixer-dec]

@isben here are some alternative plugins that are functional and compatible with a1111: stable.art and APSDP