XLabs FLUX IP-Adapter v2 - Githubissues

invoke-ai / InvokeAI

Invoke is a leading creative engine for Stable Diffusion models, empowering professionals, artists, and enthusiasts to generate and create visual media using the latest AI-driven technologies. The solution offers an industry leading WebUI, and serves as the foundation for multiple commercial products.

https://invoke-ai.github.io/InvokeAI/

Apache License 2.0

23.77k stars 2.44k forks source link

XLabs FLUX IP-Adapter v2 #7338

Closed RyanJDick closed 1 week ago

RyanJDick commented 1 week ago

Summary

This PR adds support for the XLabs FLUX IP-Adapter v2 model (https://huggingface.co/XLabs-AI/flux-ip-adapter-v2).

Known v1 regression

The recommended inference configuration for XLabs IP v2 is different than that for v1:

v1 works well with cfg_start_step=1 and cfg_scale=3
v2 works well with cfg_scale=1 (i.e. no CFG)

This PR updates the Linear UI behavior to optimize for v2. As a result, the performance of v1 models in the Linear UI will be significantly degraded.

It is still possible to use the v1 models by manually constructing a workflow like the one shown on the original IP-Adapter v1 PR: https://github.com/invoke-ai/InvokeAI/pull/7157

Example 1:

Promp: a statue wearing sunglasses
ip-adapter weight: 0.8
In order (left to right, top to bottom):
- Original
- IP-Adapter v2
- IP-Adapter v1 degraded after this PR
- IP-Adapter v1 before this PR

Example 2:

Prompt: A hooded warrior shown from behind overlooking a vibrant underwater sea world. There are sea turtles and colorful fish swimming around.
ip-adapter weight: 0.8
In order (left to right, top to bottom):
- Original
- IP-Adapter v2
- IP-Adapter v1 degraded after this PR
- IP-Adapter v1 before this PR

QA Instructions

[x] Install XLabs IP-Adapter v2 from the Starter Models tab.
[x] Run the XLabs IP-Adapter v2 model from the Linear UI.
[x] The XLabs IP-Adapter v1 model can still be used in the Linear UI, but with severely degraded performance.
[x] Smoke test SD-1 and SDXL IP-Adapters

Checklist

[x] The PR has a short but descriptive title, suitable for a changelog
[x] Tests added / updated (if applicable)
[x] Documentation added / updated (if applicable)
[ ] Updated What's New copy (if doing a release after this PR)