search

comfyanonymous / ComfyUI

The most powerful and modular diffusion model GUI, api and backend with a graph/nodes interface.
https://www.comfy.org/
GNU General Public License v3.0
58.39k stars 6.2k forks source link

Support SDXL T2IAdapters #1236

Open n00mkrad opened 1 year ago

n00mkrad commented 1 year ago

For example: https://huggingface.co/SargeZT/t2i-adapter-sdxl-multi

Would be great to get this working especially since T2IA models are much smaller (and I assume faster) than ControlNets.

comfyanonymous commented 1 year ago

These are not in a standard format so I feel like a script that renames the keys would be more appropriate than supporting it directly in ComfyUI. Once the keys are renamed to ones that follow the current t2i adapter standard it should work in ComfyUI.

doctorpangloss commented 1 year ago

Just an FYI, these are the keys:

["adapter.body.0.resnets.0.block1.bias", "adapter.body.0.resnets.0.block1.weight", "adapter.body.0.resnets.0.block2.bias", "adapter.body.0.resnets.0.block2.weight", "adapter.body.0.resnets.1.block1.bias", "adapter.body.0.resnets.1.block1.weight", "adapter.body.0.resnets.1.block2.bias", "adapter.body.0.resnets.1.block2.weight", "adapter.body.1.in_conv.bias", "adapter.body.1.in_conv.weight", "adapter.body.1.resnets.0.block1.bias", "adapter.body.1.resnets.0.block1.weight", "adapter.body.1.resnets.0.block2.bias", "adapter.body.1.resnets.0.block2.weight", "adapter.body.1.resnets.1.block1.bias", "adapter.body.1.resnets.1.block1.weight", "adapter.body.1.resnets.1.block2.bias", "adapter.body.1.resnets.1.block2.weight", "adapter.body.2.in_conv.bias", "adapter.body.2.in_conv.weight", "adapter.body.2.resnets.0.block1.bias", "adapter.body.2.resnets.0.block1.weight", "adapter.body.2.resnets.0.block2.bias", "adapter.body.2.resnets.0.block2.weight", "adapter.body.2.resnets.1.block1.bias", "adapter.body.2.resnets.1.block1.weight", "adapter.body.2.resnets.1.block2.bias", "adapter.body.2.resnets.1.block2.weight", "adapter.conv_in.bias", "adapter.conv_in.weight"]

from https://github.com/AMorporkian/T2IAdapter-SDXL-Diffusers/tree/master

which appears to be diffusers format

doctorpangloss commented 1 year ago 
Error(s) in loading state_dict for Adapter:
    Missing key(s) in state_dict: "body.6.block1.weight", "body.6.block1.bias", "body.6.block2.weight", "body.6.block2.bias", "body.7.block1.weight", "body.7.block1.bias", "body.7.block2.weight", "body.7.block2.bias".
doctorpangloss commented 1 year ago

it's a custom format, nvm