CCpt5
commented
5 months ago There's a blurb at the bottom of this model card (by the guy who converted the SDXL inpainting beta model to safetensors) which mentions how to merge it to make any models inpaint models. I guess a base SDXL model w/ the VAE baked in needs to be used: https://huggingface.co/benjamin-paine/sd-xl-alternative-bases
wkpark
The developer branch of A1111 allows for use of SDXL inpaint models. SAI quietly released a very early beta inpaint model for SDXL months ago in diffuser format, and I found a safetensor conversion of it hosted here:
Card: https://huggingface.co/wangqyqq/sd_xl_base_1.0_inpainting_0.1.safetensors Safetensor: https://huggingface.co/wangqyqq/sd_xl_base_1.0_inpainting_0.1.safetensors/tree/main
I've been using it w/ this simple outpainting extension today (https://github.com/Haoming02/sd-webui-mosaic-outpaint) and it's worked pretty well. I was hoping to try merging it w/ some of my dreambooth (kohya) trained models but am getting the following error: RuntimeError: The size of tensor a (9) must match the size of tensor b (4) at non-singleton dimension 1 - (See full console in details)
Is there a fundamental issue w/ this beta SDXL inpainting model that would prevent it from being compatible for merging w/ other SDXL models? Or is there a way the model could be mixed w/ "normal" SDXL models (a la how you can do that w/ v1.5's inpainting model) to allow for better inpainting.
Thanks for any insight!!
Details
venv "D:\stable-diffusion-webui\venv\Scripts\Python.exe" Python 3.10.6 (tags/v3.10.6:9c7b4bd, Aug 1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)] Version: v1.7.0-437-gce168ab5 Commit hash: ce168ab5dbc8b54b7245f352a2eaa55a37019b91 Launching Web UI with arguments: --opt-sdp-attention --no-half-vae --opt-channelslast --skip-torch-cuda-test --skip-version-check --ckpt-dir e:\Stable Diffusion Checkpoints No module 'xformers'. Proceeding without it. *** Extension "sd-webui-lama-cleaner-masked-content" requires "sd-webui-controlnet" which is not installed. ControlNet preprocessor location: D:\stable-diffusion-webui\extensions\3sd-webui-controlnet\annotator\downloads 2024-02-01 14:36:46,452 - ControlNet - INFO - ControlNet v1.1.440 2024-02-01 14:36:46,552 - ControlNet - INFO - ControlNet v1.1.440 [-] ADetailer initialized. version: 24.1.2, num models: 9 Tag Autocomplete: Could not locate model-keyword extension, Lora trigger word completion will be limited to those added through the extra networks menu. Loading weights [fe1b97fe65] from e:\Stable Diffusion Checkpoints\SDXL\sd_xl_base_1.0_inpainting_0.1.safetensors Creating model from config: D:\stable-diffusion-webui\configs\sd_xl_inpaint.yaml Loading VAE weights specified in settings: D:\stable-diffusion-webui\models\VAE\sdxl_vae.safetensors Applying attention optimization: sdp-no-mem... done. Model loaded in 2.7s (create model: 0.2s, apply weights to model: 2.0s). 2024-02-01 14:36:51,003 - ControlNet - INFO - ControlNet UI callback registered. Running on local URL: http://127.0.0.1:7860 To create a public link, set `share=True` in `launch()`. Startup time: 11.6s (prepare environment: 0.6s, import torch: 2.6s, import gradio: 0.7s, setup paths: 0.7s, initialize shared: 0.2s, other imports: 0.5s, list SD models: 0.2s, load scripts: 2.1s, refresh VAE: 0.1s, create ui: 3.4s, gradio launch: 0.4s). img2img: topnotch artstyle debugs = [] use_extra_elements = True - mm_max_models = 7 config hash = 2733d0e96220606eb11a46bc48f171b0c6c24c67b4b6e2cdcad8e7bc11cbf64d - mm_use [True, False, False, False, False, False, False] - model_a SDXL\sd_xl_base_1.0_inpainting_0.1.safetensors [fe1b97fe65] - base_model None - max_models 7 - models ['SDXL\\2023-11-28 - Topnotch Artstyle (Mostly Ruler-Math) - RealVision Try - 17img (20repeats) - 900reg - b2 - 4000max-step00005400.safetensors'] - modes ['Sum'] - calcmodes ['Normal'] - usembws [[]] - weights ['0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5'] - alpha [0.5] - adjust - use elemental [False] - elementals [''] model_a = SDXL_sd_xl_base_1.0_inpainting_0.1 Loading SDXL\sd_xl_base_1.0_inpainting_0.1.safetensors [fe1b97fe65] from loaded model... - loading script.patches... - base lora_patch Applying attention optimization: sdp-no-mem... done. isxl = True , sd2 = False compact_mode = False - check possible UNet partial update... - partial changed blocks = ['BASE', 'IN00', 'IN01', 'IN02', 'IN03', 'IN04', 'IN05', 'IN06', 'IN07', 'IN08', 'M00', 'OUT00', 'OUT01', 'OUT02', 'OUT03', 'OUT04', 'OUT05', 'OUT06', 'OUT07', 'OUT08'] - UNet partial update mode Loading model SDXL_2023-11-28 - Topnotch Artstyle (Mostly Ruler-Math) - RealVision Try - 17img (20repeats) - 900reg - b2 - 4000max-step00005400... Loading from file e:\Stable Diffusion Checkpoints\SDXL\2023-11-28 - Topnotch Artstyle (Mostly Ruler-Math) - RealVision Try - 17img (20repeats) - 900reg - b2 - 4000max-step00005400.safetensors... Calculating sha256 for e:\Stable Diffusion Checkpoints\SDXL\2023-11-28 - Topnotch Artstyle (Mostly Ruler-Math) - RealVision Try - 17img (20repeats) - 900reg - b2 - 4000max-step00005400.safetensors: 3183e182eb1a9439f63712afe40c9933a3f20da78655647392b1e337199e278f mode = Sum, alpha = 0.5 Stage #1/2: 0%| | 0/2255 [00:00