search

leejet / stable-diffusion.cpp

Stable Diffusion and Flux in pure C/C++
MIT License
3.54k stars 306 forks source link

Flux-controlnet #438

Open apepkuss opened 1 month ago

apepkuss commented 1 month ago

Hi guys,

I'm trying XLabs-AI/flux-controlnet-canny-v3 with stable-diffusion.cpp and run into the following error message:

[ERROR] model.cpp:1694 - tensor 'input_hint_block.10.bias' has wrong shape in model file: got [16, 1, 1, 1], expected [96, 1, 1, 1]
[WARN ] model.cpp:1592 - process tensor failed: 'input_hint_block.10.bias'
[ERROR] model.cpp:1710 - load tensors from file failed
[ERROR] control.hpp:449  - load control net tensors from model loader failed
new_sd_ctx_t failed

I am not sure if it was caused by the model itself. Anyone here ran into this issue? The entire log can be found below. Thanks a lot!

Log ```bash sd --diffusion-model flux1-dev-Q4_0.gguf --vae ae.safetensors --clip_l clip_l.safetensors --t5xxl t5xxl-Q8_0.gguf --control-net flux-canny-controlnet-v3.safetensors --control-image input_canny.png -p "cyberpank dining room, full hd, cinematic" -v Option: n_threads: 4 mode: txt2img model_path: wtype: unspecified clip_l_path: clip_l.safetensors t5xxl_path: t5xxl-Q8_0.gguf diffusion_model_path: flux1-dev-Q4_0.gguf vae_path: ae.safetensors taesd_path: esrgan_path: controlnet_path: flux-canny-controlnet-v3.safetensors embeddings_path: stacked_id_embeddings_path: input_id_images_path: style ratio: 20.00 normalize input image : false output_path: output.png init_img: control_image: input_canny.png clip on cpu: false controlnet cpu: false vae decoder on cpu:false strength(control): 0.90 prompt: cyberpank dining room, full hd, cinematic negative_prompt: min_cfg: 1.00 cfg_scale: 7.00 guidance: 3.50 clip_skip: -1 width: 512 height: 512 sample_method: euler_a schedule: default sample_steps: 20 strength(img2img): 0.75 rng: cuda seed: 42 batch_count: 1 vae_tiling: false upscale_repeats: 1 System Info: BLAS = 0 SSE3 = 1 AVX = 1 AVX2 = 1 AVX512 = 0 AVX512_VBMI = 0 AVX512_VNNI = 0 FMA = 1 NEON = 0 ARM_FMA = 0 F16C = 1 FP16_VA = 0 WASM_SIMD = 0 VSX = 0 [DEBUG] stable-diffusion.cpp:180 - Using CPU backend [INFO ] stable-diffusion.cpp:202 - loading clip_l from 'clip_l.safetensors' [INFO ] model.cpp:793 - load clip_l.safetensors using safetensors format [DEBUG] model.cpp:861 - init from 'clip_l.safetensors' [INFO ] stable-diffusion.cpp:209 - loading t5xxl from 't5xxl-Q8_0.gguf' [INFO ] model.cpp:790 - load t5xxl-Q8_0.gguf using gguf format [DEBUG] model.cpp:807 - init from 't5xxl-Q8_0.gguf' WARNING: Behavior may be unexpected when allocating 0 bytes for ggml_calloc! [INFO ] stable-diffusion.cpp:216 - loading diffusion model from 'flux1-dev-Q4_0.gguf' [INFO ] model.cpp:790 - load flux1-dev-Q4_0.gguf using gguf format [DEBUG] model.cpp:807 - init from 'flux1-dev-Q4_0.gguf' WARNING: Behavior may be unexpected when allocating 0 bytes for ggml_calloc! [INFO ] stable-diffusion.cpp:223 - loading vae from 'ae.safetensors' [INFO ] model.cpp:793 - load ae.safetensors using safetensors format [DEBUG] model.cpp:861 - init from 'ae.safetensors' [INFO ] stable-diffusion.cpp:235 - Version: Flux Dev [INFO ] stable-diffusion.cpp:266 - Weight type: f16 [INFO ] stable-diffusion.cpp:267 - Conditioner weight type: f16 [INFO ] stable-diffusion.cpp:268 - Diffusion model weight type: q4_0 [INFO ] stable-diffusion.cpp:269 - VAE weight type: f32 [DEBUG] stable-diffusion.cpp:271 - ggml tensor size = 400 bytes [DEBUG] clip.hpp:171 - vocab size: 49408 [DEBUG] clip.hpp:182 - trigger word img already in vocab [DEBUG] ggml_extend.hpp:1046 - clip params backend buffer size = 235.06 MB(RAM) (196 tensors) [DEBUG] ggml_extend.hpp:1046 - t5 params backend buffer size = 9083.77 MB(RAM) (219 tensors) [DEBUG] ggml_extend.hpp:1046 - flux params backend buffer size = 6394.53 MB(RAM) (780 tensors) [DEBUG] ggml_extend.hpp:1046 - vae params backend buffer size = 94.57 MB(RAM) (138 tensors) [DEBUG] stable-diffusion.cpp:398 - loading weights [DEBUG] model.cpp:1530 - loading tensors from clip_l.safetensors [DEBUG] model.cpp:1530 - loading tensors from t5xxl-Q8_0.gguf [INFO ] model.cpp:1685 - unknown tensor 'text_encoders.t5xxl.encoder.embed_tokens.weight | q8_0 | 2 [4096, 32128, 1, 1, 1]' in model file [DEBUG] model.cpp:1530 - loading tensors from flux1-dev-Q4_0.gguf [DEBUG] model.cpp:1530 - loading tensors from ae.safetensors [INFO ] control.hpp:434 - loading control net from 'flux-canny-controlnet-v3.safetensors' [DEBUG] ggml_extend.hpp:1046 - control_net params backend buffer size = 511.14 MB(RAM) (340 tensors) [INFO ] model.cpp:793 - load flux-canny-controlnet-v3.safetensors using safetensors format [DEBUG] model.cpp:861 - init from 'flux-canny-controlnet-v3.safetensors' [DEBUG] model.cpp:1530 - loading tensors from flux-canny-controlnet-v3.safetensors [INFO ] model.cpp:1685 - unknown tensor 'controlnet_blocks.0.bias | bf16 | 1 [3072, 1, 1, 1, 1]' in model file [INFO ] model.cpp:1685 - unknown tensor 'controlnet_blocks.0.weight | bf16 | 2 [3072, 3072, 1, 1, 1]' in model file [INFO ] model.cpp:1685 - unknown tensor 'controlnet_blocks.1.bias | bf16 | 1 [3072, 1, 1, 1, 1]' in model file [INFO ] model.cpp:1685 - unknown tensor 'controlnet_blocks.1.weight | bf16 | 2 [3072, 3072, 1, 1, 1]' in model file [INFO ] model.cpp:1685 - unknown tensor 'double_blocks.0.img_attn.norm.key_norm.scale | bf16 | 1 [128, 1, 1, 1, 1]' in model file [INFO ] model.cpp:1685 - unknown tensor 'double_blocks.0.img_attn.norm.query_norm.scale | bf16 | 1 [128, 1, 1, 1, 1]' in model file [INFO ] model.cpp:1685 - unknown tensor 'double_blocks.0.img_attn.proj.bias | bf16 | 1 [3072, 1, 1, 1, 1]' in model file [INFO ] model.cpp:1685 - unknown tensor 'double_blocks.0.img_attn.proj.weight | bf16 | 2 [3072, 3072, 1, 1, 1]' in model file [INFO ] model.cpp:1685 - unknown tensor 'double_blocks.0.img_attn.qkv.bias | bf16 | 1 [9216, 1, 1, 1, 1]' in model file [INFO ] model.cpp:1685 - unknown tensor 'double_blocks.0.img_attn.qkv.weight | bf16 | 2 [3072, 9216, 1, 1, 1]' in model file [INFO ] model.cpp:1685 - unknown tensor 'double_blocks.0.img_mlp.0.bias | bf16 | 1 [12288, 1, 1, 1, 1]' in model file [INFO ] model.cpp:1685 - unknown tensor 'double_blocks.0.img_mlp.0.weight | bf16 | 2 [3072, 12288, 1, 1, 1]' in model file [INFO ] model.cpp:1685 - unknown tensor 'double_blocks.0.img_mlp.2.bias | bf16 | 1 [3072, 1, 1, 1, 1]' in model file [INFO ] model.cpp:1685 - unknown tensor 'double_blocks.0.img_mlp.2.weight | bf16 | 2 [12288, 3072, 1, 1, 1]' in model file [INFO ] model.cpp:1685 - unknown tensor 'double_blocks.0.img_mod.lin.bias | bf16 | 1 [18432, 1, 1, 1, 1]' in model file [INFO ] model.cpp:1685 - unknown tensor 'double_blocks.0.img_mod.lin.weight | bf16 | 2 [3072, 18432, 1, 1, 1]' in model file [INFO ] model.cpp:1685 - unknown tensor 'double_blocks.0.txt_attn.norm.key_norm.scale | bf16 | 1 [128, 1, 1, 1, 1]' in model file [INFO ] model.cpp:1685 - unknown tensor 'double_blocks.0.txt_attn.norm.query_norm.scale | bf16 | 1 [128, 1, 1, 1, 1]' in model file [INFO ] model.cpp:1685 - unknown tensor 'double_blocks.0.txt_attn.proj.bias | bf16 | 1 [3072, 1, 1, 1, 1]' in model file [INFO ] model.cpp:1685 - unknown tensor 'double_blocks.0.txt_attn.proj.weight | bf16 | 2 [3072, 3072, 1, 1, 1]' in model file [INFO ] model.cpp:1685 - unknown tensor 'double_blocks.0.txt_attn.qkv.bias | bf16 | 1 [9216, 1, 1, 1, 1]' in model file [INFO ] model.cpp:1685 - unknown tensor 'double_blocks.0.txt_attn.qkv.weight | bf16 | 2 [3072, 9216, 1, 1, 1]' in model file [INFO ] model.cpp:1685 - unknown tensor 'double_blocks.0.txt_mlp.0.bias | bf16 | 1 [12288, 1, 1, 1, 1]' in model file [INFO ] model.cpp:1685 - unknown tensor 'double_blocks.0.txt_mlp.0.weight | bf16 | 2 [3072, 12288, 1, 1, 1]' in model file [INFO ] model.cpp:1685 - unknown tensor 'double_blocks.0.txt_mlp.2.bias | bf16 | 1 [3072, 1, 1, 1, 1]' in model file [INFO ] model.cpp:1685 - unknown tensor 'double_blocks.0.txt_mlp.2.weight | bf16 | 2 [12288, 3072, 1, 1, 1]' in model file [INFO ] model.cpp:1685 - unknown tensor 'double_blocks.0.txt_mod.lin.bias | bf16 | 1 [18432, 1, 1, 1, 1]' in model file [INFO ] model.cpp:1685 - unknown tensor 'double_blocks.0.txt_mod.lin.weight | bf16 | 2 [3072, 18432, 1, 1, 1]' in model file [INFO ] model.cpp:1685 - unknown tensor 'double_blocks.1.img_attn.norm.key_norm.scale | bf16 | 1 [128, 1, 1, 1, 1]' in model file [INFO ] model.cpp:1685 - unknown tensor 'double_blocks.1.img_attn.norm.query_norm.scale | bf16 | 1 [128, 1, 1, 1, 1]' in model file [INFO ] model.cpp:1685 - unknown tensor 'double_blocks.1.img_attn.proj.bias | bf16 | 1 [3072, 1, 1, 1, 1]' in model file [INFO ] model.cpp:1685 - unknown tensor 'double_blocks.1.img_attn.proj.weight | bf16 | 2 [3072, 3072, 1, 1, 1]' in model file [INFO ] model.cpp:1685 - unknown tensor 'double_blocks.1.img_attn.qkv.bias | bf16 | 1 [9216, 1, 1, 1, 1]' in model file [INFO ] model.cpp:1685 - unknown tensor 'double_blocks.1.img_attn.qkv.weight | bf16 | 2 [3072, 9216, 1, 1, 1]' in model file [INFO ] model.cpp:1685 - unknown tensor 'double_blocks.1.img_mlp.0.bias | bf16 | 1 [12288, 1, 1, 1, 1]' in model file [INFO ] model.cpp:1685 - unknown tensor 'double_blocks.1.img_mlp.0.weight | bf16 | 2 [3072, 12288, 1, 1, 1]' in model file [INFO ] model.cpp:1685 - unknown tensor 'double_blocks.1.img_mlp.2.bias | bf16 | 1 [3072, 1, 1, 1, 1]' in model file [INFO ] model.cpp:1685 - unknown tensor 'double_blocks.1.img_mlp.2.weight | bf16 | 2 [12288, 3072, 1, 1, 1]' in model file [INFO ] model.cpp:1685 - unknown tensor 'double_blocks.1.img_mod.lin.bias | bf16 | 1 [18432, 1, 1, 1, 1]' in model file [INFO ] model.cpp:1685 - unknown tensor 'double_blocks.1.img_mod.lin.weight | bf16 | 2 [3072, 18432, 1, 1, 1]' in model file [INFO ] model.cpp:1685 - unknown tensor 'double_blocks.1.txt_attn.norm.key_norm.scale | bf16 | 1 [128, 1, 1, 1, 1]' in model file [INFO ] model.cpp:1685 - unknown tensor 'double_blocks.1.txt_attn.norm.query_norm.scale | bf16 | 1 [128, 1, 1, 1, 1]' in model file [INFO ] model.cpp:1685 - unknown tensor 'double_blocks.1.txt_attn.proj.bias | bf16 | 1 [3072, 1, 1, 1, 1]' in model file [INFO ] model.cpp:1685 - unknown tensor 'double_blocks.1.txt_attn.proj.weight | bf16 | 2 [3072, 3072, 1, 1, 1]' in model file [INFO ] model.cpp:1685 - unknown tensor 'double_blocks.1.txt_attn.qkv.bias | bf16 | 1 [9216, 1, 1, 1, 1]' in model file [INFO ] model.cpp:1685 - unknown tensor 'double_blocks.1.txt_attn.qkv.weight | bf16 | 2 [3072, 9216, 1, 1, 1]' in model file [INFO ] model.cpp:1685 - unknown tensor 'double_blocks.1.txt_mlp.0.bias | bf16 | 1 [12288, 1, 1, 1, 1]' in model file [INFO ] model.cpp:1685 - unknown tensor 'double_blocks.1.txt_mlp.0.weight | bf16 | 2 [3072, 12288, 1, 1, 1]' in model file [INFO ] model.cpp:1685 - unknown tensor 'double_blocks.1.txt_mlp.2.bias | bf16 | 1 [3072, 1, 1, 1, 1]' in model file [INFO ] model.cpp:1685 - unknown tensor 'double_blocks.1.txt_mlp.2.weight | bf16 | 2 [12288, 3072, 1, 1, 1]' in model file [INFO ] model.cpp:1685 - unknown tensor 'double_blocks.1.txt_mod.lin.bias | bf16 | 1 [18432, 1, 1, 1, 1]' in model file [INFO ] model.cpp:1685 - unknown tensor 'double_blocks.1.txt_mod.lin.weight | bf16 | 2 [3072, 18432, 1, 1, 1]' in model file [INFO ] model.cpp:1685 - unknown tensor 'guidance_in.in_layer.bias | bf16 | 1 [3072, 1, 1, 1, 1]' in model file [INFO ] model.cpp:1685 - unknown tensor 'guidance_in.in_layer.weight | bf16 | 2 [256, 3072, 1, 1, 1]' in model file [INFO ] model.cpp:1685 - unknown tensor 'guidance_in.out_layer.bias | bf16 | 1 [3072, 1, 1, 1, 1]' in model file [INFO ] model.cpp:1685 - unknown tensor 'guidance_in.out_layer.weight | bf16 | 2 [3072, 3072, 1, 1, 1]' in model file [INFO ] model.cpp:1685 - unknown tensor 'img_in.bias | bf16 | 1 [3072, 1, 1, 1, 1]' in model file [INFO ] model.cpp:1685 - unknown tensor 'img_in.weight | bf16 | 2 [64, 3072, 1, 1, 1]' in model file [ERROR] model.cpp:1694 - tensor 'input_hint_block.10.bias' has wrong shape in model file: got [16, 1, 1, 1], expected [96, 1, 1, 1] [WARN ] model.cpp:1592 - process tensor failed: 'input_hint_block.10.bias' [ERROR] model.cpp:1710 - load tensors from file failed [ERROR] control.hpp:449 - load control net tensors from model loader failed new_sd_ctx_t failed ```
grauho commented 1 month ago

I don't think Flux has control net support yet.

apepkuss commented 1 month ago

You're right. I got the answer from XLabs AI.