search

Mikubill / sd-webui-controlnet

WebUI extension for ControlNet
GNU General Public License v3.0
17.04k stars 1.96k forks source link

[Bug]: Animal Openpose does not reflect to generated pictures on Google Colab #2695

Closed sst030164 closed 7 months ago

sst030164 commented 7 months ago

Is there an existing issue for this?

What happened?

Animal Openpose is working (key line is detected) but the generated picture is not reflected openpose on Google Colab.

animal_openpose

Steps to reproduce the problem

  1. !git clone https://github.com/AUTOMATIC1111/stable-diffusion-webui
  2. %cd /content/stable-diffusion-webui !python launch.py --share --enable-insecure-extension-access
  3. Install extension from the following URL, and apply to Web UI. (WebUI will stop) https://github.com/Mikubill/sd-webui-controlnet
  4. !curl -Lo /content/stable-diffusion-webui/extensions/sd-webui-controlnet/models/control_sd15_animal_openpose_fp16.pth https://huggingface.co/huchenlei/animal_openpose/blob/main/control_sd15_animal_openpose_fp16.pth
  5. Run WebUI again, and access it.
  6. Choose ControlNet (Unit1 or Unit2) and Check [Enable]
  7. Put source image in canvas of ControlNet tab
  8. Choose preprocessor [animal_openpose] and Model[control_sd15_animal_openpose_fp16]
  9. Run Preprocessor (generate Preprocessor Preview (Key line)).
  10. Input prompt (example:cat) and Push [Generate]

What should have happened?

Generate picture that reflected animal_openpose

Commit where the problem happens

webui: v1.8.0 controlnet: v1.1.441

What browsers do you use to access the UI ?

Google Chrome

Command Line Arguments

Run the following command and option to start WebUI
!python launch.py --share --enable-insecure-extension-access

List of enabled extensions

image

Console logs

/content/stable-diffusion-webui
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]
Version: v1.8.0
Commit hash: bef51aed032c0aaa5cfd80445bc4cf0d85b408b5
ControlNet init warning: Unable to install insightface automatically. Please try run `pip install insightface` manually.
Launching Web UI with arguments: --share --enable-insecure-extension-access
2024-03-19 06:59:52.972120: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-03-19 06:59:52.972184: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-03-19 06:59:52.974158: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-03-19 06:59:54.368616: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
no module 'xformers'. Processing without...
no module 'xformers'. Processing without...
No module 'xformers'. Proceeding without it.
ControlNet preprocessor location: /content/stable-diffusion-webui/extensions/sd-webui-controlnet/annotator/downloads
2024-03-19 06:59:58,370 - ControlNet - INFO - ControlNet v1.1.441
2024-03-19 06:59:58,607 - ControlNet - INFO - ControlNet v1.1.441
Loading weights [6ce0161689] from /content/stable-diffusion-webui/models/Stable-diffusion/v1-5-pruned-emaonly.safetensors
2024-03-19 06:59:58,999 - ControlNet - INFO - ControlNet UI callback registered.
Running on local URL:  http://127.0.0.1:7860
Creating model from config: /content/stable-diffusion-webui/configs/v1-inference.yaml
Running on public URL: https://81a09ee95a0f44b38a.gradio.live

This share link expires in 72 hours. For free permanent hosting and GPU upgrades, run `gradio deploy` from Terminal to deploy to Spaces (https://huggingface.co/spaces)
Startup time: 22.1s (prepare environment: 3.5s, import torch: 5.5s, import gradio: 1.4s, setup paths: 4.3s, initialize shared: 0.2s, other imports: 0.8s, load scripts: 1.2s, create ui: 0.7s, gradio launch: 4.4s).
Applying attention optimization: Doggettx... done.
Model loaded in 8.0s (load weights from disk: 1.3s, create model: 0.6s, apply weights to model: 5.2s, apply half(): 0.2s, load textual inversion embeddings: 0.4s, calculate empty prompt: 0.2s).
2024-03-19 07:11:09,199 - ControlNet - INFO - Preview Resolution = 512
Downloading: "https://huggingface.co/yzd-v/DWPose/resolve/main/yolox_l.onnx" to /content/stable-diffusion-webui/extensions/sd-webui-controlnet/annotator/downloads/openpose/yolox_l.onnx

100% 207M/207M [00:01<00:00, 154MB/s]
Downloading: "https://huggingface.co/bdsqlsz/qinglong_controlnet-lllite/resolve/main/Annotators/rtmpose-m_simcc-ap10k_pt-aic-coco_210e-256x256-7a041aa1_20230206.onnx" to /content/stable-diffusion-webui/extensions/sd-webui-controlnet/annotator/downloads/openpose/rtmpose-m_simcc-ap10k_pt-aic-coco_210e-256x256-7a041aa1_20230206.onnx

100% 52.0M/52.0M [00:00<00:00, 96.4MB/s]
2024-03-19 07:11:51,805 - ControlNet - INFO - unit_separate = False, style_align = False
2024-03-19 07:11:52,190 - ControlNet - INFO - Loading model: control_sd15_animal_openpose_fp16 [e3b0c442]
*** Error running process: /content/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py
    Traceback (most recent call last):
      File "/content/stable-diffusion-webui/modules/scripts.py", line 784, in process
        script.process(p, *script_args)
      File "/content/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py", line 1279, in process
        self.controlnet_hack(p)
      File "/content/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py", line 1264, in controlnet_hack
        self.controlnet_main_entry(p)
      File "/content/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py", line 936, in controlnet_main_entry
        model_net, control_model_type = Script.load_control_model(p, unet, unit.model)
      File "/content/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py", line 367, in load_control_model
        control_model = Script.build_control_model(p, unet, model)
      File "/content/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py", line 395, in build_control_model
        state_dict = load_state_dict(model_path)
      File "/content/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/utils.py", line 22, in load_state_dict
        state_dict = unsafe_torch_load(ckpt_path, map_location=torch.device(location))
      File "/usr/local/lib/python3.10/dist-packages/torch/serialization.py", line 1040, in load
        return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
      File "/usr/local/lib/python3.10/dist-packages/torch/serialization.py", line 1258, in _legacy_load
        magic_number = pickle_module.load(f, **pickle_load_args)
    _pickle.UnpicklingError: invalid load key, '<'.

---
  0% 0/20 [00:00<?, ?it/s]
  5% 1/20 [00:01<00:25,  1.34s/it]
 10% 2/20 [00:01<00:12,  1.49it/s]
 15% 3/20 [00:01<00:07,  2.19it/s]
 20% 4/20 [00:01<00:05,  2.80it/s]
 25% 5/20 [00:02<00:04,  3.32it/s]
 30% 6/20 [00:02<00:03,  3.66it/s]
 35% 7/20 [00:02<00:03,  3.98it/s]
 40% 8/20 [00:02<00:02,  4.21it/s]
 45% 9/20 [00:02<00:02,  4.36it/s]
 50% 10/20 [00:03<00:02,  4.47it/s]
 55% 11/20 [00:03<00:01,  4.53it/s]
 60% 12/20 [00:03<00:01,  4.60it/s]
 65% 13/20 [00:03<00:01,  4.60it/s]
 70% 14/20 [00:04<00:01,  4.64it/s]
 75% 15/20 [00:04<00:01,  4.66it/s]
 80% 16/20 [00:04<00:00,  4.70it/s]
 85% 17/20 [00:04<00:00,  4.72it/s]
 90% 18/20 [00:04<00:00,  4.69it/s]
 95% 19/20 [00:05<00:00,  4.73it/s]
100% 20/20 [00:05<00:00,  3.75it/s]

Total progress: 100% 20/20 [00:04<00:00,  4.58it/s]

Additional information

it's almost same as "https://github.com/Mikubill/sd-webui-controlnet/issues/301#issuecomment-1751930000"

huchenlei commented 7 months ago

Please make sure you correctly downloaded the model not an HTML page. You can tell this by file size. The fp16 model should be around 700MB.