[Bug]: The images produced using sd-webii-controllnet v1.1 on Apple Silicon are completely disorganized

[Gitawake]

Is there an existing issue for this?

• [X] I have searched the existing issues and checked the recent builds/commits of both this extension and the webui

What happened?

After updating Apple Silicon to version 1.1, the log display was activated, and the images generated by Controlnet were completely irrelevant：

Couldn't install sd-webui-controlnet requirement: mediapipe==0.9.1.0.
Command: "/Users/gitawake0w/PycharmProjects/stable-diffusion-webui/venv/bin/python3.10" -m pip install mediapipe==0.9.1.0 --prefer-binary
Error code: 1
stdout: <empty>
stderr: ERROR: Could not find a version that satisfies the requirement mediapipe==0.9.1.0 (from versions: 0.9.3.0)
ERROR: No matching distribution found for mediapipe==0.9.1.0

Warning: Failed to install mediapipe==0.9.1.0, some preprocessors may not work.
Installing sd-webui-controlnet requirement: opencv-python```

### Steps to reproduce the problem

1. After Apple Silicon updates version 1.1, the startup log displays
2.the startup log displays :ERROR: No matching distribution found for mediapipe==0.9.1.0

### What should have happened?

Installing sd-webui-controlnet requirement: mediapipe==0.9.1.0
success

### Commit where the problem happens

webui: 
controlnet: 

### What browsers do you use to access the UI ?

Google Chrome

### Command Line Arguments

```Shell
./webui.sh

Console logs

################################################################
Install script for stable-diffusion + Web UI
Tested on Debian 11 (Bullseye)
################################################################

################################################################
Running on gitawake0w user
################################################################

################################################################
Repo already cloned, using it as install directory
################################################################

################################################################
Create and activate python venv
################################################################

################################################################
Launching launch.py...
################################################################
Python 3.10.11 (main, Apr  7 2023, 07:24:53) [Clang 14.0.0 (clang-1400.0.29.202)]
Commit hash: 22bcc7be428c94e9408f589966c2040187245d81
Installing requirements for Web UI
Installing sd-webui-controlnet requirement: mediapipe==0.9.1.0
Couldn't install sd-webui-controlnet requirement: mediapipe==0.9.1.0.
Command: "/Users/gitawake0w/PycharmProjects/stable-diffusion-webui/venv/bin/python3.10" -m pip install mediapipe==0.9.1.0 --prefer-binary
Error code: 1
stdout: <empty>
stderr: ERROR: Could not find a version that satisfies the requirement mediapipe==0.9.1.0 (from versions: 0.9.3.0)
ERROR: No matching distribution found for mediapipe==0.9.1.0

Warning: Failed to install mediapipe==0.9.1.0, some preprocessors may not work.
Installing sd-webui-controlnet requirement: opencv-python

Launching Web UI with arguments: --skip-torch-cuda-test --upcast-sampling --no-half-vae --use-cpu interrogate
Warning: caught exception 'Torch not compiled with CUDA enabled', memory monitor disabled
No module 'xformers'. Proceeding without it.
==============================================================================
You are running torch 1.12.1.
The program is tested to work with torch 1.13.1.
To reinstall the desired version, run with commandline flag --reinstall-torch.
Beware that this will cause a lot of large files to be downloaded, as well as
there are reports of issues with training tab on the latest version.

Use --skip-version-check commandline argument to disable this check.
==============================================================================
Loading weights [abcaf14e5a] from /Users/gitawake0w/PycharmProjects/stable-diffusion-webui/models/Stable-diffusion/anything-v3-full.safetensors
Creating model from config: /Users/gitawake0w/PycharmProjects/stable-diffusion-webui/configs/v1-inference.yaml
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.52 M params.
Applying cross attention optimization (InvokeAI).
Textual inversion embeddings loaded(0): 
Model loaded in 6.9s (load weights from disk: 0.4s, create model: 0.5s, apply weights to model: 4.1s, apply half(): 1.1s, move model to device: 0.6s).
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Startup time: 12.7s (import torch: 1.4s, import gradio: 1.3s, import ldm: 0.2s, other imports: 1.8s, load scripts: 0.6s, load SD checkpoint: 6.9s, create ui: 0.2s).

Additional information

No response

[Gitawake]

The generated image is completely unrelated to the sample, and I am currently using the old version to function properly.

[Gitawake]

生成的图像与示例完全无关，我目前使用旧版本可以正常运行。

There were no errors during the production process. I replaced mediapipe=0.9.1.0 with 0.9.1 and still couldn't install it. Replacing it with 0.9.3.0 allowed installation, but Controlnet still couldn't function properly.

[TobyDustin]

Hello, I am also having the problem. It seems like controlnet is cropping to the top right corner of the image. See attached

[TobyDustin]

Here is an example of my settings:

And the output image

[TobyDustin]

Using controlNet v1.1 models made the image slightly comprehensible, but still nothing like a the images you see online where the image structure remains the same.

[lllyasviel]

strange. let me take a look. both nvidia and amd cannot reprodece and i do not have apple

[TobyDustin]

@Gitawake you mentioned you are using the old version of controlNet? How did you revert back to the old version of controlNet?

(Sorry if this is an obvious question, I am still new to stable diffusion)

[lllyasviel]

hi apple users, can you try to set preprocessor to inpaint_global_harmonious and then draw a random mask and then use control_v11p_sd_15_inpaint to see what is your outputs?

Please use exactly same setting as

The image is

and make sure that you use up-to-date version of controlnet

[Gitawake]

@Gitawake you mentioned you are using the old version of controlNet? How did you revert back to the old version of controlNet?

(Sorry if this is an obvious question, I am still new to stable diffusion)

I copied it online, but I don't know which version it is, but it does work.

[Mikubill]

Add no half flag if torch<2.0

[Gitawake]

hi apple users, can you try to set preprocessor to inpaint_global_harmonious and then draw a random mask and then use control_v11p_sd_15_inpaint to see what is your outputs?

Please use exactly same setting as

The image is

and make sure that you use up-to-date version of controlnet

The produced image will have a blue semi transparent overlay on half the edge @Mikubill

[lllyasviel]

i really do not know what is going on. anyone ideas?

[MaximumBlast]

I am teaching ai-art at uni, my class just updated their auto1111 control net installation and everything was broken. we found out about the new models and added them, but still the output has nothing to do with the input controlnet-images. we hope you find a solution. good luck!

in the meantime, where can we find the older version?

[Gitawake]

@MaximumBlast If you encounter the same problem as me, or you can execute a command to roll back to the version a week ago in the stable-diffusion-webui/extensions/sd-webui-controlnet folder git reset --hard 0f549888fd49aea48a4a5049f75c2e87ad3affad

[TobyDustin]

@MaximumBlast If you encounter the same problem as me, or you can execute a command to roll back to the version a week ago in the stable-diffusion-webui/extensions/sd-webui-controlnet folder git reset --hard 0f549888fd49aea48a4a5049f75c2e87ad3affad

thank you for this workaround!

[Gitawake]

@lllyasviel I found that the version from a week ago was working. In order to verify which commit caused the issue, I repeatedly pulled different SHAs using "git reset --hard" and "git pull". Eventually, I discovered that as long as I passed through a certain version, updating to the latest version would work correctly. To test this, I re-pulled AUTOMATIC1111/stable-diffusion-webui and installed the latest version of sd-webui-controlnet without doing anything else. However, it did not work correctly. When I used "git reset --hard 0f549888fd49aea48a4a5049f75c2e87ad3affad" to return to a week ago and then restarted, it worked correctly. This time, I did not try jumping up one version at a time, but instead pulled the latest version with "git pull" and found that it still did not work correctly. Perhaps by pulling a new stable-diffusion-webui and installing the latest controlnet, I could reproduce this issue?

[lllyasviel]

did you find out which commit caused the problem?

[Gitawake]

@lllyasviel I'm not sure yet, pulling is too frequent and the workload is huge. Perhaps you can help me give it a try by pulling a brand new AUTOMATIC1111/stable diffusion webui and installing a brand new SD webui control net. Can you replicate this issue on your device? I'll try again to determine which submitted the issue.

[lllyasviel]

Linux and Windows works perfectly in my 4 devices in all commits as far as I know.

[Gitawake]

@lllyasviel The submission of 2770751 will no longer work, and b7519b6 is still normal

[lllyasviel]

thanks! do you mean https://github.com/Mikubill/sd-webui-controlnet/commit/277075100370c2b9a21fba4cecda456377353649 is not working and https://github.com/Mikubill/sd-webui-controlnet/commit/b7519b62142d08d1528f9e5f52190b8bee8b6ed3 is working?

[Gitawake]

thanks! do you mean 2770751 is not working and b7519b6 is working?

yes

[lllyasviel]

thanks very much for the goden information! wanted to note that after you change the commit of controlnet, you need to restart the terminal (not only a1111's restart button but also the entire terminal) that is to say, if you use commit A and then shift to B without restart terminal, the nyou are still using the codes of A. We want to finally confirm that is your case:

1. checkout to https://github.com/Mikubill/sd-webui-controlnet/commit/277075100370c2b9a21fba4cecda456377353649, then completely restart terminal (not only a1111 restart button), it does not work.
2. checkout to https://github.com/Mikubill/sd-webui-controlnet/commit/b7519b62142d08d1528f9e5f52190b8bee8b6ed3, then completely restart terminal (not only a1111 restart button), it does work and results are good.

is that correct?

[Gitawake]

thanks very much for the goden information! wanted to note that after you change the commit of controlnet, you need to restart the terminal (not only a1111's restart button but also the entire terminal) that is to say, if you use commit A and then shift to B without restart terminal, the nyou are still using the codes of A. We want to finally confirm that is your case:

1. checkout to 2770751, then completely restart terminal (not only a1111 restart button), it does not work.
2. checkout to b7519b6, then completely restart terminal (not only a1111 restart button), it does work and results are good.

is that correct?

I turn off the terminal, turn it on again, refresh the web page, and re import the image to Controlnet

[lllyasviel]

thanks very much for the information! i will take a look.

[lllyasviel]

in https://github.com/Mikubill/sd-webui-controlnet/commit/277075100370c2b9a21fba4cecda456377353649, do you see the warning log:

                print('ERROR: Failed to apply Guess Mode or Shuffle. You may try to add --always-batch-cond-uncond to your flags')
                print('ERROR: Begin to use backup method without Guess Mode or Shuffle.')
                print('ERROR: Results might be worse, but the webui will not fail.')

and fail, or does it fail without any logs?

[Gitawake]

`./webui.sh

################################################################ Install script for stable-diffusion + Web UI Tested on Debian 11 (Bullseye) ################################################################

################################################################ Running on gitawake0w user ################################################################

################################################################ Repo already cloned, using it as install directory ################################################################

################################################################ Create and activate python venv ################################################################

################################################################ Launching launch.py... ################################################################ Python 3.10.11 (main, Apr 7 2023, 07:24:53) [Clang 14.0.0 (clang-1400.0.29.202)] Commit hash: 22bcc7be428c94e9408f589966c2040187245d81 Installing requirements for Web UI

Launching Web UI with arguments: --skip-torch-cuda-test --upcast-sampling --no-half-vae --use-cpu interrogate Warning: caught exception 'Torch not compiled with CUDA enabled', memory monitor disabled No module 'xformers'. Proceeding without it.

You are running torch 1.12.1. The program is tested to work with torch 1.13.1. To reinstall the desired version, run with commandline flag --reinstall-torch. Beware that this will cause a lot of large files to be downloaded, as well as there are reports of issues with training tab on the latest version.

Use --skip-version-check commandline argument to disable this check.

Loading weights [6ce0161689] from /Users/gitawake0w/PycharmProjects/stable-diffusion-webui/models/Stable-diffusion/v1-5-pruned-emaonly.safetensors Creating model from config: /Users/gitawake0w/PycharmProjects/stable-diffusion-webui/configs/v1-inference.yaml LatentDiffusion: Running in eps-prediction mode DiffusionWrapper has 859.52 M params. Applying cross attention optimization (InvokeAI). Textual inversion embeddings loaded(0): Model loaded in 4.0s (load weights from disk: 0.2s, create model: 0.6s, apply weights to model: 2.1s, apply half(): 0.5s, move model to device: 0.6s). Running on local URL: http://127.0.0.1:7860

To create a public link, set share=True in launch(). Startup time: 6.9s (import torch: 0.7s, import gradio: 0.6s, import ldm: 0.2s, other imports: 0.6s, load scripts: 0.5s, load SD checkpoint: 4.0s, create ui: 0.2s). Loading model: control_v11p_sd15_canny [d14c016b] Loaded state_dict from [/Users/gitawake0w/PycharmProjects/stable-diffusion-webui/extensions/sd-webui-controlnet/models/control_v11p_sd15_canny.pth] ControlNet model control_v11p_sd15_canny [d14c016b] loaded. Loading preprocessor: canny 100%|███████████████████████████████████████████| 20/20 [01:04<00:00, 3.21s/it] Total progress: 100%|███████████████████████████| 20/20 [01:00<00:00, 3.02s/it] Total progress: 100%|███████████████████████████| 20/20 [01:00<00:00, 3.40s/it]`

Or rather, he didn't fail, but the images he produced were messy

[lllyasviel]

and b7519b6 and restart completely, then results are good?

very weird.. let me take a deeper look

[lllyasviel]

wait a moment, from the code it seems that you are using guess mode or shuffle mode. are you using guess mode?

[Gitawake]

[Gitawake]

wait a moment, from the code it seems that you are using guess mode or shuffle mode. are you using guess mode?

[lllyasviel]

ok I see. please try one more thing:

please shift to the working commit (b7519b6), get good results, then shift your sampler to DDIM (not Eular A), then see if the results are bad again?

[Gitawake]

ok I see. please try one more thing:

please shift to the working commit (b7519b6), get good results, then shift your sampler to DDIM (not Eular A), then see if the results are bad again?

[lllyasviel]

this is just impossible in your case you have self.is_vanilla_samplers=False param.guess_mode=False param.global_average_pooling=False

then the two codes are absolutely equivalent. https://github.com/Mikubill/sd-webui-controlnet/commit/277075100370c2b9a21fba4cecda456377353649

[lllyasviel]

so the only possibility is that you have param.global_average_pooling=True, but this only happens if you use shuffle model

[lllyasviel]

can you share with me the full content of your default model config file?

[Gitawake]

I don't quite understand. Can you please specify which file or parameter is specific

can you share with me the full content of your default model config file?

[lllyasviel]

can you give me a screenshot of your controlnet setting?

[Gitawake]

[lllyasviel]

please also give the a1111 setting's ControlNet Tab. it is in a1111 setting

[Gitawake]

[lllyasviel]

please open models/cldm_v15.yaml and copy all contents and put it here

[Gitawake]

  target: cldm.cldm.ControlLDM
  params:
    linear_start: 0.00085
    linear_end: 0.0120
    num_timesteps_cond: 1
    log_every_t: 200
    timesteps: 1000
    first_stage_key: "jpg"
    cond_stage_key: "txt"
    control_key: "hint"
    image_size: 64
    channels: 4
    cond_stage_trainable: false
    conditioning_key: crossattn
    monitor: val/loss_simple_ema
    scale_factor: 0.18215
    use_ema: False
    only_mid_control: False

    control_stage_config:
      target: cldm.cldm.ControlNet
      params:
        image_size: 32 # unused
        in_channels: 4
        hint_channels: 3
        model_channels: 320
        attention_resolutions: [ 4, 2, 1 ]
        num_res_blocks: 2
        channel_mult: [ 1, 2, 4, 4 ]
        num_heads: 8
        use_spatial_transformer: True
        transformer_depth: 1
        context_dim: 768
        use_checkpoint: True
        legacy: False

    unet_config:
      target: cldm.cldm.ControlledUnetModel
      params:
        image_size: 32 # unused
        in_channels: 4
        out_channels: 4
        model_channels: 320
        attention_resolutions: [ 4, 2, 1 ]
        num_res_blocks: 2
        channel_mult: [ 1, 2, 4, 4 ]
        num_heads: 8
        use_spatial_transformer: True
        transformer_depth: 1
        context_dim: 768
        use_checkpoint: True
        legacy: False

    first_stage_config:
      target: ldm.models.autoencoder.AutoencoderKL
      params:
        embed_dim: 4
        monitor: val/rec_loss
        ddconfig:
          double_z: true
          z_channels: 4
          resolution: 256
          in_channels: 3
          out_ch: 3
          ch: 128
          ch_mult:
          - 1
          - 2
          - 4
          - 4
          num_res_blocks: 2
          attn_resolutions: []
          dropout: 0.0
        lossconfig:
          target: torch.nn.Identity

    cond_stage_config:
      target: ldm.modules.encoders.modules.FrozenCLIPEmbedder

[lllyasviel]

please open the control_v11p_sd15_canny.yaml and copy all contents and put it here

[Gitawake]

  target: cldm.cldm.ControlLDM
  params:
    linear_start: 0.00085
    linear_end: 0.0120
    num_timesteps_cond: 1
    log_every_t: 200
    timesteps: 1000
    first_stage_key: "jpg"
    cond_stage_key: "txt"
    control_key: "hint"
    image_size: 64
    channels: 4
    cond_stage_trainable: false
    conditioning_key: crossattn
    monitor: val/loss_simple_ema
    scale_factor: 0.18215
    use_ema: False
    only_mid_control: False

    control_stage_config:
      target: cldm.cldm.ControlNet
      params:
        image_size: 32 # unused
        in_channels: 4
        hint_channels: 3
        model_channels: 320
        attention_resolutions: [ 4, 2, 1 ]
        num_res_blocks: 2
        channel_mult: [ 1, 2, 4, 4 ]
        num_heads: 8
        use_spatial_transformer: True
        transformer_depth: 1
        context_dim: 768
        use_checkpoint: True
        legacy: False

    unet_config:
      target: cldm.cldm.ControlledUnetModel
      params:
        image_size: 32 # unused
        in_channels: 4
        out_channels: 4
        model_channels: 320
        attention_resolutions: [ 4, 2, 1 ]
        num_res_blocks: 2
        channel_mult: [ 1, 2, 4, 4 ]
        num_heads: 8
        use_spatial_transformer: True
        transformer_depth: 1
        context_dim: 768
        use_checkpoint: True
        legacy: False

    first_stage_config:
      target: ldm.models.autoencoder.AutoencoderKL
      params:
        embed_dim: 4
        monitor: val/rec_loss
        ddconfig:
          double_z: true
          z_channels: 4
          resolution: 256
          in_channels: 3
          out_ch: 3
          ch: 128
          ch_mult:
          - 1
          - 2
          - 4
          - 4
          num_res_blocks: 2
          attn_resolutions: []
          dropout: 0.0
        lossconfig:
          target: torch.nn.Identity

    cond_stage_config:
      target: ldm.modules.encoders.modules.FrozenCLIPEmbedder

[lllyasviel]

ok I need to make a final confirmation that, rightnow, you shift to https://github.com/Mikubill/sd-webui-controlnet/commit/277075100370c2b9a21fba4cecda456377353649 and then restart completely, and then results are bad?

[lllyasviel]

if that is true, can you try to move the file control_v11p_sd15_canny.pth to extensions\sd-webui-controlnet\models and see if the results are still bad?

[Gitawake]

ok I need to make a final confirmation that, rightnow, you shift to 2770751 and then restart completely, and then results are bad?

[lllyasviel]

but this is just impossible, you do not even step into modified parts. the changes between the two commits do not have influence to your case and you are just running exactly same codes.

[lllyasviel]

let me take a look again