[Bug]: What is the API keyword for selecting "control mode" (guess mode) in 1.1?

[Jot-De]

Is there an existing issue for this?

• [X] I have searched the existing issues and checked the recent builds/commits of both this extension and the webui

What happened?

The API keyword guessmode no longer has any effect in ControlNet 1.1? What is the correct keyword in API to change control mode (guess mode) to balanced / prompt / control?

Steps to reproduce the problem

# prepare payload for POST
method
payload = {
 
  "init_images": [img],

  "denoising_strength": 1,

  "prompt": "a woman",
 
 "sampler_name": "DDIM",

  "steps": 30,
 
 "cfg_scale": 7.5,

  "negative_prompt": "poor quality",
 
 "include_init_images": True,
 
 "seed":23,
  
 "batch_size":1,
 
 "alwayson_scripts": {
  
  "controlnet": {
  
      "args": [
 
         {
  
          "model": "control_hed-fp16 [13fee50b]",
 
           "enabled":True,

       "input_image":[img],
 
           "module": "softedge_hed",
 
           "weight": 2.0,

            "processor_res": 512,
 
           "guidance_end": 1.0,
 
           "guessmode": True

          }]
  
}

}

}

# send request

response = requests.post(url=f'{url}/sdapi/v1/txt2img', json=payload)

r = response.json()

output = Image.open(io.BytesIO(base64.b64decode(r['images'][0].split(",",1)[0])))

output

What should have happened?

guessmode OR any other proper keyword which I don't know should change guess mode.

Commit where the problem happens

webui: 22bcc7b controlnet: a07f6e8

What browsers do you use to access the UI ?

No response

Command Line Arguments

--api

Console logs

Loading model from cache: control_hed-fp16 [13fee50b]
Loading preprocessor: hed
preprocessor resolution = 512
Data shape for DDIM sampling is (1, 4, 64, 64), eta 0.0
Running DDIM Sampling with 31 timesteps
DDIM Sampler: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 31/31 [00:06<00:00,  4.61it/s] 
Total progress: 93it [10:08,  4.61it/s]

Additional information

No response

[joaxkal]

I have the same issue. Changing guessmode values (True / False) has no effect

[dermesut]

yes, unfortunately there seems to be a number of new items missing when it comes to the api route. judging by the server:7860/docs "pixel perfect" seems to have been added, but besides from that, the rest seems to be the same/similar as/to the "old" controlnet.

similarly, i can't reach the new preprocessors.

the "module_list" provides:

{ "module_list": [ "none", "canny", "depth", "depth_leres", "hed", "hed_safe", "mediapipe_face", "mlsd", "normal_map", "openpose", "openpose_hand", "openpose_face", "openpose_faceonly", "openpose_full", "clip_vision", "color", "pidinet", "pidinet_safe", "pidinet_sketch", "pidinet_scribble", "scribble_xdog", "scribble_hed", "segmentation", "threshold", "depth_zoe", "normal_bae", "oneformer_coco", "oneformer_ade20k", "lineart", "lineart_coarse", "lineart_anime", "lineart_standard", "shuffle", "tile_resample", "inpaint", "invert" ] }

while the web-ui provides:

{ "id": 144, "type": "dropdown", "props": { "choices": [ "none", "invert (from white bg & black line)", "canny", "depth_leres", "depth_midas", "depth_zoe", "inpaint_global_harmonious", "lineart_anime", "lineart_coarse", "lineart_realistic", "lineart_standard (from white bg & black line)", "mediapipe_face", "mlsd", "normal_bae", "normal_midas", "openpose", "openpose_face", "openpose_faceonly", "openpose_full", "openpose_hand", "scribble_hed", "scribble_pidinet", "scribble_xdog", "seg_ofade20k", "seg_ofcoco", "seg_ufade20k", "shuffle", "softedge_hed", "softedge_hedsafe", "softedge_pidinet", "softedge_pidisafe", "t2ia_color_grid", "t2ia_sketch_pidi", "t2ia_style_clipvision", "threshold", "tile_resample" ], "value": "none", "label": "Preprocessor", "show_label": true, "name": "dropdown", "visible": true, "elem_classes": [ "gradio-dropdown" ], "style": {} } }

(via get config).

would have been nice if i had known that prior to updating to the most recent version, so i could have waited until the api would have been updated as well. seems there's always the possibility to go via the get_config way, but i believe this to be quite uncomfortable, since the 'id' seems to be changing over time, if i remember correctly.

really unfortunate.

[ljleb]

The API key for control mode is "control_mode". It takes either the literal string values in the ui (properly capitalized) or an int value, in the order they are listed here:

https://github.com/Mikubill/sd-webui-controlnet/blob/a07f6e8a1a4d4d6ced366aec30fd8244f1fde26d/scripts/external_code.py#L20-L22

So 0 for BALANCED, 1 for PROMPT, etc.

As for the preprocessors, the 2 lists are identical, except for the fact that in the second list some entries are replaced with their alias, defined here:

https://github.com/Mikubill/sd-webui-controlnet/blob/a07f6e8a1a4d4d6ced366aec30fd8244f1fde26d/scripts/global_state.py#L80-L96

We need to update the wiki to let users know we can pass either the aliases or the real preprocessor names.

[lllyasviel]

hello we are still working on upgrading api if you have important jobs to perform and requires reproducing all previous results, an option is use previous version. See "Use Previous Version 1.0" in Readme

[ljleb]

The only conversion we need to do is add an alias for guessmode and guess_mode that will be converted into control_mode appropriately.

[Jot-De]

@ljleb Sadly, passing "control_mode": 0 (or 1 or 2) has no effect for me

[ljleb]

All json object keys should be forwarded to the control units here:

https://github.com/Mikubill/sd-webui-controlnet/blob/a07f6e8a1a4d4d6ced366aec30fd8244f1fde26d/scripts/external_code.py#L203

Can you share the object you used in the updated request?

[ljleb]

Ah I know what the problem is. Even though control mode is advertised as accepting an int, str or ControlMode enum, it only accepts string values at the moment.

[ljleb]

We need to convert control_mode to an enum before reading its value here:

https://github.com/Mikubill/sd-webui-controlnet/blob/a07f6e8a1a4d4d6ced366aec30fd8244f1fde26d/scripts/controlnet.py#L1145-L1146

[Jot-De]

@ljleb

# prepare payload for POST method
payload = {
  "init_images": [img],
  "denoising_strength": 1,
  "prompt": "a woman",
  "sampler_name": "DDIM",
  "steps": 30,
  "cfg_scale": 7.5,
  "negative_prompt": "poor quality",
  "include_init_images": True,
  "seed":23,
  "batch_size":1,
  "alwayson_scripts": {
    "controlnet": {
        "args": [
          {
            "model": "control_hed-fp16 [13fee50b]",
            "enabled":True,
            "input_image":[img],
            "module": "softedge_hed",
            "weight": 2.0,
            "processor_res": 512,
            "guidance_end": 1.0,
            "resize_mode":2,
            "control_mode":2
          }]
  }
}
}
# send request
response = requests.post(url=f'{url}/sdapi/v1/txt2img', json=payload)
r = response.json()
output = Image.open(io.BytesIO(base64.b64decode(r['images'][0].split(",",1)[0])))
output

[dermesut]

hello we are still working on upgrading api if you have important jobs to perform and requires reproducing all previous results, an option is use previous version. See "Use Previous Version 1.0" in Readme

it's not about reproducing previous results, it's more about recreating the drop-down options, that i can see in the web-ui. previously i could get those by reaching out to the get_modules api, fed that list into a drop-down in my plug-in ui and let the user choose the preprocessor, knowing that the options were identical to the option one could get in the web-ui. if understand correctly, this workflow would not work anymore, since list pf preprocessors that is being provided by the api-get_modules would need to be manipulated first, according to the aliases mentioned in the global_state.py? also, the control_modes need to be fetched out of the external_code.py (or hard-coded).

it was way more convenient to just grab, what was provided by the api and rest assured that it is consistent with what the web-ui was showing.

is there a way to get back that convenience?

[ljleb]

The API automatically converts the aliases when it receives one:

https://github.com/Mikubill/sd-webui-controlnet/blob/a07f6e8a1a4d4d6ced366aec30fd8244f1fde26d/scripts/api.py#L205

the names are different, but they have the same behavior. The exposed names are more friendly to use in code, but not in user interfaces. We could add a parameter to show the alias names instead of the id names I think for this purpose.

[ljleb]

@Jot-De the guess_mode and control_mode parameters should now both work.

[ljleb]

In fact GET routes cannot take a json body. They can with fastapi, but it is discouraged. Our options seem to be:

• always return alias ui names from /controlnet/module_list
• create a new route for the ui names
• add a parameter anyway (will not show up in /docs: https://fastapi.tiangolo.com/tutorial/body/)

[dermesut]

The API automatically converts the aliases when it receives one:

https://github.com/Mikubill/sd-webui-controlnet/blob/a07f6e8a1a4d4d6ced366aec30fd8244f1fde26d/scripts/api.py#L205

the names are different, but they have the same behavior. The exposed names are more friendly to use in code, but not in user interfaces. We could add a parameter to show the alias names instead of the id names I think for this purpose.

well, it would be great, if the content of the modulelist could be at least somewhat more obvious, if not identical to the web-ui drop-down. e.g. segmentation preprocessors: in the web-ui all three of them start with "seg": "seg_ofade20k, seg_ufade20k, seg_ofcoco", which makes it mentally easier to assign those to the concept of segmentation. the api provides "segmentation, oneformer_coco, oneformer_ade20k". so... how would a user know, which one is which?

[ljleb]

The issue is that ui names could be inconvenient for scripts because some of them contain spaces (i.e. see the alias for "invert"). It makes sense to want to access them for consistency though, I've listed a couple options above that we can consider.

[lllyasviel]

I am going to close it since resolved. feel free to still discuss

[ljleb]

Ah wait it seems that although a request body is not supported, we can pass request params. I'll try this approach and merge a fix if it works.

[dermesut]

The issue is that ui names could be inconvenient for scripts because some of them contain spaces (i.e. see the alias for "invert"). It makes sense to want to access them for consistency though, I've listed a couple options above that we can consider.

ok, i see what you mean.

the second option you suggested seems to be a good idea, also for future "naming"-consistencies between web-ui and api-data. a kind of a "translation"-layer that could conveniently be fetched and presented. and expended if needed in future additions.

i guess, one can see, i am a lazy sucker for convenience :)

[ljleb]

I am adding a request param in #1040. You will be able to get the alias names by passing the alias_names=true request param like this:

GET http://localhost:7860/controlnet/module_list?alias_names=true

Edit: param is now added in version 1.1.111.

[dermesut]

and just like that, the drop-down in cinema4d shows the same names as the web-ui :) thank you for your work.

[Jot-De]

Thank you for your quick help!