[Bug]: Can't add point prompts or pick a box threshold

[copybaras]

Is there an existing issue for this?

• [X] I have searched the existing issues and checked the recent builds/commits of both this extension and the webui

Have you updated WebUI and this extension to the newest version?

• [X] I have updated WebUI and this extension to the most up-to-date version

Do you understand that you should go to https://github.com/IDEA-Research/Grounded-Segment-Anything/issues if you cannot install GroundingDINO?

• [X] My problem is not about installing GroundingDINO

What happened?

After dropping image to Segment Anyting tab I can't add point prompts or pick a box threshold after checking Enable GroundingDINO.

Steps to reproduce the problem

1. Go to img2img
2. Open Segment Anyting tab
3. Select a sam model
4. Drop image under selecting model tab
5. Try to click on the image with right and left mouse buttons
6. Check Enable GroundingDINO.
7. ???
8. Profit

What should have happened?

Dots should appear on the image after clicking at it.

Commit where the problem happens

webui: latest A11111 as for 14.04.23 extension: sd-webui-segment-anything 16.04.23

What browsers do you use to access the UI ?

No response

Command Line Arguments

@echo off

set PYTHON=
set GIT=
set VENV_DIR=
set COMMANDLINE_ARGS= --xformers --no-half-vae
set ATTN_PRECISION=fp16
call webui.bat

Console logs

No errors when clicking on dropped image

This error after entering text in GroundingDINO Detection Prompt:

127.0.0.1/:1 Uncaught (in promise) API Error
Promise.then (async)
(anonymous) @ index.4395ab38.js:76
(anonymous) @ index.4395ab38.js:4
le @ index.4395ab38.js:4
x @ index.4395ab38.js:79
(anonymous) @ index.4395ab38.js:4
(anonymous) @ index.4395ab38.js:4
u @ index.4395ab38.js:78
t.$$.update @ index.4395ab38.js:78
ql @ index.4395ab38.js:4
bt @ index.4395ab38.js:4
Promise.then (async)
yo @ index.4395ab38.js:4
Yl @ index.4395ab38.js:4
(anonymous) @ index.4395ab38.js:4
g @ index.4395ab38.js:34
i @ index.4395ab38.js:34
(anonymous) @ index.4395ab38.js:4
S @ index.4395ab38.js:79
i @ index.4395ab38.js:79
(anonymous) @ index.4395ab38.js:4
k @ index.4395ab38.js:78

Additional information

No response

[continue-revolution]

Your terminal must have thrown some exceptions. Check your terminal and let me know the error.

[Rootybooty]

Having the same issue. No exceptions here. Updated to the latest version of everything. The image just isn't interactable after uploading it. Left clicking does nothing. Right clicking shows the usual browser right click menu as shown here: https://i.imgur.com/XHVUpdI.png

[continue-revolution]

https://github.com/continue-revolution/sd-webui-segment-anything/issues/10

The reason is probably due to some other extensions. Could you share with me all extensions you have installed?

[Rootybooty]

Sure thing.

Edit: I went and disabled all the extensions above besides this one but still doesn't seem to work.

[Someonetoldme584]

Same issuie

[continue-revolution]

I really have no idea why. Someone has two PCs with similar environment, one can, one cannot, unable to really find a reason. A reason I guess might be that your browser somehow blocked my JavaScript, but I’m not sure. I can really not find a way to reproduce. My best advice is to pick another environment/different browser to re-try.

[B-rowser]

I have the same issue. Tried it with Chrome and Firefox. I should be getting terminal messages when I load new SAM models in right? Because I don't get those. So I suspect something is wrong.

[continue-revolution]

I have the same issue. Tried it with Chrome and Firefox. I should be getting terminal messages when I load new SAM models in right? Because I don't get those. So I suspect something is wrong.

Yes. If there is no change on both your terminal and your UI, then your server somehow disconnected to the browser. If you have updated your extension to the newest, you should get a message on your UI even if you simply click ‘preview segmentation’ without uploading an image.

[continue-revolution]

In the update when I post this message, when you simply upload an image without adding whatever point prompt or groundingdino and you have clicked Preview prompts, you should see these on your terminal

Start SAM Processing
Initializing SAM
Running SAM Inference (1520, 1152, 3)

and this on your browser

[continue-revolution]

if anyone here have time please update the extension and give it a try. I'm not sure where the problem is. I can only try narrowing down the problem. After checking this, you can try using GroundingDINO without point to see if it is working. If GroudingDINO is working, you can disable GroundingDINO and try adding point prompts.

[B-rowser]

To be clear the WebUI is still running I can load SD models and everything else seems to work just fine, interacting with the extension does not yield even an error message until I enable GroundingDINO which won't install properly and then returns an error, breaking the extension. No idea if copybaras has the same issue

[continue-revolution]

To be clear the WebUI is still running I can load SD models and everything else seems to work just fine, interacting with the extension does not yield even an error message until I enable GroundingDINO which won't install properly and then returns an error, breaking the extension. No idea if copybaras has the same issue

If your problem is about '_C', please follow the step:

1. go to nvidia website to download appropriate version of CUDA (you should be able to see the cuda version which match your torch version on the botton of the webui page)
2. Initiate your local terminal, switch to your virtual environment and run

   from torch.utils.cpp_extension import CUDA_HOME
   print(CUDA_HOME)

   If it's printing some path, then you are good to go.

3. Uninstall groundingdino via either pip uninstall groundingdino or you manually remove groundingdino directory inside your virtual environment, it is most likely something named venv (your virtual environment, assume you are using some bat on windows) then Lib then site-packages
4. re-initiate webui

Please do not skip any step.

[continue-revolution]

This _C problem has been asked multiple time but is very hard to resolve. #31 is about this.

I would assume @copybaras @Rootybooty @Someonetoldme584 have another very mysterious environment problem.

[Rootybooty]

I noticed that in one of your demos, you're running python 3.8.10. It might not be the issue but just wanted to confirm, this extension should work with python 3.10.6 release right?

[continue-revolution]

Python version should not be a problem. Higher version normally supports everything lower version supports

[Matti-Koopa]

I have the same problem. In a docker environment, if this helps. No additional extensions loaded (except stock). https://github.com/AbdBarho/stable-diffusion-webui-docker Shouldn't be hard to reproduce. Just install the container, add sd-webui-segment-anything extension to /data/config/auto/extensions and run auto like the wiki describes:

docker compose --profile download up --build
docker compose --profile auto up --build

[continue-revolution]

I have the same problem. In a docker environment, if this helps. No additional extensions loaded (except stock). https://github.com/AbdBarho/stable-diffusion-webui-docker Shouldn't be hard to reproduce. Just install the container, add sd-webui-segment-anything extension to /data/config/auto/extensions and run auto like the wiki describes:

docker compose --profile download up --build
docker compose --profile auto up --build

thanks, this is much more helpful. I'll try and see what's going wrong.

[Matti-Koopa]

I am able to provide more information. On an instance that works, this line in sam.js is executed first: image.addEventListener("click", event => { and afterwards this line in contextMenus.js is executed: gradioApp().addEventListener("click", function(e) {. On a non-working instance, the line on contextMenus.js is executed first and the line on sam.js is never executed at all.

Edit: It seems like on a working instance samTabPrefix() is txt2img_sam_ while on a non-working one it's _sam_ - which is just wrong. This causes the div never to be recognized by this line: const sam_image = gradioApp().getElementById(samTabPrefix() + "input_image"). Therefore the event is never attached in the onUiUpdate method.

I hope this is enough information for you to fix this.

[continue-revolution]

It seems that you understand javascript. From the logic of my samTabPrefix(), it first checks whether tabs exist, then check whether button exists, then check whether the first button or the second button is selected (txt2img or img2img). Could you help me to check what fails when you are in txt2img page?

function samTabPrefix() {
    const tabs = gradioApp().querySelector('#tabs');
    if (tabs) {
        const buttons = tabs.querySelectorAll('button');
        if (buttons) {
            if (buttons[0].className.includes("selected")) {
                return "txt2img_sam_"
            } else if (buttons[1].className.includes("selected")) {
                return "img2img_sam_"
            }
        }
    }
    return "_sam_"
}

[Matti-Koopa]

It seems like you determine the active tab using the selected css class. In a non-working instance this just isn't there. Active class is: bg-white px-4 pb-2 pt-1.5 rounded-t-lg border-gray-200 -mb-[2px] border-2 border-b-0 Inactive is px-4 pb-2 pt-1.5 border-transparent text-gray-400 hover:text-gray-700 -mb-[2px] border-2 border-b-0 So it seems to depend on the selected theme.

[continue-revolution]

My button[0] looks like this <button class="selected svelte-1g805jl">文生图/txt2img </button>. Could you show me what your button[0] looks like?

[Matti-Koopa]

<button class="px-4 pb-2 pt-1.5 border-transparent text-gray-400 hover:text-gray-700 -mb-[2px] border-2 border-b-0">txt2img </button><button class="bg-white px-4 pb-2 pt-1.5 rounded-t-lg border-gray-200 -mb-[2px] border-2 border-b-0">img2img </button><button class="px-4 pb-2 pt-1.5 border-transparent text-gray-400 hover:text-gray-700 -mb-[2px] border-2 border-b-0">Extras </button><button class="px-4 pb-2 pt-1.5 border-transparent text-gray-400 hover:text-gray-700 -mb-[2px] border-2 border-b-0">PNG Info </button><button class="px-4 pb-2 pt-1.5 border-transparent text-gray-400 hover:text-gray-700 -mb-[2px] border-2 border-b-0">Checkpoint Merger </button><button class="px-4 pb-2 pt-1.5 border-transparent text-gray-400 hover:text-gray-700 -mb-[2px] border-2 border-b-0">Train </button><button class="px-4 pb-2 pt-1.5 border-transparent text-gray-400 hover:text-gray-700 -mb-[2px] border-2 border-b-0">Settings </button><button class="px-4 pb-2 pt-1.5 border-transparent text-gray-400 hover:text-gray-700 -mb-[2px] border-2 border-b-0">Extensions </button>

img2img selected tab here

[continue-revolution]

Thank you very much. This is super helpful. However, at this moment I have no good idea on how to resolve this problem. Please give me some time to find the best solution. Also, for everyone above, if you have time, please open webui on your browser, F12 open the console, click the top left arrow on the console and click the txt2img tab. You should see something like this on your console Please either screenshot or copy-paste this list of buttons information for me to see whether you are having the same problem.

[Matti-Koopa]

Digging a little bit deeper, I think this has something to do with the version of Gradio. They might have changed the CSS classes between versions. On my working instance, a much newer version is used (3.23.0) while on my non-working instance 3.16.2 is used. My guess is, that at some point some other extension required a newer version of Gradio there, while on the non-working instance it just uses the old version webui ships with out of the box.

So this might be fixed if you just force-install a newer Gradio version in your plugin requirements. I am unexperienced with Gradio development (and Python development in general) though and therefore am not able to verify that.

[continue-revolution]

Your frontend skill has already saved my life. I'd say some people might not have updated their webui. If they update their ui to 22bcc7be, the webui should force update gradio (happened to me when I updated my webui) Even if they can somehow bypass this error, they will meet other errors using an old version of webui (or an old version of gradio).

Anyway, gradio is VERY buggy, even for 3.23.0

@copybaras states to be using the newest webui. Please check gradio version on the bottom of the webui page to see if it is 3.23.0

[git9471163]

I met the same problem. A stable diffusion version that qiuye developed work well and the gradio: 3.23.0 was utilized. However, another stable diffusion version that I bought from taobao can meet the problem, whose gradio version was 3.16.2. I want to renew the gradio to 3.23.0 but do not know how to operate, can you give me some suggestions?

[lin88882001]

你好，这需要怎么调试，我不会代码，目前出错了，无法显示出来。看这上面显示是没有加载进去，不知道是什么软件？能否方便帮我看一下嘛？ Error loading script: sam.py Traceback (most recent call last): File "H:\chatAI\novelai-webui-aki-v3\modules\scripts.py", line 248, in load_scripts script_module = script_loading.load_module(scriptfile.path) File "H:\chatAI\novelai-webui-aki-v3\modules\script_loading.py", line 11, in load_module module_spec.loader.exec_module(module) File "", line 883, in exec_module File "", line 241, in _call_with_frames_removed File "H:\chatAI\novelai-webui-aki-v3\extensions\sd-webui-segment-anything\scripts\sam.py", line 20, in from scripts.auto import clear_sem_sam_cache, register_auto_sam, semantic_segmentation, sem_sam_garbage_collect, image_layer_internal, categorical_mask_image File "H:\chatAI\novelai-webui-aki-v3\extensions\sd-webui-segment-anything\scripts\auto.py", line 11, in from modules.paths import extensions_dir ImportError: cannot import name 'extensions_dir' from 'modules.paths' (H:\chatAI\novelai-webui-aki-v3\modules\paths.py)

load Sadtalker Checkpoints from H:\chatAI\novelai-webui-aki-v3\extensions\SadTalker\checkpoints Running on local URL: http://127.0.0.1:7860

[Jetbleu]

To be clear the WebUI is still running I can load SD models and everything else seems to work just fine, interacting with the extension does not yield even an error message until I enable GroundingDINO which won't install properly and then returns an error, breaking the extension. No idea if copybaras has the same issue

If your problem is about '_C', please follow the step:

1. go to nvidia website to download appropriate version of CUDA (you should be able to see the cuda version which match your torch version on the botton of the webui page)
2. Initiate your local terminal, switch to your virtual environment and run

from torch.utils.cpp_extension import CUDA_HOME
print(CUDA_HOME)

If it's printing some path, then you are good to go. 3. Uninstall groundingdino via either pip uninstall groundingdino or you manually remove groundingdino directory inside your virtual environment, it is most likely something named venv (your virtual environment, assume you are using some bat on windows) then Lib then site-packages 4. re-initiate webui

Please do not skip any step.

Just wanted to contribute a bit here after hours of struggling,

Just follow these steps as they are, I just wanted to add more information:

1. Be sure to download and install the right CUDA version that you are using for your Stable Diffusion
2. Don't skip the second step until you get the path installation of the CUDA installed in your computer, for me the path was like this Mydisk\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8
3. To remove the Grounding Dino go to the folder stable-diffusion-webui\venv\Lib\site-packages here just delete the "groundingdino" and "grounding-0.1.0.dist-info" folders
4. Launch SD web ui again and try to use grounding dino again, it should reinstall grounding dino and it should work fine now

[shenwang2]

安装dinno出现这个，该怎么解决

[lcolok]

To be clear the WebUI is still running I can load SD models and everything else seems to work just fine, interacting with the extension does not yield even an error message until I enable GroundingDINO which won't install properly and then returns an error, breaking the extension. No idea if copybaras has the same issue

If your problem is about '_C', please follow the step:

1. go to nvidia website to download appropriate version of CUDA (you should be able to see the cuda version which match your torch version on the botton of the webui page)
2. Initiate your local terminal, switch to your virtual environment and run

from torch.utils.cpp_extension import CUDA_HOME
print(CUDA_HOME)

If it's printing some path, then you are good to go. 3. Uninstall groundingdino via either pip uninstall groundingdino or you manually remove groundingdino directory inside your virtual environment, it is most likely something named venv (your virtual environment, assume you are using some bat on windows) then Lib then site-packages 4. re-initiate webui

Please do not skip any step.

When installing in Docker, if you encounter an issue about '_C', you can refer to this solution: https://github.com/IDEA-Research/GroundingDINO/issues/8#issuecomment-1559541382.

[luohj000]

你好，这需要怎么调试，我不会代码，目前出错了，无法显示出来。看这上面显示是没有加载进去，不知道是什么软件？能否方便帮我看一下嘛？ Error loading script: sam.py Traceback (most recent call last): File "H:\chatAI\novelai-webui-aki-v3\modules\scripts.py", line 248, in load_scripts script_module = script_loading.load_module(scriptfile.path) File "H:\chatAI\novelai-webui-aki-v3\modules\script_loading.py", line 11, in load_module module_spec.loader.exec_module(module) File "", line 883, in exec_module File "", line 241, in _call_with_frames_removed File "H:\chatAI\novelai-webui-aki-v3\extensions\sd-webui-segment-anything\scripts\sam.py", line 20, in from scripts.auto import clear_sem_sam_cache, register_auto_sam, semantic_segmentation, sem_sam_garbage_collect, image_layer_internal, categorical_mask_image File "H:\chatAI\novelai-webui-aki-v3\extensions\sd-webui-segment-anything\scripts\auto.py", line 11, in from modules.paths import extensions_dir ImportError: cannot import name 'extensions_dir' from 'modules.paths' (H:\chatAI\novelai-webui-aki-v3\modules\paths.py)

load Sadtalker Checkpoints from H:\chatAI\novelai-webui-aki-v3\extensions\SadTalker\checkpoints Running on local URL: http://127.0.0.1:7860

This is a problem caused by a different SD version, I also had this exception before, because your modules\path.py file is missing extensions_dir parameter, you can manually add the following code in this file to fix this exception: extensions_dir = os.path.join(data_path, "extensions")

[luohj000]

Your frontend skill has already saved my life. I'd say some people might not have updated their webui. If they update their ui to 22bcc7be, the webui should force update gradio (happened to me when I updated my webui) Even if they can somehow bypass this error, they will meet other errors using an old version of webui (or an old version of gradio).

Anyway, gradio is VERY buggy, even for 3.23.0

@copybaras states to be using the newest webui. Please check gradio version on the bottom of the webui page to see if it is 3.23.0

Your frontend skill has already saved my life. I'd say some people might not have updated their webui. If they update their ui to 22bcc7be, the webui should force update gradio (happened to me when I updated my webui) Even if they can somehow bypass this error, they will meet other errors using an old version of webui (or an old version of gradio).

Anyway, gradio is VERY buggy, even for 3.23.0

@copybaras states to be using the newest webui. Please check gradio version on the bottom of the webui page to see if it is 3.23.0

Today I deployed the latest version of SD on my cloud server, using gradio version 3.16.2, but this problem also appeared, combined with your conversation with Matti-Koopa, in fact the final problem performance is caused by CSS selector problems, I made an update to sam.js, and after testing can add point markers normally now. Because fixing this problem also took me longer, and I ended up seeing your conversation to fix the problem, so just a suggestion, could you do a compatibility update for this problem? That way others may not have to waste unnecessary time? The following code:

function samTabPrefix() {
    const tabs = gradioApp().querySelector('#tabs');
    if (tabs) {
        const buttons = tabs.querySelectorAll('button');
        if (buttons) {
            if (buttons[0].className.includes("selected") || buttons[0].className.includes("bg-white")) {
                return "txt2img_sam_"
            } else if (buttons[1].className.includes("selected") || buttons[1].className.includes("bg-white")) {
                return "img2img_sam_"
            }
        }
    }
    return "_sam_"
}

[continue-revolution]

@luohj000 I did not support gradio 3.16.2, while gradio >= 3.23.0 should be alright. Also, the latest webui requires gradio 3.31.0, I think users will be force-updated to that version if you update your webui.

However, I appreciate your contribution. Do you mind submitting a pull request?