[Bug]: API GroundingDino does not work ?

psykokwak-com commented 1 year ago

Is there an existing issue for this?

[X] I have searched the existing issues and checked the recent builds/commits of both this extension and the webui

Have you updated WebUI and this extension to the newest version?

[X] I have updated WebUI and this extension to the most up-to-date version

Do you understand that you should go to https://github.com/IDEA-Research/Grounded-Segment-Anything/issues if you cannot install GroundingDINO?

[X] My problem is not about installing GroundingDINO

What happened?

I try the GroundingDino API.

Steps to reproduce the problem

Here is the test script :

import base64
import requests
from PIL import Image
from io import BytesIO

url = "http://127.0.0.1:7860/sam-webui/image-mask";

def image_to_base64(img_path: str) -> str:
    with open(img_path, "rb") as img_file:
        img_base64 = base64.b64encode(img_file.read()).decode()
    return img_base64

payload = {
    "image": image_to_base64("out1.png"),
    "prompt": "body",
    "box_threshold": 0.3
}
res = requests.post(url, json=payload)

print(res)

for dct in res.json():
    image_data = base64.b64decode(dct['image'])
    image = Image.open(BytesIO(image_data))
    image.show()

The Execution is :

C:\Users\Jyce\Desktop>stable-diffusion-webui\venv\Scripts\python.exe sagd.py
<Response [500]>
Traceback (most recent call last):
  File "C:\Users\Jyce\Desktop\sagd.py", line 23, in <module>
    image_data = base64.b64decode(dct['image'])
TypeError: string indices must be integers

What should have happened?

Generate the mask images ?

Commit where the problem happens

webui: 22bcc7be428c94e9408f589966c2040187245d81 extension: a5c000f05f8c1f86be6d814d78d58848466ac7a7

What browsers do you use to access the UI ?

No response

Command Line Arguments

Launching Web UI with arguments: --xformers --api --gradio-img2img-tool color-sketch

Console logs

Start SAM Processing
Running GroundingDINO Inference
Initializing GroundingDINO GroundingDINO_SwinT_OGC (694MB)
final text_encoder_type: bert-base-uncased
C:\Users\Jyce\Desktop\stable-diffusion-webui\venv\lib\site-packages\transformers\modeling_utils.py:768: FutureWarning: The `device` argument is deprecated and will be removed in v5 of Transformers.
  warnings.warn(
Initializing SAM
Running SAM Inference (512, 512, 3)
SAM inference with 2 boxes, point prompts disgarded
Creating output image
API error: POST: http://127.0.0.1:7860/sam-webui/image-mask {'error': 'AttributeError', 'detail': '', 'body': '', 'errors': "'list' object has no attribute 'save'"}
Traceback (most recent call last):
  File "C:\Users\Jyce\Desktop\stable-diffusion-webui\venv\lib\site-packages\anyio\streams\memory.py", line 94, in receive
    return self.receive_nowait()
  File "C:\Users\Jyce\Desktop\stable-diffusion-webui\venv\lib\site-packages\anyio\streams\memory.py", line 89, in receive_nowait
    raise WouldBlock
anyio.WouldBlock

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Jyce\Desktop\stable-diffusion-webui\venv\lib\site-packages\starlette\middleware\base.py", line 78, in call_next
    message = await recv_stream.receive()
  File "C:\Users\Jyce\Desktop\stable-diffusion-webui\venv\lib\site-packages\anyio\streams\memory.py", line 114, in receive
    raise EndOfStream
anyio.EndOfStream

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Jyce\Desktop\stable-diffusion-webui\modules\api\api.py", line 145, in exception_handling
    return await call_next(request)
  File "C:\Users\Jyce\Desktop\stable-diffusion-webui\venv\lib\site-packages\starlette\middleware\base.py", line 84, in call_next
    raise app_exc
  File "C:\Users\Jyce\Desktop\stable-diffusion-webui\venv\lib\site-packages\starlette\middleware\base.py", line 70, in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
  File "C:\Users\Jyce\Desktop\stable-diffusion-webui\venv\lib\site-packages\starlette\middleware\base.py", line 108, in __call__
    response = await self.dispatch_func(request, call_next)
  File "C:\Users\Jyce\Desktop\stable-diffusion-webui\modules\api\api.py", line 110, in log_and_time
    res: Response = await call_next(req)
  File "C:\Users\Jyce\Desktop\stable-diffusion-webui\venv\lib\site-packages\starlette\middleware\base.py", line 84, in call_next
    raise app_exc
  File "C:\Users\Jyce\Desktop\stable-diffusion-webui\venv\lib\site-packages\starlette\middleware\base.py", line 70, in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
  File "C:\Users\Jyce\Desktop\stable-diffusion-webui\venv\lib\site-packages\starlette\middleware\gzip.py", line 24, in __call__
    await responder(scope, receive, send)
  File "C:\Users\Jyce\Desktop\stable-diffusion-webui\venv\lib\site-packages\starlette\middleware\gzip.py", line 44, in __call__
    await self.app(scope, receive, self.send_with_gzip)
  File "C:\Users\Jyce\Desktop\stable-diffusion-webui\venv\lib\site-packages\starlette\middleware\exceptions.py", line 79, in __call__
    raise exc
  File "C:\Users\Jyce\Desktop\stable-diffusion-webui\venv\lib\site-packages\starlette\middleware\exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "C:\Users\Jyce\Desktop\stable-diffusion-webui\venv\lib\site-packages\fastapi\middleware\asyncexitstack.py", line 21, in __call__
    raise e
  File "C:\Users\Jyce\Desktop\stable-diffusion-webui\venv\lib\site-packages\fastapi\middleware\asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "C:\Users\Jyce\Desktop\stable-diffusion-webui\venv\lib\site-packages\starlette\routing.py", line 718, in __call__    await route.handle(scope, receive, send)
  File "C:\Users\Jyce\Desktop\stable-diffusion-webui\venv\lib\site-packages\starlette\routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "C:\Users\Jyce\Desktop\stable-diffusion-webui\venv\lib\site-packages\starlette\routing.py", line 66, in app
    response = await func(request)
  File "C:\Users\Jyce\Desktop\stable-diffusion-webui\venv\lib\site-packages\fastapi\routing.py", line 237, in app
    raw_response = await run_endpoint_function(
  File "C:\Users\Jyce\Desktop\stable-diffusion-webui\venv\lib\site-packages\fastapi\routing.py", line 163, in run_endpoint_function
    return await dependant.call(**values)
  File "C:\Users\Jyce\Desktop\stable-diffusion-webui\extensions\sd-webui-segment-anything\scripts\api.py", line 53, in process_image
    response = [{"image": pil_image_to_base64(mask)} for mask in masks]
  File "C:\Users\Jyce\Desktop\stable-diffusion-webui\extensions\sd-webui-segment-anything\scripts\api.py", line 53, in <listcomp>
    response = [{"image": pil_image_to_base64(mask)} for mask in masks]
  File "C:\Users\Jyce\Desktop\stable-diffusion-webui\extensions\sd-webui-segment-anything\scripts\api.py", line 28, in pil_image_to_base64
    img.save(buffered, format="JPEG")
AttributeError: 'list' object has no attribute 'save'

Additional information

Extra bonus : Is it possible to add an option on the API to select the "expand mask" value ?

continue-revolution commented 1 year ago

@jordan-barrett-jm

jordan-barrett-jm commented 1 year ago

Currently investigating

jordan-barrett-jm commented 1 year ago

The issue was related to the API expecting the sam_predict masks at an incorrect index. This has been fixed. Additionally, a "padding" param can now be optionally passed in the payload that will pad the masks. It should be an integer.

psykokwak-com commented 1 year ago

It works. Thanks.

continue-revolution / sd-webui-segment-anything