Draw a bounding box on top of an image

[Sumith1896]

• [Y] I have searched to see if a similar issue already exists.

We were looking to prepare a demo for an inpainting tool that accepts an image and a bounding box drawn on top of the image. Similar to this https://huggingface.co/spaces/akhaliq/lama but inpainting a bounding box region instead of brush scribbles.

Currently, the best we can do is ask the user to input coordinates of the bounding box. Could a feature for drawing a bounding box be added? I think the issue here is tangentially related: https://github.com/gradio-app/gradio/issues/1836

Additionally, even for the above inpainting demo, being able to control the brush size would be really helpful. Masking out a large region of an image currently seems very difficult to do.

Thank you! Sumith

[gpantaz]

Hello, I am also looking for something similar, is there any update on this?

[abidlabs]

Hi @gpantaz currently this is not possible, but we are going to make it possible for users to create custom components to make such modifications to existing components easier. This is on the roadmap!

[gpantaz]

Thanks for the quck response, looking forward to it :)

[pkuliyi2015]

I wonder if we have the possibility to implement by ourselves with necessary hacking tricks? My project really need this feature and I do not how to achieve this (I'm willing to do some hacks). Just for confirm, If there's really no possibilities, I can temporarily switch to some alternative ways.

[j-min]

+1. This bounding box drawing feature would be very crucial for building demos for many recent layout-guided image generation/editing models. If it's hard to support with current components, then would you please guide how we can achieve a workaround by editing existing components?

[pkuliyi2015]

I have done it with custom js in my extension. Please give me a star if you use that.

[gpantaz]

I managed to use existing components to create a (very) custom app that draws and annotates bboxes on images. Let me know if you would like me to share it in a public repo or something :)

[j-min]

@pkuliyi2015 @gpantaz Sounds good, if you have a public repo, would you please point out the code lines to your implementations?

[pkuliyi2015]

It lies in the only js file in my automatic1111 extension. I didn't rely on gradio to paint it.

[johnson7788]

thanks @ pkuliyi2015 , i wrote one : https://github.com/johnson7788/gradio_bbox_labeling

[gpantaz]

Hi, I ended up using this one https://github.com/gpantaz/image_bbox_app

Its not much, just a gallery component where I draw bounding boxes using open-cv.

[X1AOX1A]

I write a demo with streamlit and streamlit_drawable_canvas that can received multi-bboxes inputs, hope it can help you.

[joytsay]

Yeah so I needed this to work, this is a work around Using sliders to achive Region Of Interest (ROI) rect [x,y,w,h] is just cumbersome The following will draw ROI on top of input image You click on the left image to get the rect and show on right: with the use of gr.AnnotatedImage and image gr.Image.selectto get mouse click coordinate here is the simple code:

import gradio as gr

ROI_coordinates = {
    'x_temp': 0,
    'y_temp': 0,
    'x_new': 0,
    'y_new': 0,
    'clicks': 0,
}

def get_select_coordinates(img, evt: gr.SelectData):
    sections = []
    # update new coordinates
    ROI_coordinates['clicks'] += 1
    ROI_coordinates['x_temp'] = ROI_coordinates['x_new']
    ROI_coordinates['y_temp'] = ROI_coordinates['y_new']
    ROI_coordinates['x_new'] = evt.index[0]
    ROI_coordinates['y_new'] = evt.index[1]
    # compare start end coordinates
    x_start = ROI_coordinates['x_new'] if (ROI_coordinates['x_new'] < ROI_coordinates['x_temp']) else ROI_coordinates['x_temp']
    y_start = ROI_coordinates['y_new'] if (ROI_coordinates['y_new'] < ROI_coordinates['y_temp']) else ROI_coordinates['y_temp']
    x_end = ROI_coordinates['x_new'] if (ROI_coordinates['x_new'] > ROI_coordinates['x_temp']) else ROI_coordinates['x_temp']
    y_end = ROI_coordinates['y_new'] if (ROI_coordinates['y_new'] > ROI_coordinates['y_temp']) else ROI_coordinates['y_temp']
    if ROI_coordinates['clicks'] % 2 == 0:
        # both start and end point get
        sections.append(((x_start, y_start, x_end, y_end), "ROI of Face Detection"))
        return (img, sections)
    else:
        point_width = int(img.shape[0]*0.05)
        sections.append(((ROI_coordinates['x_new'], ROI_coordinates['y_new'], 
                          ROI_coordinates['x_new'] + point_width, ROI_coordinates['y_new'] + point_width),
                        "Click second point for ROI"))
        return (img, sections)

with gr.Blocks() as demo:
    with gr.Row():
        input_img = gr.Image(label="Click")
        img_output = gr.AnnotatedImage(label="ROI", 
                                       color_map={"ROI of Face Detection": "#9987FF",
                                                  "Click second point for ROI": "#f44336"})
    input_img.select(get_select_coordinates, input_img, img_output)

if __name__ == '__main__':
    demo.launch(inbrowser=True)

there is a known issue for gr.AnnotatedImage with smaller image the AnnotatedImage results gives shift rect results due to alignment of input image to the left upper corner: larger images shows correct results

[abidlabs]

Hey! We've now made it possible for Gradio users to create their own custom components -- meaning that you can write some Python and JavaScript (Svelte), and publish it as a Gradio component. You can use it in your own Gradio apps, or share it so that anyone can use it in their Gradio apps. Here are some examples of custom Gradio components:

• A "Rich Textbox" that allows you to write bold/italics/colored text
• A "Folium Map Viewer" component that allows you to use interactive maps

You can see the source code for those components by clicking the "Files" icon and then clicking "src". The complete source code for the backend and frontend is visible. In particular, its very fast if you want to build off an existing component. We've put together a Guide: https://www.gradio.app/guides/five-minute-guide, and we're happy to help. Hopefully this will help address this issue.

[abidlabs]

Good news, there's now a custom Gradio component that lets you do this! Please try out the gradio-image-prompter here: https://github.com/PhyscalX/gradio-image-prompter