gereleth / jupyter-bbox-widget

A Jupyter widget for annotating images with bounding boxes
BSD 3-Clause "New" or "Revised" License
118 stars 19 forks source link

support marimo notebooks #31

Open sinia opened 1 month ago

sinia commented 1 month ago

First of all many thanks for the awesome tool :)

Would it be possible to make this widget compatible with marimo notebooks? marimo supports jupyter widgets based on anywidget, but jupyter-bbox-widget does not seem to work for some reason.

The example below in marimo loads, displays widget interface, except that the actual image does not show up.

from jupyter_bbox_widget import BBoxWidget
widget = BBoxWidget(
    image='fruit.jpg',
    classes=['apple', 'orange', 'pear'],
)
widget
gereleth commented 1 month ago

I haven't tried the widget with marimo. But the fact that the UI shows up is promising already =).

Could you verify that there actually is an image file at the path 'fruit.jpg'? If this isn't a FileNotFoundError then I'll try to take a look during the weekend.

sinia commented 1 month ago

Yes, I confirm that the image is there.

gereleth commented 1 month ago

Ok, there looks to be some error in transferring binary image data to the frontend in marimo. I can get the image displayed by base64-encoding it. But then when I try to draw any bboxes marimo gives me this error:

An exception was raised by a UIElement's on_change handler:
traitlets.traitlets.TraitError: The 'image_bytes' trait of a BBoxWidget instance expected a bytes object, not the str ''.

So for some reason the widget gets an empty string instead of image bytes and then marimo complains about that. Even if we don't use the image_bytes trait at all by going the base64 route.

I don't really know what to do about this, maybe we should ask marimo whether they support traitlets.Bytes traits in widgets.

gereleth commented 1 month ago

I reported some obvious binary data related issues to marimo that they promptly fixed. But as of now the widget still doesn't work there. It's gotten harder to debug because the combo of (bbox widget + marimo) raises no more errors and just displays confusing and incorrect behaviour =).

So my current plan is to wait until marimo's support of anywidgets matures a little. Then I'll give this another try.

sinia commented 1 month ago

Yes, I agree. Thanks for looking into this!