Multiple Annotation per image

[thomassajot]

Is it possible to have multiple label for the same sample (image, text, ...). Example use-case, related to janfreyberg/superintendent#48, is to identify all polygons within the same image and being able to see them all at once.

A workaround so far is to duplicate the sample n times. However, if not enough duplicates are provided, then the user has to copy the labels, and recreate the Labeller with more duplicates.

[janfreyberg]

Hi @UrbanSolution

This should work already - you can certainly add multiple polygons to one image using the superintendent / ipyannotate combination.

Simply:

1. Start a polygon by clicking on the corners
2. "finish" it by clicking on the starting point
3. Start a new polygon and repeat the process
4. Once you've added all you need, click "Submit"

does this not work for you?

[thomassajot]

Sorry, the post is not specific enough. It is indeed possible to get multiple polygons per image.

I am trying to get the text within a Polygon. Which means that for a given image, the goal is extract multiple polygons along with the text that it contains. I did not manage to build this part.

[janfreyberg]

OK, I see. That is a slightly more difficult issue, but I can see why you would need this. I will implement this, but it might not be for a while. You could try "composing" the widget yourself, using this strategy:

• Make a superintendent widget with bounding box annotations (using ipyannotations)
• Compose this widget inside a ipywdidgets.VBox, alongside a textbox
• use an on_submit callback on the ipyannotations to access the value of this text box widget and somehow add it to your data (e.g. set the value of the ipyannotations Annotator widget

[thomassajot]

I will try it. Thank you for the help. I would like to contribute to this library. Is there any good first issue task that you could add on the Issue board to get started ?

[thomassajot]

Hi @janfreyberg. I recently tried to add the text along with the annotation. The submit button will provide multiple polygon at once, whereas the text box refers to only one of them. The following snippets and image shows that we lose the connection between a polygon and the text it contains. Would you have some advices about how to tackle this issue ?

from ipyannotations import PolygonAnnotator
from ipywidgets import VBox, Textarea, HBox

text = Textarea(description='Text:', layout={'width': '99%'})
widget = PolygonAnnotator(canvas_size=[800, 1000], options=list(map(str, range(10))))
widget.display("../../data/Datasets/Master_Invoices/invoice-002181-lay008.png")
widget.children = widget.children[::-1]
data_widgets = widget.children[0].children[1]
data_widgets.children = (data_widgets.children[0],
                         VBox((data_widgets.children[1], text)),
                         *data_widgets.children[2:])

widget.on_submit(lambda x: x.update(text=text.value))
widget

[janfreyberg]

Hi @thomassajot,

thanks for your message! And apologies for not responding about contributing - I was busy at the time and must have just missed it.

That looks like an interesting usecase and I like the UI you have with the Text box. But it sounds like there needs to be better integration between the PolygonAnnotator widget and the text area widget, so that the "active" polygon (the one you've most recently created / are still creating) has a unique text associated with it that you gets stored alongside it.

I think the way to achieve this is to edit the following class:

https://github.com/janfreyberg/ipyannotations/blob/master/src/ipyannotations/images/annotator.py#L48-L52

You probably want to add an optional argument to the Annotator class to set the "class selector" to a Textarea widget - the rest should follow from there.

However, this means you can't both a class label and a free-text string. Is that something you need? If so, it's probably better to add an additional widget, instead of replacing the "class selector" widget.

[janfreyberg]

PS: I will move this issue over to ipyannotations, as that is probably where the changes need to be made.

[thomassajot]

Hi @janfreyberg In this use case, there is a need to both add a class and extract the text from it (the task is Information extraction).

[janfreyberg]

Hi,

OK - thanks for the feedback. I think I have a solution that works quite well and is generalisable.

I have pushed a new branch called extra-data-input. If you could try and install the library from that branch and test it, I would appreciate any feedback.

The way it works:

• you pass any additional inputs in a dictionary, called extra_inputs
• then, when you modify those additional input widgets, the most recent annotation receives the data from that widget as an additional value in the dictionary, under the key under which the input widget was passed

for example:

from ipyannotations import BoxAnnotator
import ipywidgets as widgets

input_widget = BoxAnnotator(
    options=["a", "b"],
    extra_inputs={
        "text": widgets.Textarea(description="Text:")
    }
)
input_widget.display("https://doc.laserfiche.com/laserfiche.documentation/english/docs/Subsystems/ProcessAutomation/Content/Images/previewtestvalues.png")
input_widget

This looks like:

After you draw a box, you can enter text (e.g. John Smith, as I have done in the screenshot). This is then accessible in the input_widget.data attribute:

input_widget.data

# Output:
# [{'type': 'box',
#   'label': 'a',
#   'xyxy': (527, 120, 575, 138),
#   'text': 'Invoice'},
#  {'type': 'box',
#   'label': 'a',
#   'xyxy': (74, 195, 143, 216),
#   'text': 'John Smith'}]