Tensorflow Tag Suggestions plugin not working with OSD Annotorious 2.3.0+?

[MyleneSimon]

Hi, I am testing a few of the plugins with OpenSeaDragon Annotorious and was trying to combine the Tensorflow Tag Suggestions plugin with the Shape Labels plugin. I tested a variety of version combinations but was not able to get both plugins to work together.

From my tests, it looks like:

• Tensorflow Tag Suggestions works with OSD-Annotorious 2.2.0, but not with OSD-Annotorious 2.3.0+

  • with 2.3.0+ I get the following error after the first annotation (and then cannot create other annotations):
  • Uncaught (in promise) TypeError: Cannot read properties of undefined (reading 'snippet')

• Shape Labels works from OSD-Annotorious 2.4.1 and up (with versions before that, the labels are not displayed)

Am I missing something? Both plugins are pretty cool so it would be great to be able to use them together.

[rsimon]

Yes, both correct. ShapeLabels uses a feature that was added later. (2.4 sounds about right.) The TensorFlow plugin was only ever I tech demo, in a way. I never got to use it in a real project and, thus, ended up not having the bandwidth to maintain it. The fixes needed might not be huge, I guess. But, alas, too much to maintain, too little time ;-)

[MyleneSimon]

Fully understand that ;) We are evaluating replacements options for our in-house OSD annotation plugin (we use OSD a lot for visualizing large microscopy images), so it is great to have access to all the plugins to see what can be achieved, we can switch versions to check out the different plugins.

A few follow-up questions if you don't mind:

• has anyone created a plugin/demo similar to the tag suggestions but that would do some kind of segmentation or object detection instead?
• are there any known converters that would go from W3C annotations produced in Annotorious to mask image, or vice versa?

[rsimon]

Re segmentation: I have toyed with it a while back. But that code is even more unfinished than the tag suggestions :-( (Plus, it looks like I never actually committed the second half of the code, which translates the segmented raster to a vector polygon...)

I'm not aware anyone has continued working on this. Not in Annotorious at least, unfortunately.

Concerning the conversion between Web Annotation and raster masks: nothing that I'm aware of. But we should talk if you're interested. It looks like I may be building a conversion Annotorious-to-raster for one of my current projects.

[rsimon]

Re segmentation: I have toyed with it a while back. But that code is even more unfinished than the tag suggestions. (Plus, it looks like I never actually committed the second half of the code, which translates the segmented raster to a vector polygon...)

I'm not aware anyone has continued working on this. Not in Annotorious at least, unfortunately.

Concerning the conversion between Web Annotation and raster masks: nothing that I'm aware of. But we should talk if you're interested. It looks like I may be building a conversion Annotorious-to-raster for one of my current projects.

[rsimon]

P.S.: I'm really interested in the microscopy use case. (I heard of several users using Annotorious in a similar context.) One issue people have told me about is that they had lots of annotations, and Annotorious gets slow & sluggish after about 1.000-3.000 annotations. In case you're grappling with similar issues: just to say I'm in the process of rewriting Annotorious from scratch, on a WebGL basis. Will still take a while, but should then be good up to >100.000 annotations.

[MyleneSimon]

Sorry to follow-up a few months later, we are now starting to evaluate and test annotation tools more in depth, one of them being Annotorious :) Regarding the microscopy use case, this a the kind of images we often deal with (here displayed in our custom OpenSeaDragon-based viewer): https://isg.nist.gov/deepzoomweb/view/image/replica1 Right now I don't think that we have the amount of annotations you mentioned, but this is likely to change as we are looking into bootstrapping the annotation process for the user with some rough segmentation, and then having them refine these annotations as opposed to having to manually draw all annotations from scratch.

Some of the things we are looking at are:

• annotations to raster and raster to annotations conversions, the idea being that some of these annotations would be used for AI training,
• annotation shape modifications: it looks like for things like circles or polygons it is already available (by dragging parts of the contour), but we would have to implement something custom for freehand annotations, correct?
• ease of adding new drawing tools: do you think something like a magic wand type of tool could be a possible customization?
• collaborative capabilities: would the user auth info support having multiple users annotating the same image? would overlapping annotations be an issue?

[rsimon]

No worries! A few answers to your specific points:

annotations to raster and raster to annotations conversions

I built that annotations-to-raster conversion meanwhile. Eventually, it ended up a bit simpler than initially planned. (A script that takes Web Annotations as input, and paints the shapes to PNG raster as output. A small config file is used to map different tags to mask colors.) It wouldn't be a huge deal to implement this as some kind of service component, though.

Raster to annotations might be more tricky. Essentially a segmentation or flood-filling task, although (obviously) with fairly "clean" output material. Likewise, something that would have to happen "outside" of Annotorious, I guess, as some kind of companion service.

implement something custom for freehand annotations, correct?

Yes, probably. There is a freehand tool for Annotorious 2.x in principle. But if you wanted to make, for example, freehand corrections to an exiting polygon shape, that would be a new tool. The tool extension API for Anntorious v2 is a bit... ad hoc. But I think this will become much better with Annotorious v3, where you can build tools with Svelte. We'd have to talk about specifics, but I could see this become pretty useful for other people & use cases, too.

do you think something like a magic wand type of tool could be a possible customization?

That's absolutely stuff we are looking into! Still very early days. But I've done some prototypes a while back (for the same project that needed the raster output), experimenting with in-browser keypoint- and contour detection. See e.g.:

https://twitter.com/aboutgeo/status/1492485333082071044 https://twitter.com/aboutgeo/status/1487739631382802432 https://twitter.com/aboutgeo/status/1481302036721324038

And more recently, we've been looking into SegmentAnything, too of course. It's all a bit tricky to combine with OpenSeadragon, and our new 100k+ annotations WebGL renderer that's coming for Annotorious v3. But we're getting there.

Speaking of which: there's been lots of progress on a bigger open source annotation platform that we're building around Annotorious 3 (and RecogitoJS, Annotorious's text annotation counterpart). We're about to start a closed alpha testing phase soon, and a beta phase in October. If you are interested, we could schedule a call to talk about (and show) it. Perhaps some time in August or September? (BTW: the platform also includes full realtime collaboration for text & image annotation.)

If you're interested, do drop me a message & we can arrange a date! Contact options are at rainersimon.io.

[MyleneSimon]

All of that sounds great! I chatted with the team and the demo will be of interest, I will be in contact towards the end of August to arrange a date.

[rsimon]

Excellent, looking forward to it! (CC-ing @jamiefolsom)