show all annotations in unified mode

[bbernhard]

I am currently working on a button that allows to show all existing annotations for an image in unified mode. If I remember correctly, this idea was first proposed by you @dobkeratops. I always loved that suggestion, but never found time to work on that. But as the unified mode is now becoming more and more powerful, I think this would be another great addition to unified mode's feature set.

At the moment, I am thinking about adding a new "eye" button to the toolbar. e.g:

In order to show all annotations, one needs to press and hold the "eye button". The annotations are hidden again, when the button is released.

As always: suggestions & feedback on that are highly welcome :)

[dobkeratops]

Great idea. I agree the eye icon is a good choice. It’s reminiscent of photoshop layer visibility. It just happens to be “all layer visibility” vs “single layer”. Another photoshop inspired tool that might be useful is the “eye dropper” .. in photoshop it selects colour; I imagined this selecting any existing label under the cursor. That might be useful later in conjunction with the view all button. Very low priority suggestion however .. I wish GitHub had a way to prioritise .. but imagine that as part of a replacement for the explore/inspect mode..)

Incidentally if you’re worried about the complexity of that toolbar, does it allow spacers? (group the draw/select tools, then the visibility/viewport changes , then the rest?)

[bbernhard]

It’s reminiscent of photoshop layer visibility. It just happens to be “all layer visibility” vs “single layer”. Another photoshop inspired tool that might be useful is the “eye dropper” .. in photoshop it selects colour; I imagined this selecting any existing label under the cursor. That might be useful later in conjunction with the view all button. Very low priority suggestion however .. I wish GitHub had a way to prioritise .. but imagine that as part of a replacement for the explore/inspect mode..)

very cool idea - I really like that one! :)

Incidentally if you’re worried about the complexity of that toolbar, does it allow spacers? (group the draw/select tools, then the visibility/viewport changes , then the rest?)

that's indeed a good question, will look into that. At the moment, I am hiding some of those elements on smaller devices (like mobile screens) as we are running out of space there. But you are right, grouping (and maybe splitting the toolbar up in two rows) could really help here. Not sure if the UI framework supports that, but I'll definitely have a look!

[dobkeratops]

Actually instead of a dedicated eye dropper tool, maybe the select tool in conjunction with view all would handle it. Clicking any visible polygon could also select its label? How does that sound.. would the label change be unexpected, or the fact you had to enable all first would make it less of a surprise?

[bbernhard]

interesting idea - I think that could also be pretty cool :)

btw: I think I am now almost done with the first draft. I'll do a bit more testing tomorrow and will then push the changes to production (hopefully tomorrow). Would love to get some feedback on that then. As already mentioned, it's just a first draft. Fortunately, it wasn't that complicated to implement, so even if it turns out that it's not that suitable, I could easily rework it without much effort. But I think having a first version (even if its garbage) is a great base for discussion and improvement suggestions.

[bbernhard]

okay it's live now. In order to test it, you need to press and hold the "eye button". The annotations are shown as long as the button is pressed. When the button is released again, the annotations are hidden again.

edit: While testing the feature on the production system, I realized that there are still some performance issues :/. Especially if there's an image with a lot of polypoints. In that case it takes a few seconds between pressing the "eye button" and until the annotations are actually shown. I'll have a look whether I can do something to speed that up - because in the current state it's barely usable. I think the problem is the drawing library (fabric.js) that is used. Unfortunately, the drawing of polygons with a lot of polypoints seems to be pretty slow. I am thinking about using simplify.js (https://mourner.github.io/simplify-js/) in order to reduce the number of points - I think this could really speed up the drawing. I think for the "show annotation coverage", it doesn't really matter whether the polygons are shown pixel perfect or not - it should just give a quick overview of what's already covered with annotations in an image.

[dobkeratops]

Ok I’ve tried it out

• works as you describe on a desktop PC
• on an iPad , it doesn’t seem to work; I suspect the OS is intercepting “press and hold” and handling it differently (touchscreen UIs use press and hold for accessing more options sometimes)

Regarding the slow redraw and simplify idea, I must say if it needs simplifying it might not be useful .. that would change boundaries

Idea: Is it any faster to draw just outlines without vertices ? ( eg only draw the vertices for the current label?) - I was also going to suggest drawing the other labels in a different colour , or feinter (50% alpha?). You might be lucky eg if a lot of the slowdown is changing states between the outline and vertex rendering .. I notice it does seem to draw each outline over previous vertices,so I think the current implementation is swapping states more.

I did notice my own attempt at an image labelling tool also slowed with a lot of items.. it might just be the limitations of html5 canvas itself.

If it’s too difficult to make it useable I wouldn’t worry about it so much. An eye dropper (instead of view all) might be an alternative. It might be possible to draw all the outlines incrementally into a copy of the image bitmap, which gets refreshed if you move or delete any polygons (and view all would just toggle showing the raw or fully annotated version of the background) but it does sound like a lot of effort

[bbernhard]

on an iPad , it doesn’t seem to work; I suspect the OS is intercepting “press and hold” and handling it differently (touchscreen UIs use press and hold for accessing more options sometimes)

oh, good catch. I'll see whether I can change that behavior for tablets/smartphone (maybe switch to a single tap there?)

Regarding the slow redraw and simplify idea, I must say if it needs simplifying it might not be useful .. that would change boundaries

yeah, right, that could indeed be a problem. I've used simplify.js in the past a few times and so far, I've been always blown away by how much it was able to simplify the number of points in a polyline while still preserving it's shape. There is also demo on the author's website which is pretty impressive. I don't know if there are certain cases where the algorithm completely fails, but I was hoping that as we are only using it for visualization purposes, that it maybe won't matter that much if a polypoint is a few pixels off? So, in the backend we would still store the original (unmodified) polypoints. The same is true when you click on a label in the unified mode - then also the original (unmodified) polypoints would be loaded. The only time where the simplification would happen, is when the "eye button" is clicked. I was hoping that this quick visual overview could give an indication about how much of an image is already covered by annotations.

Many thanks also for all the other ideas - I really like the one with the bitmap. As you already mentioned, it's probably quite a bit of work to implement - but man, that's a really creative approach :D I'll read up a bit on the fabric.js documentation, maybe they have some functionality exposed that could help us here.

I think I might give simplify.js a quick try - I am really curious how/if that would solve the problem. It should be rather easy to implement and I can throw it away again if it's not working properly.

[bbernhard]

Ok, I've just tried it with simplify.js...and it doesn't make much difference. But to my surprise, removing the drag handles (the little white circles on top of the polygon) makes a huge difference - the delay is now completely gone. I guess, that's what you meant with this, right?

You might be lucky eg if a lot of the slowdown is changing states between the outline and vertex rendering .. I notice it does seem to draw each outline over previous vertices,so I think the current implementation is swapping states more.

Wow, I am really surprised by how much difference that actually makes. I think I should really read up a bit how a renderer actually works. :grinning:

I'll clean up the code a bit and probably push the changes to production in the next days :)

[dobkeratops]

Great news! Perhaps if it’s fast enough it could be a toggle instead of “hold to peek”, that could work the same on desktop and mobile? Perhaps just draw the drag handles for the current label? you could experiment with colour coding , but each colour change will be a state change.

Yes that’s what I was guessing. It’s also possible they don’t have a super optimised way of drawing those points (eg it might be calculating a little circle manually each time) .. you might again get lucky and find drawing unselected labels in a feinter colour is fast enough aswell

Sometimes renderers have some minimum optimum batch size between state changes but then again for coloured lines with thickness it’s also possible to do that efficiently with parameters on each vertex.

[bbernhard]

I just pushed the changes to production.

Great news! Perhaps if it’s fast enough it could be a toggle instead of “hold to peek”, that could work the same on desktop and mobile?

I've changed the default behavior now to toggle. Let's see if that works :)

It’s also possible they don’t have a super optimised way of drawing those points (eg it might be calculating a little circle manually each time)

I guess that's my fault :sweat_smile:. When I started using fabric.js only version 1.x was available which didn't support drag/drop handles for polygons - so I built that functionality on top of the publicly available API. I guess that's probably much slower than native support. I saw that the upcoming 4.x release (which is currently in beta) now supports that - so maybe it makes sense to switch to 4.x once it is stable.

you might again get lucky and find drawing unselected labels in a feinter colour is fast enough aswell

will definitely give that a shot :+1:

[bbernhard]

short update: I've tried to extend the functionality to show the label on hover, but unfortunately it's not always working nicely (sometimes it doesn't detect that the mouse "entered" an object) :/

I think that's a bug in fabric.js. But as we are running a pretty old version (Version 1.x) which is not supported anymore, I can't create a bug report for that. So I guess, I'll have look whether I can replace Version 1.x with the newest version. They introduced some breaking changes, but I hope it won't be that much effort to port that over :crossed_fingers:. The new version also improved quite a bit performance wise, so I think it will be worth the effort :)

I'll track the progress here: https://github.com/ImageMonkey/imagemonkey-core/issues/274

[dobkeratops]

As an alternative to hover detection or drawing label overlays perhaps you could do a bit of colour coding, although this will only work for a few categories.

A range of saturated colours (pure red, green,blue,cyan,magenta,yellow) would give 6 options that should stand out from eachother and natural image pixels

Possible ideas :-

• just give a unique colour to the 1st 5 annotated labels (hope the ordering puts the most obvious objects first in the list?)

• highlight components (head, wheel, hand*, ..) versus main things (man woman car etc)

• highlight a few key contrasts if a scene contains both (road vs pavement is the one that jumps out for SDC) *highlight moveable or moving objects versus static (people,vehicles, animals,tools,personal items, containers,foods) vs (buildungs mountains rivers, road,paths,grass,trees)

• plurals versus single objects ?

• rely on label list ordering (give a unique colour to the 1st 5 annotated labels)

• choose a colour coding specific to certain scene labels

• Choose a colour coding based on the users recently used labels .. “track the users N most recently used labels, and give the first 5 annotations found in the current image a unique colour, default red for all else” ... this might be a handy alternative without adding lots of UI for user “favourite label” management, or code to figure out “most common concurring labels per scene”. So if you focus on street scenes you’ll end up with “road,car,pavement,person,building“, if you focus on annotating people in detail you’d end up with “head,arm,leg,hand,foot”