suggestion - non-rectangular areas

[dobkeratops]

imagine if the default shape of an annotation was was a rectangle with some % (e.g. 10-20%) of the corners clipped off to make an octagonal polygonal region .. then you go in and tweak for further adjustment (e.g. changing the amount of each corner clipped , pinching toward an edge for more trapezoidal regions to approximate perspective, etc)

Another option might be Convex Polygons, less fiddly than arbitrary polygons because you could click vertices in any order, and it encloses them all for you

[bbernhard]

that's an interesting idea. I initially thought about letting the user select different shapes (similar to this: http://www.robots.ox.ac.uk/~vgg/software/via/via_demo.html) but my concern was that a shape selection maybe distracts some users (-> better keep it simple). But your idea could be a good compromise on that. :)

[dobkeratops]

that looks neat. (I wish GitHub had voting on issues.. I'll leave a note here to say I think the 'heirarchical labels'/'multiple labels' is the most important idea )

[dobkeratops]

more comments... just been trying to use http://labelme.csail.mit.edu/Release3.0/ .. I must admit I'm finding it fairly horrible to use (so it seems there's room for improvement), I'll list my opinions of that here incase it inspires ideas on how to do things better..

• 'ESC' doesn't work, if there is a key to cancel once you started drawing, I don't know what it is

  • you can click back on the polygon's first vertex, but it thinks you made a single point , instead of just cancelling it.. you still have to go through the 'create label dialogue' (looks like the 'erase' icon cancels.. that is weird)

• it would be nice to separate editing polys and entering the labels - you have to constantly switch from mouse/trackpad to keys (with simpler bounding boxes that might be less of a problem)

  • coming back to the simple 'icon idea' for broad categories: what if whilst in 'poly editing mode' you had a palette of rough common object categories ('vehicle, tree,building..) just as icons, easy to switch between, then you could blast through highlighting all the cars, all the trees etc.

• it would be nice to be able to repeat a label , e.g. having a 'current label' mode , such that repeated drawing would use that (set label = tree, highlight all the trees, set label =car, highlight all the cars, etc)

• polygon editing is clumsy when you can't refine it, e.g. going back to split edges, tweak points. (i've used 3d modellers.. of course they have more of a learning curve..) .. it would be nice to have an 'edit' tool i.e. splitting edges/moving vertices, which could modify polygons including those made with the rectangle. Then you could quickly 'rough things out', then go back and refine.

• despite the tool being around for a long time, I couldn't find several labels that sounded fairly obvious to me ("tree canopy", there's tree trunk), types of car ('hatchback', 'saloon'). When adding a new label it's not obvious what's going to happen with that.

*there's button to go back form the editing view to the collection

• browsing between the collections seems too restrictive, I don't see anything to browse by label or the usual 'tags based search' I've come to expect in image sites

• 'ESC' does cancel the whole polygon if you press it while entering the text after you made the polygon.. which is really annoying because sometimes you just want to cancel the 'autocomplete label dialogue' .. then you accidentally lose the polygon (that could be fixed if you could seperate area and label editing .. )

• you can select the existing areas whilst in the 'drawing tools', which does simplify the UI but makes it more fiddly .. because you keep accidentally getting the edit dialogue back when you just wanted to make another polyon/rectangle near to an existing one (seperate select/draw modes would be better IMO.. you've got the list on the side for pure selection).. although it does have 'pre-selection highlight', i guess.

• It groups the images into 'collections' , it does show labels in the thumbnails -which is nice, because you can browse to look for unlabelled images; it would be nice to have a search option there (show unlablled..)

• I wonder if you could have a labelling mode where you could keep your hands on the keyboard by allowing the cursor keys to move between regions .

I get the impression a friendly tool to handle all this is quite a lot of UI work, it will be interesting to see how your approach of simplified single-task UI's pans out

[bbernhard]

WOW, huge thanks for evaluating the service and writing down the things we can do better. Very much appreciated - thanks! This will for sure be useful!

One thing we probably also should consider are smartphone users. Do we want to support smartphone users as well or should they use primarily the App? From the sourcecode maintainence point of view it would really be better to have just one thing to maintain, but I am afraid that if the labeling gets more feature rich that we need a dedicated App to make it work smoothly.

[dobkeratops]

been using it a bit more today; Got the hang of it a bit more.. sat through making a few hundred labels (trees/grass, people, body parts) . It remains interesting because it's the only thing I can find so far that has the range of labelling that I'm interested in (scenes & components).

My comments still stand - it takes a while to get into it, and you could definitely speed up the workflow (the the repeated/common label hotkeys).

To re-iterate: it's alternating every time between the spatial/pointing based tools and the fiddly dialogue box (enter text with radio buttons and confirm) that slows it down.. constantly moving between points, UI elements, and the keyboard). I think you could still provide the same fidelity of information but through a smarter workflow (alternate between marking regions, then type in several labels at once)

[bbernhard]

Thanks a lot for the info - very much appreciated!

Just out of interest: Could you imagine doing such "hardcore labeling sessions" on a regular basis or does that get boring very quickly? I am just thinking whether it makes sense to create some sort of "PRO Mode" for users who really like to label things efficiently (with key bindings and all that stuff you suggested). So that users start per default start in the basic mode with the possibility to switch over to the advanced mode.

[dobkeratops]

Could you imagine doing such "hardcore labeling sessions" on a regular basis or does that get boring very quickly?

Yes, and Yes. :) Different people have different workflows.. firstly I'd say you have the right idea focussing on something 'very low effort', low learning curve at first, but similarly some people will focus for some length of time (even if they eventually get bored).

in any tools I've written, ironically I've always actually found it easier to start with hotkeys (ie. less effort to write than a mouse based interface.. thinking about the visual layout and so on)

I suppose you could have both extremes covered (with the 'simple reaction to an image', and a hotkey-driven labelling tool).

Playing around I must have clicked out 1000+ labels on LabelMe so far over a few sessions.. but it does get laborious (for the reasons mentioned): my thinking becomes that a sub-optimal UI is a sub-optimal way of using one's own time, so you don't want to put too much into it.

I'm probably using it for about an hour at a time. I wouldn't want to use it all day. I have the 'idealistic motivation' .. but my patience does wear thin

[bbernhard]

Many thanks - really appreciated ;)

The more I think about it, the more I like your approach with the hotkeys. I always thought that implementing such a "Pro Mode" would be a total waste of time as there is no need for that. But I think it makes definitely sense to implement such a thing. I'll for sure create a ticket before I start with it, to gether some ideas and feedback :)