feedback - tried using with wacom tablet

[dobkeratops]

so I tried using this with a wacom tablet instead of a laptop-trackpad; historically I wouldn't have tried it with geometric tools, only paint, but i was pleasantly surprised

(EDIT summaries inline to save you needing to read all of this..)

(i) it's much faster to add polygon outline points, making some of the more complex cases approachable , allowing tackling more buildings/trees which i've been skipping up until now( EDIT: after trying a load of trees, going for foliage instead is easier, see #154)

• this made me want to switch strategy r.e. occlusion assumptions, i.e. tracing around the occluded parts
  • however if you could see the annotations in context with eachother, you could see which ones are occluding e.g. if you see buildings occluded by cars, you could draw the cars, then draw the buildings in a layer behind it, knowing the combination of both outlines gives full pixel labelling. here you want to stay focussed on the image and handle a few labels in succession.

Drawing occluded parts is preferable IMO because it's giving information that a human can often figure out - also I think it emphasises for training that many objects give hints about what's usually behind them.. the lower part of a car viewed from a low angle is often road or pavement, the upper part is building/tree/sky.. whilst a car view from a higher angle is usually just infront of pavement/road; Similarly a human seeing a car behind something can infer that there is probably a road there too..

• LabelMe has an explicit occlusion flag. (i suspect people started out doing it by naming convention then after seeing loads of examples they added the flag). I'm assuming the curated label list is an opportunity to set default occlusion - moving/moveable objects > static objects/structures > ground types.. The best might be default occlusion which you could override ?

an explicit 'unoccluded' annotation could give a clipping hint

(ii) I found these complex cases often you want to do plural/area annotation rather than individuals .. whilst there are still some parts that you want to draw as individuals (leveraging the easier outlining) - increasing the demand for an explicit plural flag (edit: for trees, the solution is do foliage instead)

(iii) Downside: it's harder to be precise with the points - this makes me wonder if a precision (i.e blurring of the outline) could be specified (imagine drawing the red outline as 2 pieces - single pixel between the points, and a transparent halo indicating the size of the blur - or perhaps just doing it for the vertices, if it's difficult to handle that regarding outline direction changes.. the exact calculation of a blurred polygon outline is a nontrivial geometric task, but easier to do with a bitmap). this would be great for trees/grass anyway.

You probably want to make a default blurriness assumption based on the data you end up with.. you could always give a refinement option

(iv) the 'accidental close' bug matters more

Summary:-

I'd re-iterate the desire for plural and occlusion flags to clarify radically different annotations, and I still find myself really wanting to switch labels on the fly and think about dividing up the area .. also the stress caused by 'all' (should you leave parts as tree/bush, foilage - (which is often a better label), tree trunk assuming a future label for that,..)

(actually r.e. trees switching to looking for foliage instead definitely helps - the word seems more accurate for what you can do)

[dobkeratops]

there's still cases like this where switching to an inverse/anti label would be easier (i.e draw the holes rather than draw the tree)

I was going to suggest a label "unclouded sky" which could be used as a hole for the upper part of buildings and trees - you could assume with layering that it would cut anything else

[dobkeratops]

occlusion example- 'annotate building...' it's easy here to draw around the wooden pallet and metal barrel , then have a simpler outline of the building behind it .. you want to think about several elements together.. (but with the current setup and a tablet you have an OCD urge to just trace the given 'building' task around the foreground objects.. knowing we dont have labels for those. This emphasises that sometimes it would be advantageous to draw things even without labels, waiting for later labelling .. you could still give them depth information ("occluding object", "foreground object" , whatever)

[dobkeratops]

making more mistakes :) - hotkey requests: perhaps 'del' for the delete command, and 'v' 'm', 's' for the move/select tool (what's more intuitive? seems photoshop uses 'V' for move, but there's others for select)

[dobkeratops]

Here's a good example of occlusion ambiguity & lack of contextmaking you uncertain of strategy - here you really want to alternate between 2 labels to cover the area accurately (trees, buildings). It might even be safer to simplify the task by drawing a broad area with a blend "tree/building", then drawing patches of predominant tree, predominant building over that.

There's uncertainty because the tasks divided by label mean you dont know what the next user will choose, and they can't see what you already did to guide the best complement. (could this be fixed by seeing the existing annotations drawn feint?)

I think for this kind of example it's easier to think in terms of area coverage ("which pixels can I isolate? how do I describe those pixels?;.. which pixels remain un-described?.. whats the best order in which to cut the pixels out to describe the image optimally?")

[dobkeratops]

another good example of occlusion/layering ambiguity: a good description might be 'background tree/mountain' for the distant area, then a few 'foreground tree' infront

you could just try to ignore that and cover most of the image minus sky ,lake, person as 'tree' but that seems unsatisfactory given the separation of types you can see

[dobkeratops]

here it's almost possible to isolate individual trees, but the branches become unclear (i've got a label request tree trunk which would let you cut out the easily distinct instances)

[dobkeratops]

contrasting example - this is a rare example of an image with several distinct trees, easier to draw individuals. Identifying whole isolatable individual examples would be really useful for artists CG reference

[dobkeratops]

(idea with label blending... 'bark' for an area description of tree trunk.. you could do 'bark/foliage' as an explicit area version of tree.. bark might sometimes be nicer than tree trunk because it would handle being zoomed in on other branches too)

[bbernhard]

not related to that github ticket: Many thanks for all the tickets you created. I am a bit short on time, so I just scrolled a bit through it; will look into it on the weekends in detail. So, many thanks for all the suggestion & ideas - very much appreciated! :)

a short update from my side: Unfortunately, there are so many great ideas and not enough time to tackle them all at once. So I decided to focus on two topics, which will (hopefully) have a big impact and bring us forward.

• possibility to rework/improve existing annotation taks. At the moment it's all or nothing. If you are working on an annotation task, you always have to annotate all objects. My goal is to losen that restriction a bit and give users the possibility to improve existing annotation tasks or correct wrong ones. I am not 100% sure if it's possible to integrate something like this in the task based workflow, but i'll do my best to make it work (if it absolutely doesn't work out, then we are at least aware of the limits of the current solution -> "fail fast")

• possibility to define labels/attributes on a per annotation basis: For me personally that's one of the biggest features that are missing at the moment. Of course there are also a lot of other things to improve, but for me this one really stands out, as it's something completely new. I have an idea how such a mode could work, but I am not sure if it really is as great as I imagine it to be. I am pretty sure I won't nail it the first time..but I am hoping that a first implementation can serve as foundation for further improvement (similar to what happened with the annotation mode: there was a rudimentary implementation at first and while using it, you came up with the idea of using a browse based approach - which made it so much better)

oh yeah, and besides that: I'll also make another batch of trending labels productive in the next days :) Thanks a lot for all the contributions @dobkeratops - it's awesome to see so much activity on the site!

[dobkeratops]

there are so many great ideas and not enough time to tackle them all at once

right, I know how it is.. ideas always exceed what can be done in practice

I keep meaning to have another bash at actually compiling and running this tool, but just dipping in and giving intensive use probably helps just as much

I can do any markup offline in GIMP etc, but it's so much better to have it online contributing activity (maybe eventually i can do something to convert these to outlines to submit)

At the moment it's all or nothing. If you are working on an annotation task, you always have to annotate all objects.

I think this will be a big help :)

possibility to define labels/attributes on a per annotation basis: that would be rather interesting

another batch of trending labels productive

awesome. I've done some more experimental ideas for naming convention hints, which I'm sure you'll see