suggestion r.e. parts

[dobkeratops]

I'd almost recomend disabling the parts (head/person etc) until there's a better structure for that workflow: you might end up with a lot of inaccurate data;

Basically with the street scenes it's often asking me to annotate 'head/person' .. sometimes I do try but it's too difficult because the rectangle boundary is about 4x as thick as the actual part i'm trying to cover.. you wont even be able to validate it either because that has the same problem. I'd be best off skipping this.. but what will other users do

The ideal would be to reserve 'parts' for specific instances, once you have a person bounding box, and have validated that it's got one person (extend the 'one/few/many'idea to individual annotations, because for scenes we sometimes need to draw a box around a group) - then show that region cropped, and ask "annotate the head of this person" (confident that it's one person , so that bounding box can be stored as a hierarchical child element)

there's many other ways it could be done (a case of finding what the path of minimum resistance) for example what about letting the user switch (as part of the skip dialog idea), or associating each image with a labelling strategy first ('scene -> annotate it's objects' vs 'focussed single object: annotate it's parts')

I think the predefined, curated labels can work (and I know the reason for wanting it - avoiding conflicting naming conventions, and spam/abuse) - it just needs a bit more structure to match what users can do with free form labelling

example

^^ no way of knowing if I did it right

one extreme to the other:-

^^ in this case, I want to skip 'person' and move straight onto 'head/person', 'hand/person'

[dobkeratops]

( I suppose you could make a judgement on the bounding box size: assume in all the data collected up until now that there's a fuzziness on the placement equal to the thickness of the outline; you could convert the rects that are smaller than that to a sort of 'point+errorRadius'

I suppose what you might have in mind here is the casual user with a touchscreen.. whilst with a mouse or even trackpad, it is possible to do near pixel-accurate bounding boxes )

[bbernhard]

yeah, you are right. At the moment, the sublabel "head" will be created automatically when uploading an image via web ui. So for all the photos you have uploaded via web ui a "head" entry was automatically created. I once thought that it might be an option to speed up labeling, but I guess it won't do us any good in the current situation.

For all the images that you pushed via script, I excplicitly disabled that behaviour (you can search for "add_sublabels" in the Python script - it should be set to false).

I'll disable it also in the webui and push it to producation during the next maintenance window (probably next weekend). I'll also start working on a "skip this annotation" feature - I guess this could be a useful, independently from the other improvements.

In the end I could see it going in the direction you described:

The ideal would be to reserve 'parts' for specific instances, once you have a person bounding box, and have validated that it's got one person (extend the 'one/few/many'idea to individual annotations, because for scenes we sometimes need to draw a box around a group) - then show that region cropped, and ask "annotate the head of this person" (confident that it's one person , so that bounding box can be stored as a hierarchical child element)

But that's a little bit more work to do...so I guess it makes sense to add the "skip annotation" first. A thing we probably should also consider: When can we say that a annotation is valid? Or in other words: When can we unlock the annotation of the "head"? Is it sufficient to have just one person saying "yes, that's a person" or do we need more? I guess sometimes one validation might not be enough, as people could miss something...or they accidentally clicked "yes". - In that case we would again end up in the situation that we would need to annotate something impossible. But on the other hand, if we need too many validations before we unlock "head" we probably kill parallelism.

[dobkeratops]

When can we say that a annotation is valid? Or in other words: When can we unlock the annotation of the "head"?

I suppose it might be enough after one person annotated it; .. actually trying to label head gives another opportunity to confirm the original ('done' / 'skip ->why?-> not a person')

I just uploaded a few more images today (10s not 100's).. just crops of the same videos, but I was trying to consciously pick images with 1-10 object-occurrences, rather than the scenes where perspective brings in huge numbers in the distance. I was thinking these will go better in your existing UI; Another thing bugging me though (I was going to post yet another issue..) is these are all from fish-eye action cam images, and once you crop you've got completely arbitrary distortions .. whole image training might want to know the shape of the camera. Again to mitigate that I was trying to make uneven shapes (-> 'is the aspect ratio close to 16:9 - yes, it's probably a whole image; no - it's probably a crop')

good example of context.. I actually see a person and 3 cars, .. the 2 most distant objects (person, car) are hardly individually nrecognisable, but inferable from the surrounding context of the road & pavement

[bbernhard]

I just uploaded a few more images today (10s not 100's).. just crops of the same videos, but I was trying to consciously pick images with 1-10 object-occurrences, rather than the scenes where perspective brings in huge numbers in the distance. I was thinking these will go better in your existing UI;

cool, many thanks!

Another thing bugging me though (I was going to post yet another issue..) is these are all from fish-eye action cam images, and once you crop you've got completely arbitrary distortions .. whole image training might want to know the shape of the camera.

out of interest, do you know if labelme is taking care of that somehow? I am wondering if they already came up with a solution/best practice.

Regarding the "not-annotable" option. What do you think about the following workflow:

• there will be some UI element (probably a button?) where you can mark the annotation task as not-doable (do we need to ask for a reason here or is it sufficient to say that it's just not doable?)
• if you are not authenticated you might get the same annotation task twice (in case are unlucky), as we do not know anything about your identity.
• if you are authenticated the system remembers your decision and won't serve you the annotation task again - but the annotation task will still be served to other people.
• we probably want to make the implications of your decision clear. I am thinking about a "Are you really sure?" dialog here, that informs you, that you might not see the image again. (I am afraid that people otherwise might use that button to skip annotation tasks that they don't want to do now.)

Would that be a viable workflow?

[dobkeratops]

there will be some UI element (probably a button?) where you can mark the annotation task as not-doable Would that be a viable workflow?

yes, BUT it should probably be a general 'skip' rather than not-doable. It might be possible, just not interesting for the user's current mood, or in my case I just deem it 'sub-optimal' (e.g. photos of a hand holding an object and it asks 'annotate all occurrences of person': I would rather just draw a quick box around 'hand', instead of the fiddlier task of outlining the arm+hand as 'person', and move on);

Also the need to skip should be temporary: the goal is to extend the system such that you can always provide corrective information. ('too many+too small' -> eventually it will just ask you to 'indicate the general area containing <$x>' instead of 'annotate all occurences'. Complex trees will become 'trunk'+'general area of foliage' etc.)

If you start adding 'part' tasks for each sufficiently large annotation (certainly anything >10% of the image), I think you'll have a nice growing pile of sensible tasks, such that the 'skip' cases will re-appear infrequently.

[dobkeratops]

I am afraid that people otherwise might use that button to skip annotation tasks that they don't want to do now.)

ah, that's part of my intent: really you must expect and accept that people will do this. People are doing this voluntarily and they will be fickle as to the type of annotations they want to do.. I've got my own interest here (e.g. possible art resource for indy-games, and a desire for decent open-source self-driving cars, etc) ... so there's plenty of cases which might be doable, they are just of no interest to me. My own interest is busy real scenes, but I'm sure other users might prefer the focussed object images (e.g. I know a lot of AI people wanted to prove the ability to distinguish dog-breeds, and some people will want systems that can recognise emotions from facial expressions, etc etc). I dont personally want to trace outlines of people: but I do want to place information that indicates their skeleton (rough boxes around the head, hands,feet, elbows, knees, shoulders,hips,torso). I might eventually like to do accurate outline annotation but (from my own experiment) I think a decent/efficient vertex/edge/poly editor for the masses is hard .. IMO nested bounding boxes are the optimal blend of ease/expressivity.

So I would just have one 'skip' button as a catch all, and later figure out how to correct the whole workflow for the tricky cases we mentioned (it could be a second dialog that 'skip' opens).

Tangentially I was thinking r.e. "annotation preference" , what if the whole page just had an option ('next:?' as a dropbox to optionally specify a specific type, or 'more of the current one'). It could be a faint shade in the corner if you're worried about too many controls. I'm a bit worried about authentication being needed (to store a user preference) because that's another barrier to contribution (creating a login , remembering passwords etc ... and a whole extra dimension to the UI ) - I hope with enough well chosen simple fixes, we wont need to resort to 'advanced' options. (Actually for my preference of scene editing, it would also be fine if you could jump into annotation from the image browser.. I think that side of label-me is actually good.. you can see the remaining work)

Most of my issues would be fixed by being able to explicitely switch into 'parts' mode, and having an explicit "one / group" option ' (imagine the plural form being shown greyed out: "annotate each [car]/[group of cars]"), and you can click it to toggle). Some of the cases that are too small would be perfectly doable, it's just the current thick-outline makes it too hard .. just make a single pixel transparent outline, and it's fixed.

[dobkeratops]

imagine if the labels have 3 potential states:-.. (this should apply to all labels ... and later you'll need a distinct idea for uncountable/continuum objects, i.e. road/pavement/grass/water)

• 'person/people' (i guess 'person(s)' is ok too), 'car(s)' , 'tree(s)' (<<<that should be the default; you should retroactively assume the existing annotations are this) "cover all the areas containing person/people"

• just this minor tweak to the UI ( implication of what the label meaning it) would let users continue without this tension of not knowing/or not being able to handle the situation accurately

• 'individual person' (after another 'validation/quiz' step confirms this bounding box contains one)

• 'group of people' (.. if a user confirmed the bounding box contains several)

[bbernhard]

ah, that's part of my intent: really you must expect and accept that people will do this. People are doing this voluntarily and they will be fickle as to the type of annotations they want to do.. I've got my own interest here (e.g. possible art resource for indy-games, and a desire for decent open-source self-driving cars, etc) ... so there's plenty of cases which might be doable, they are just of no interest to me. My own interest is busy real scenes,

totally understandable. I probably explained it the wrong way. What I meant with

I am afraid that people otherwise might use that button to skip annotation tasks that they don't want to do now

is, that you shouldn't be using this option, if you want to work on the annotation task later, because it blacklists this task for you. If you aren't in the mood for annotating a big scene with a lot of objects, you can skip it temporarily by using the browser's refresh button (then you would get another random annotation task from the queue - if there are enough tasks in the queue, you almost always get a different one.). But if you want to remove it from your queue (as you are not interested in that type of task), you could blacklist the entry. (I am not using "skip" here, as "skip" for me implies that you just skip it temporarily.)

Tangentially I was thinking r.e. "annotation preference" , what if the whole page just had an option ('next:?' as a dropbox to optionally specify a specific type, or 'more of the current one'). It could be a faint shade in the corner if you're worried about too many controls. I'm a bit worried about authentication being needed (to store a user preference) because that's another barrier to contribution (creating a login , remembering passwords etc ... and a whole extra dimension to the UI ) - I hope with enough well chosen simple fixes, we wont need to resort to 'advanced' options.

I think that should be doable. I am also in favor of a authentication-less system, as it lowers the entry barrier dramatically. But for some things (like blacklisting) pictures I think we can't get around that. I guess there could be a need for blacklisting tasks, as people have different interests and they might not want to get tasks that they are not interested in.

Actually for my preference of scene editing, it would also be fine if you could jump into annotation from the image browser.. I think that side of label-me is actually good.. you can see the remaining work)

yeah, right, that's definitely a good idea. I'll put that also on my list :)

imagine the plural form being shown greyed out: "annotate each [car]/[group of cars]"), and you can click it to toggle).

Like that one! I think that should also be fairly easy doable with the current database schema - i'll have a look.