quick fix.. 'primary labels'/'priority labels'

[dobkeratops]

what if you could mark one (or more) of the labels on an image as a priority task.. and the random/default mode would serve these before any others (perhaps prefixing with an asterisk in the label list), eg *car,*road,*bin,building,tree,pavement,person for an image where the cars and road are easy to pick out, but the pavement is thin , and there's just a few pixels of people in the distance. *head/person,person for a closeup crowd scene. easy to pick out the heads first. etc

• the idea would be to mark the most prominent and easy objects (bigger) as such;
• the upload page (where it asks you to pick a label) could assume that is going to be a primary/priority label?

This could perhaps improve the situation of searching for easy tasks - if these labels are the more obvious things in the scenes.

I would see this as a temporary fix prior to getting a unified change-label-as-you-annotate mode; so the question is : would this extra complication just clutter the system (another thing to explain to users). would it be better to ignore this idea and just focus on a new unified mode and take the time to do it well.

one potential ambiguity is: this would be task planning , which might be different to selecting the most prominent label. e.g: a simple picture of a dog: dog is the best single label to describe the image, but I'd prioritise *head/dog, *paw/dog, *tail/dog, as the easy tasks to do first

[bbernhard]

You raised an interesting point here.

I think it would be really great to have, although I am not sure if it will be used much (as soon as we a have a unified mode). Personally, I am more the occasional contributor, that does a few contributions throughout the day (on my smartphone). Having such a prioritization would definitely help me here (there are only a certain type of annotations that one can do on a smartphone).

Would a prioritization add any additional value or will it be just for prioritizing the taks? If it's just for the latter, I am not sure if it's worth the effort. I mean, if it's heavily used, it's definitely worth it. But I am not sure if there will be much use, as soon as the unified mode is ready?

I am wondering, if we can implicitly assume that the first added labels are the most prominent ones? If that holds true, I think we could easily add another query option to query the dataset for those annotation tasks. We could do something similar to https://github.com/bbernhard/imagemonkey-core/issues/211.

Something along the lines of: car !order by priority desc //returns all the car annotation tasks ordered by priority (i.e: images where the label car was added either first or second will be shown first)

That's just a quick idea that came to my mind...not sure if it's a good one.

[dobkeratops]

I am not sure if it's worth the effort.

right that's what worried me about suggesting it - know a myriad of small features can themselves become overwhelming.. it might just distract from the big long term feature (unified mode).

I am wondering, if we can implicitly assume that the first added labels are the most prominent ones?

that's an interesting idea. I suppose you could still pick images at random, then chose their task from the next one in the given order. That could involve no new UI (just inform the user the order is used), unless you wanted to be able to edit it

[dobkeratops]

Ah another compromise suggestion:-

How about having a default priority set per label.. i.e. the label list itself is ordered.

the system could pick images at random to serve, but within the image present the labelling tasks in an order based on the default priority

I would suggests:-

(i) components (head/person etc) - but never add them automatically. rely on "add labels" mode for users to specify which images are worth doing the components on. I only add these where I'd want to draw boxes around these parts. (ii) objects like people, cars, items of furniture = high priority (iii) 'solid' urban ground types road,pavement (v) buildings (these are 'urban landscape' elements, or sometimes more like objects dotted in natural landscapes) (iv) natural features (river, sea,mountain,..) (v) trees,forest,sky - almost always extremely difficult to do, so always make these lowest priority .. so users are unlikely to be asked to label them.. but they can still choose to do them via search mode. the skyline is also hard because it's usually broken up by trees.

The next step would be to have a different priority list based on an overall judgement of the image (interior vs exterior, rural vs urban, workplace vs home). Small vs large objects.. would vary depending on the scene. if you were in a street scene, plates on a table infront of a cafe would be fiddly, whereas if it was a closeup of a table, those would be higher priority.. for the moment I would suggest just lumping large and small objects together as priority (ii)

not sure where to put 'bush', 'grass', 'soil' - possibly at (v) if in doubt, or between them (just make sure tree is the lowest..)

[bbernhard]

Interesting idea!

Not sure if I understood it correctly, so please correct me, if I am wrong.

• each label has a pre-defined priority (defined in the labels.json?)
• if a user selects an image for annotation, the system picks the annotation tasks with the easy labels first
• in order to improve the "pick easy tasks" algorithm, we also look at the scene, object size...

I am wondering whether a neural net can help us with the prioritization here? Otherwise, I imagine it will be pretty hard to give a good priority estimation, as there are a lot of things to consider (object size, scene, image quality, image size, ...). Personally, e.q I am totally fine with doing pixel based annotations of complex objects, as soon as the object is large enough and the image quality is good.

Not sure if that actually works, but what we could try is: We could take a pre-trained Mask-RCNN model that was trained on a lot of different labels and let our images run through Mask-RCNN. Mask RCNN segments our image and (hopefully) finds a few shapes. The shapes are probably not pixel perfect, but hopefully not that far off the real thing. If the found shape size is >= half of the image size, we mark it as easy to annotate.

[dobkeratops]

• each label has a pre-defined priority (defined in the labels.json?)

yes. I suppose you could just default all to "medium" and some are manually set to "high", "low" .. or you could set the priority as a signed int perhaps 0=default (medium), increasing+ = high, increasing- = low

• if a user selects an image for annotation, the system picks the annotation tasks with the easy labels first

my idea was to pick the images (with remaining tasks) at random, but then choose the label within each image based on the priority; the hope is this would still give a range of tasks, just tending to bias toward the easy ones - and there would still be exceptions to the rule (if there are fewer labels in an image, overall they might be easier examples?)

• in order to improve the "pick easy tasks" algorithm, we also look at the scene, object size...

that could work for components perhaps? - e.g once you've got a large 'dog' annotated then it might be safe to automatically add it's peices as new tasks?