suggestion - possible procedure for growing the label list

[dobkeratops]

ok so at the moment you have the trending labels , but I still think this isn't quite ideal; I think we need to reason about it, eg decide through discussion threads. It's like a palettization algorithm: you can't just pick "popular colours", you need representative examples to cover the whole contrast.

eg find troublesome example images - and make sure we have the labels to cover it in a sensible way. Examples could be given in a paint program to illustrate

The game would be:-

• look through the images until you find a gap that you can't label, then add the label for it. Continue until fatigued.

a second game would be: when you see two areas or images with the same label, can you add a label to distinguish them (split the graph nodes). Examples of this would be various car body types, types of ship, etc.

Using the unifiedview is much closer to the workflow I was after all along: Completeness to me is about filling the entire area,, but then you can always divide it up further (components) .. so nothing is ever really complete

this also relates to why I like the idea of a scene label as a starting point - the best single word for the whole image, then you go in and divide it up.. refining the description

adding labels to images will be easier once you ccan do it in the unified view.. it's so much easier when you can see how the image fits together and divides up.. (then you can see where the holes are)

Sometimes there's going to be combined words that would suit an area better, even if the components are labelled.. it all depends on the specific distance and prominence. An example would be "market stall" .. it's got person, strut, table,roof/canopy,fruit,crates,vegetables, but in a row of market stalls you'd just want to label that word in the distances, and only do the components closeup

[bbernhard]

adding labels to images will be easier once you ccan do it in the unified view.. it's so much easier when you can see how the image fits together and divides up.. (then you can see where the holes are)

totally agreed, that will be a huge step forward!

Maybe we should consider making the "unified mode" a completely separate thing then (i.e not a child of the "Annotation" dropdown). As soon as we have the possibility to add labels it will be become more than just a annotation drawing tool...I think that could justify a more prominent placing inside the UI.

Adding an option to add labels there as well should be pretty straight forward - at least for the labels that are already productive. But how do we deal with labels that are not yet productive? Should we allow users to add those labels, but prevent any annotations on those labels? Or should we allow annotations too?

At the moment, it's not possible to do any annotations on labels that are not yet productive. The main reason for that was, that this gives us the flexibility to correct labels without thinking too much about whether the renaming would render existing annotations useless/incorrect. Imagine the situation where someone adds a label that wouldn't fit our naming guidelines (currently there do not exist any guidelines or best practices, but let's assume for now that something like that exists). If there are no annotations for that label, we can rename any label suggestion, so that it fits our naming guidelines.

With existing annotations however, it could get more difficult, because we could theoretically (I am still thinking about a concrete example) run into situations where renaming the label would result in incorrect annotations + properties.

So I guess in the end the question is what's the most productive way of dealing with the problem? Should we choose the more conservative (and probably slower) approach and allow annotations only after the label was made productive? Or should we allow the user to annotate a non-productive label and accept the fact that if we change the labels meaning completely, that we need to remove the existing annotations + properties?

[bbernhard]

and another idea/brainstorming session regarding scene labels:

you've mentioned here (#232) some scene label examples. Up to now, I've always implicitly assumed, that scene labels are like normal labels - they exist in the dataset, but are not annotatable. They are some sort of "special labels". But is there a need that scene labels really exist?

Or are scene labels more like shortcuts? e.q: If I would enter the scene label "park" in the unified mode, it wouldn't add "park" directly, but instead replace it with "tree", "sky", "path",..etc. The user can then fine tune the auto-picked labels and remove/add ones on-demand.

The idea would be to add a bunch of pre-defined synonyms, but also give users the option to add or override existing synonyms. Imagine an option in your user profile, where you can define/upload your own synonyms. (e.q: park == tree+grass+sky+path).

Why not store scene labels too? I mean we could that too...there is no strong argument against that from my side. It's just...I think the scene labels could end up being rather vague...so sometimes it could get hard for users to validate scene labels (is that really a park , or is it a garden, or is it a place in the woods?). By sticking with the base labels..we could maybe eliminate a little bit of uncertainty. But yeah...that's just an idea.

Does that make sense?

[dobkeratops]

Up to now, I've always implicitly assumed, that scene labels are like normal labels - they exist in the dataset, but are not annotatable

right.. "unannotables" which you already have are kind of implied to be 'scene labels' already? but what you might want to do is flag any label as the 'primary label'. (see below..)

I mean we could that too...there is no strong argument against that from my side. It's just...I think the scene labels could end up being rather vague

They will sometimes be vague - the point would be an approximate image wide judgement. "street", "landscape" are both pretty vague.. but it would be a better starting point than nothing.

it could get hard for users to validate scene labels (is that really a park , or is it a garden, or is it a place in the woods?)

.. if you could blend them, ("park/garden", "park/forest") that might help.

If I would enter the scene label "park" in the unified mode, it wouldn't add "park" directly, but instead replace it with "tree", "sky", "path",

I would prefer it if the label persists, so it's searchable. I imagine scene labels will be associated with a certain probability/coverage of individual labels: by keeping them around, you'll be able to compute that.

Back to the vagueness - if you can't quite decide, just fall back to a vaguer label

a scene label being a shortcut to a label palette (which you could then flick through with the hotkeys) would be pretty interesting. but you'd have to be careful between "label palette" and "confirmed labels this image has". The scnee label would give you a speculative list, not all of which may be there.

but what you might want to do is flag any label as the 'primary label'. (see below..)

.. so if the scene label is "street", the suggested label list would be 'building, sky,pavement,road,car,person,tree,...' Howevre if it was a focussed photo of a car (in a street), if you set the scene label to 'car', then the suggested label list would be car components. ('wheel/car,hedlight/car,windscreen/car,...) . This would sae you a lot of label-entry work, and give a whole-image training signal

[bbernhard]

I would prefer it if the label persists, so it's searchable.

ok, that's a very strong argument for persisting the labels :)

.. so if the scene label is "street", the suggested label list would be 'building, sky,pavement,road,car,person,tree,...' Howevre if it was a focussed photo of a car (in a street), if you set the scene label to 'car', then the suggested label list would be car components. ('wheel/car,hedlight/car,windscreen/car,...) . This would sae you a lot of label-entry work, and give a whole-image training signal

oh..yeah..right. I always assumed that scene -and normal labels are mutually exclusive. But you are right, there are cases where a scene label should be annotatable too. So that means any label can be a potential scene label, right? Can an image have multiple scene labels?

Do we need to distinguish between scene labels and normal labels in the UI? If so, how should we do that? Should that be represented by a property? (primary, scene label,..?) Or should we maybe add a little checkbox next to each label in the unified mode labels list?

a scene label being a shortcut to a label palette (which you could then flick through with the hotkeys) would be pretty interesting. but you'd have to be careful between "label palette" and "confirmed labels this image has". The scnee label would give you a speculative list, not all of which may be there.

yeah, right. That's what I had in mind :)

[dobkeratops]

pure scene labels would be a good starting point (they could appear in the list, many label suggestions like 'landscape, valley,kitchen,harbour.coastline,cityscape etc'are really pure scene labels as for turning a normal label into 'a scene label' (or 'primary label'?) ...i'm not sure what the best way is. We don't need scene labels for everything, it would just be useful where it is possible. the obvious cases are street (and variants.. highstreet,residential street,town centre, city centre,..), and specific rooms (kitchen, living room,bedroom,hotel room, dining room, ...), seaside, riverside, marina, docks, harbour,factory,office,classroom,school,playground,...

[bbernhard]

pure scene labels would be a good starting point

like that!

I'll go through our github discussion threads and try to find the 'pure ones'.

After those are added, we can discuss the workflow of turning a normal label into a scene label.

(Thinking a bit more about it, I think a property is probably not the right thing, as it is attached to a specific bounding box. That might not be what we want for the scene labels).