search

ImageMonkey / imagemonkey-core

ImageMonkey is an attempt to create a free, public open source image dataset.
https://imagemonkey.io
47 stars 10 forks source link

'negative labels' - Foo / not Foo #57

Open dobkeratops opened 6 years ago

dobkeratops commented 6 years ago

if given a specific curated label list, could their comprehensiveness be extended by allowing one to give a judgement of what something is not,

e.g. imagine if you started with just 3 labels, 'animal, mineral,vegtable' .. that would cover any image, but not exactly tell you a huge amount. then you might divide each into 'the most common', and everything else.. 'animal' -> "human", "non-human". "mineral"->"man-made object", "not man-made"; "vegtable"->"tree","plants other than trees". "man made"-> "vehicle", the first 3 labels here really become divisions of "living"/"not living", then "animal"/"not animal" or "plant"/"not plant".

I might try to retrofit this idea to my 'label-graph', e.g. could generate a 'not' node for any label and still have 'isa's pointing at that. You'd have to work up the graph to avoid having to link "tree" is "not human","not car", "not dog" etc.. the system should be able to figure that out once you've said those other things are "animal" (="organic"&"not plant"), "mineral" (not organic) which in turn uses the graph to compute "tree" is "not car" etc. 

   total labels
    1                                object
                              ------/      \----------
                             /                        \
    2                organic                           !organic
                    /       \                        /            \
    4        plant            !plant            man_made             !man_made
            /      \          /   \            /       \                 /   \      
    8   tree      !tree   human   !human    vehicle  !vehicle      water..   !water
                +grass     ..     +quadruped +car      +building   +lake     +rock
                +bush             +bird      +truck    +furniture  +river    +sand
                +flower            ..        +bicycle  +tool       +sea      +mud
                ..                           ..        ..          ..        ..

e.g. with just ["organic","plant","tree","human","vehicle","water"] you could specify ["not organic","not vehicle"] (might be 'building,furniture,tool..', ..even though you dont yet have those labels, it's ready to search for that at the next refinement step when more are given)

Of course if you're just going to start with the 200 COCO labels in a tree that might be a more efficient way to give more info for common items, but I'm just throwing this idea out r.e. how you could maximise the utility of a limited set , e.g if you have some minimal area of screen space for a user to click on or whatever.

Actually thinking about it your "verification" mode could be thought of as giving negative label data already? (.. Might you end up with this as a common piece of data for easily mistakable categories)

(perhaps I should also look into a propper way of saying a group of labels are mutually exclusive , members of one layer)

bbernhard commented 6 years ago

Awesome idea!

I think that could also be something for the Quiz mode (still have to think how that can be integrated in a visually appealing way though). I have done quite a few extensions to the Quiz mode lately (e.q "browse by example"), as I figured out that's a great way to attract non-technical people. (my girlfriend e.q really likes to guess dog breeds).

Actually thinking about it your "verification" mode could be thought of as giving negative label data already?

haven't thought about it that way, but you are right ;)

I also thought about serving random false positives to users in order to implement some sort of "abuse detection mechanism". I thought that could be a good way to keep user's focused, as they have to think more about whether the label actually is shown in the picture, instead of blindly pressing the "yes" button.

In a first experiment, we could pick a random label and show it together with a random picture (maybe only for every n-th verification). If we are lucky, then the object is actually in the picture and we can create the label <-> picture relationship. If the object is not in the picture we could create a "not label" <-> picture relationship.

That way we could experiment a little bit with the whole !label concept, while still keeping the same UI controls.