Use ML to pre-label unlabeled images?

[bbernhard]

Based on #12:

I think the whole scene concept is a little bit too much effort to implement in a next step (although I like the general idea), so it probably is a good idea to break the idea down in smaller work packages and realize them step by step.

• make unlabeled uploads possible I am thinking about something similar to what @dobkeratops proposed: The text label becomes a dropdown list with a random word selected by default. If the user doesn't want to upload a picture which represents that label he can choose other ones from the dropdown or select "unlabeled" (maybe it's good to make that the first item in the list? or even make it more prominent by using a separate button to toggle between wordlist and unlabeled?) .

• use pre-trained ML model to guess the/a label Not sure if that even has a chance to work, but maybe we can use a pre-trained model (which was trained on some general datasets) to guess the/a label. As most of the models were only trained on a few different objects, I am not really sure if we get decent results (or if we are better off by just randomly selecting a word from our wordslist ;-)).

• present labeled image for validation here the usual workflow continues. The user gets the (machine) labeled image presented and has to verify it. If the machine guessed wrong, we could keep the result in the database as "probably-not-a-xxx" and try to machine-label it again after some time(at this point we maybe have some more indications to guess right this time)

[dobkeratops]

'random default in a drop-down, with unlabelled the first list item' sounds perfect

maybe you just hard-code detection of 'unlabelled' or ' scene', and eliminate those images from the parts of the application that assume single-object labelling? (.. until a more general mechanism to control this has been devised).. i.e. so you are never prompted to "annotate all the Unlabelled in this image..", but 'dogs' 'cars' etc work just as they do now

[bbernhard]

'random default in a drop-down, with unlabelled the first list item' sounds perfect perfect :)

maybe you just hard-code detection of 'unlabelled' or ' scene', and eliminate those images from the parts of the application that assume single-object labelling? (.. until a more general mechanism to control this has been devised).. i.e. so you are never prompted to "annotate all the Unlabelled in this image..", but 'dogs' 'cars' etc work just as they do now

That would definitely be possible.

However, if possible, I would really like to integrate the unlabeled images in the current workflow by pre-labeling them somehow. I think if we enable the possibility to upload unlabeled images we could really get a lot of image donations in a short time. Because at that point it would be possible to use some public image API (e.q flickr API) to get images which where released under the CC0 license and bulk import them to ImageMonkey. In order to avoid that we have a lot of unlabeled images in the database which can't be operated on until we have adapted the current workflow, it would be really great to give them automatically a label.

I think that's generally something we should strive for: automatical labeling. Because if we have some correctly labeled and annotated data, we should probably use that data and information to pre-classify new images.

That's for sure not possible for every image, but I am thinking about the following type of images:

Both of the above images have some pretty prominent features, so it should be possible at some point to delegate the labeling process to a machine and let the user just acknowledge the label.

I also played a little bit with the SIFT algorithm today to see if it can be used to find similar pictures. I only tested it on a few (well selected) image pairs, but the results were surprisingly good. So maybe that could be another approach?

I think the most difficult part with SIFT is: How are we gonna store the SIFT features in the database to access them and match them quickly?

some useful resources:

• https://www.researchgate.net/publication/275219135_Using_binarization_and_hashing_for_efficient_SIFT_matching
• http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.61.9520&rep=rep1&type=pdf
• https://github.com/hlzz/libvot
• http://webdiis.unizar.es/~dorian/index.php?p=31

[dobkeratops]

SIFT is certainly interesting, see also 'bag of visual words' .

Thanks, will also check that out :)

I wondered if you could crowdsource the identification of useful features (e.g. from the 'guess what this is.. ' game. random pieces that get more correct guesses of the whole object would be good features?)

that's an awesome idea! :D

Not sure if that works though, but it's for sure something to try. Is there already a ticket for that? I think that one definitely deserves it's own ticket. :)

[dobkeratops]

hi, another question and idea on this:- q1: how well does this SIFT-feature idea work on scenes (e.g. could it seperate outdoor vs indoor by virtue of trees, sky)

And suggestion-question: would it be worth combining this with a Wikimedia-Commons Scraper, to grab a tonne of images and 'scene-classify' them (maybe as simple as 'indoor vs outdoor')

you'd then have a basic label that could be refined a lot further with the question-'game'

[bbernhard]

Awesome ideas, thanks a lot!

Really like the Wikimedia-Commons-Scraper idea. While we are scraping the images we could also investigate whether it's possible to "guess" a label from the filename/URL (I think there also exists a ticket from you :))

[dobkeratops]

yeah the wikimedia-commons images are categorised; those categories are also arranged into a hierarchy. As such you've got a label to train with already, guaranteed - just scrape by category. then those images are available for further object labelling.

It's extremely unfortunate that wikimedia commons guidelines don't allow object labelling already.. they've got a feature for adding annotations, but they only allow sparing use, and their admins actively police to remove labels if you add them (!!!)