search

ImageMonkey / imagemonkey-core

ImageMonkey is an attempt to create a free, public open source image dataset.
https://imagemonkey.io
47 stars 10 forks source link

possible need for 'annotate with new label' - hard to spot validations #164

Open dobkeratops opened 6 years ago

dobkeratops commented 6 years ago

basically in some images in 'add labels' on seeing it again you add a new label you didn't spot originally

What then worried me is people would miss cases like this in validation, e.g. "ok I dont see this label.."

(one example I just did was 'escalator'.. there's just a peice of handrail visible, but then through contextual implication you can see it's the beginning of an escalator going up through an enclosed space)

As the labels get more detailed (long tail..) and you dig deeper in images.. I think the opposite workflow might be needed to complement what the site already does, i.e: select the label (including new ones) as you annotate .

I guess this will follow more naturally after the adaptation for 'optional completeness' , i.e. if you start changing label mid-annotation that could come as a hint that those annotations might not be complete

A simpler fix might be a "Dont know" option in the validation page - currently it forces a binary choice. (this might be needed for other reasons e.g. what if a user isn't sure what a label means)

bbernhard commented 6 years ago

interesting idea!

As you mentioned the annotation mode: There is another feature that I would like to add and that's the possibility to work on validation and annotation tasks where the label isn't already productive. I realized lately, that I am becoming more and more a bottleneck (unfortunately, the day has only 24hrs and besides a full time job there is only so much time to work on new stuff, unlock labels & images..etc), so I would like to take me out of the equation as much as possible.

At the moment the labels list is growing much faster than I can make them productive - don't get me wrong, I am definitely not complaining here, that's actually totally awesome! But in order to avoid that annotation tasks pile up, I think some structural changes are needed.

The things I would like to introduce are:

I guess we will come up with even more possible moderation tasks, if we think a bit more about it :). The idea is to incrementally add more and more moderation functionality, while still keeping the whole group consensus concept.

The first moderation feature that will be available is the label "removing". If moderators "remove" a label (there will be a dedicated context menu for that, to avoid that it gets accidentally clicked) their downvote counts 5 times as much as the vote from normal users. The feature is already almost done...so I think I should be able to push it to production in the next days...

If you are okay with it, I would like to make you the first moderator. You contributed so much to the dataset...I honestly can't think of a better person for that job. I would be really happy, if you want to take on the role.

dobkeratops commented 6 years ago

At the moment the labels list is growing much faster than I can make them productive

difficult question overall - because it still needs curation. (and there's plenty of typos) I wonder if you could save some time by copying the COCOS or CIFAR-100 challenge labels. I think COCOS labels are arranged in a tree aswell which even gives a starting point for a label-graph.

I wholeheartedly agree regarding a need for moderation and so on. there's the risk of spam, abuse, copyright hazards (I note the 'link tax' was rejected today.. but who knows what the future will bring)

If you are okay with it, I would like to make you the first moderator.

happy to help, as I say I am completely sold on the idea of a site like this , and want the end result.

possibility to work on validation/annotation tasks where the label isn't already productive yet:

Something else that might acheive this is label-refinement, if you had a complete set of 'basis' labels (i.e. you could draw around something knowing it can be described more accurately later). it would be a case of weighing up the risks of 'free labelling' vs the extra UI work.

bbernhard commented 6 years ago

difficult question overall - because it still needs curation. (and there's plenty of typos) I wonder if you could save some time by copying the COCOS or CIFAR-100 challenge labels. I think COCOS labels are arranged in a tree aswell which even gives a starting point for a label-graph.

My main motivation for going with the "trending labels concept" was the idea of letting the labels list grow into it's natural direction. So it's really up to the users in which direction we are going - if the majority of users upload indoor photos, we will probably end up with a lot of indoor labels; if the majority of image donations are urban scenes, we probably have more urban scene related labels. With a static labels list, I am a bit worried, that we end up with a lot of labels that nobody will (ever) use - I guess that doesn't need to be a problem per se, but I think it could make the labels list look more crowded. (one of the first labels that I added was TV - I thought that nearly everybody has a TV at home, so this will be good label...actually we only have 4 TV donations as of now...I couldn't have been more wrong :D)

I think the biggest problem at the moment isn't the unlocking of new labels per se (that's actually not that much work), it's the fact that the validation/annotation tasks are locked as long as the label is locked. So as long as our label list is still growing fast, I guess we always run into the situation that work piles up, until the label gets unlocked.

My hope was, that this will be less of a problem, if there is already the possibility to start annotating/validating if the label is still locked. Of course, there is always the possibility of typos...but as the unlocking of new labels is a manual step, I guess it should be possible to correct the labels name while still keeping the annotations.

e.q: a user labels a significant amount of images with Person and completes the corresponding annotation tasks. As the label Person was used at least 20 times (that's the current threshold for becoming a trending label) a github ticket in the trending repo will appear. There already exists a lowercase version of the label person, but for some reason the user wasn't aware of that and created the label Person instead (I guess that's a even more realistic scenario, when images are labeled with the Python script). When making the label Person productive, we can now rename the label Person to the existing label person and transfer the annotation tasks from Person to person.

As long as it's possible to grasp what the user meant with the label, I think that could work out well.

But that's just an idea...not sure if that works out as well as I imagine it in my head. ;)

happy to help, as I say I am completely sold on the idea of a site like this , and want the end result.

awesome, I'll let you know as soon as the feature is pushed to production :)

dobkeratops commented 6 years ago

With a static labels list, I am a bit worried, that we end up with a lot of labels that nobody will (ever) use - I guess that doesn't need to be a problem per se, but I think it could make the labels list look more crowded.

perhaps you could you keep a subset of labels that appear in drop-down label list, with the full list only accesible through keyboard entry... but you've also got the 'most popular labels' in add labels, which works well IMO.

I suppose the donation page is one important place where you need a user to the label list: a subset useable here could always be complemented with detail in 'add-labels' - eg using uploading as road or building usually works for a street scene (whilst those images may go on to have many more items).

The other way to think about it is that the long label list ("unused") is an invitation for growth. I always remember seeing people talk about this in arguments over red-links in wikipedia. Some thought the red-link was problem ("it makes it look unfinished") whilst other people point out the red-link is like a request for an article ("red links are how wikipedia grows"). Perhaps something like this could be made deliberate, e.g. make it clear which labels have thorough examples ("use these for training.."), whilst others are more like requests ("help wanted..")

I would think the goal is the broadest possible variety of images - I gather that it's possible to train a net for broad scenes, then using re-using it's low-level features as a faster starting point for any domain-specific training (or addition of new classes).. I heard that facebook keep such 'broad nets' lying around. I've heard that the low-level features are very re-usable.

(one of the first labels that I added was TV - I thought that nearly everybody has a TV at home, so this will be good label...actually we only have 4 TV donations as of now...I couldn't have been more wrong :D)

I still think it's a good label idea ..

I think there's a fair number of these in public places (digital signage), and in the background in home scenes - I'd guess I've seen more than 4 TV's in your stock photos, even if they're not the main subject. With a graph there may be a broader label 'flatscreen display', which would cover 'laptop screen','computer monitor','flatscreen TV' (for retro images you might need CRT's aswell).. perhaps that would work better as an invitation for popular uploads.

(tangentially this is something that contributed to me thinking LabelMe felt abandoned.. it's office scenes are visually dated by the predominance of CRT's- interesting how fast the world has changed :) )

I'm also guessing TV/monitor is a good contrasting example vs windows, paintings, billboards,mirros (even smartphone, scale..): they're both rectangular areas through which (or on which) something else is visible, but very different in regarding context, and what tends to be visible through them.