jonfroehlich
commented
7 years ago I really like this too because it helps remind Turkers what to look for! In addition, it clear grows our knowledge base by expanding the amount of useful information per label--which could be used to train automatic recognition systems or incorporated into user-facing tools (e.g., how many problems are due to poorly placed fire hydrants might be a query a government may want to run).
Incidentally, almost two years ago, I suggested auto-completing descriptions , which I think we could also do but the tags have the advantage of: (i) requiring recognition rather than recall; (ii) helping remind workers what to label; (iii) can be clicked rather than typed.
manaswisaha
misaugstad
aileenzeng
When the user clicks a tag, it becomes grey:
It could also maybe autofill in the description box?
When expanded, it could do list out the rest of the labels (probably in alphabetical order?):
When a user clicks a tag, it becomes bold:
There are some label descriptions that are likely very common, or that we would love to have information about. Examples include:
It would be great if we could get these annotations a lot more frequently. One interview participant felt that typing this in for every label was getting tedious, so they suggested a drop-down menu with common descriptions, while still having the option of the free-form text field.
I really love the idea. Not sure that a drop-down menu is the best, per se, considering it is an extra click. I figure we could just display the different default descriptions in a row, and you can click on one to add it to the text field. This actually doesn't sound like too much work, either. Here is a quick mockup of what I am imagining.
In fact, this feels a lot like a 'tagging' system. Have we thought about using pre-defined tags? Of course this would not remove the need for a free-form text box, but perhaps it would make things easier for users, be a familiar system to them, and it would make querying for example of certain kinds of problems much much easier.
I think this could even be used to help with estimating the reliability of a user's labels and prompting in-situ training for them! What we have been thinking about is for users that we suspect are not very accurate, we could give them in-situ training on things we suspect they may not be proficient in labeling, we could lower our estimated probability that their labels are accurate in some access score model, or we could have another person audit the same streets as them.
A simple way that we have discussed to identify low-quality labelers, is to compare the label type distribution of their labels to the distribution of all labels (or labels within that neighborhood). This could then be extended if we have a tagging system. For example, maybe in some neighborhood, multiple users have found and tagged a large number of surface problems as having grass b/w the sidewalk blocks. But then there is this new user who has labeled 4 miles, but none of his surface problems have the "grass" tag. Maybe we could then do the in-situ training where we show them examples of surface problems that they should be labeling that have grass b/w sidewalk blocks.