With the new Validate UI that lets users edit tags and severity, there is some concern about making it available to all users (I'm making it admin-only for now while I take my trip to Europe). The issue is that with this tool you actually get to edit the tags and severity (wiki- or osm-style) rather than just voting on whether or not you agree with the label.
If this is available to everyone, there are some situations that we likely wouldn't be happy with. For example, if an Admin adds a label on the Explore page and carefully sets the severity and tags, we don't necessarily want a totally untrained user to jump in and mess with the tags that we set. Same thing goes if an Admin takes the time to edit the tags/severity using this new UI, we don't want those changes undone so easily.
With sites like OSM or Wikipedia, there are way more users making edits and they have some tools for finding/detecting problematic data that we do not. And @yeisenberg has mentioned that this could have implications for gov't trust in the data that we provide.
Potential solution(s)
Some options:
We could simply let everyone access this tool, and treat our data like OSM or Wikipedia treats theirs. The idea being that over time, the data will trend towards being more accurate.
We could let anyone use the tool, but we could use our estimates of the expertise of users (i.e. the quality of their data) to inform what labels we serve to which users to validate. For example, maybe we wouldn't show labels from Admins to new users that are contributing for the first time. Maybe when choosing which labels to validate, we are more likely to show labels placed by users with 50% accuracy over users with 90% accuracy. We could expand our statistics on the quality of users' data and use those: maybe we calculate the agreement between their validations and overlapping validations from admins!
I feel good about this idea as practically keeping the quality of our data high over time. The downside is the lack of hard rules, meaning a lack of explainability when people ask about the quality of our data. It would be hard to say something more than "we have an algorithm", and that's not all that reassuring.
We could let users "unlock" this tool. This could be through a minimum amount of data being put into the system (labels and/or validations), a minimum level of accuracy, and/or any other metrics we want to add (could be some of the same ideas outlined in the previous section, like high agreement in validations overlapping with Admins). I think that this is the idea that we've talked about the most, so I'm sort of expecting us to go down this route (as much as I like route number 2 :wink:). Some stuff that would need to be figured out:
What are our requirements to unlock the new tool
Do we tell users that they can unlock it and how? If so, how do we want to advertise that to users throughout the site? And how do we show them their progress towards unlocking it? Do we tie this in with our system of badges, and add new ones?
What do we do if users meet the thresholds to unlock the tool, but later on their performance dips below the threshold to unlock it? What if they dip way below that threshold?
Brief description of problem/feature
With the new Validate UI that lets users edit tags and severity, there is some concern about making it available to all users (I'm making it admin-only for now while I take my trip to Europe). The issue is that with this tool you actually get to edit the tags and severity (wiki- or osm-style) rather than just voting on whether or not you agree with the label.
If this is available to everyone, there are some situations that we likely wouldn't be happy with. For example, if an Admin adds a label on the Explore page and carefully sets the severity and tags, we don't necessarily want a totally untrained user to jump in and mess with the tags that we set. Same thing goes if an Admin takes the time to edit the tags/severity using this new UI, we don't want those changes undone so easily.
With sites like OSM or Wikipedia, there are way more users making edits and they have some tools for finding/detecting problematic data that we do not. And @yeisenberg has mentioned that this could have implications for gov't trust in the data that we provide.
Potential solution(s)
Some options:
We could let anyone use the tool, but we could use our estimates of the expertise of users (i.e. the quality of their data) to inform what labels we serve to which users to validate. For example, maybe we wouldn't show labels from Admins to new users that are contributing for the first time. Maybe when choosing which labels to validate, we are more likely to show labels placed by users with 50% accuracy over users with 90% accuracy. We could expand our statistics on the quality of users' data and use those: maybe we calculate the agreement between their validations and overlapping validations from admins!
I feel good about this idea as practically keeping the quality of our data high over time. The downside is the lack of hard rules, meaning a lack of explainability when people ask about the quality of our data. It would be hard to say something more than "we have an algorithm", and that's not all that reassuring.