ProjectSidewalk / sidewalk-quality-analysis

An analysis of Project Sidewalk user quality based on interaction logs
5 stars 3 forks source link

Analyze amount of tagging vs user accuracy #40

Closed daotyl000 closed 4 years ago

daotyl000 commented 5 years ago

Do the tags users add to their labels provide an indicator for the user's accuracy? This can be logged using the ContextMenu_TagAdded action.

We could look at the total number of tags or tags per label. A feature that could be potentially interesting is the number of tags per label type since different label types have different amounts of available tags and people might be more inclined to tag certain label types.

Is there a way determining if a tag is correct without manually looking at the label?

daotyl000 commented 5 years ago

Looking at overall tags, there does not seem to be a correlation between tags and accuracy when scaled to missions or panos. There are 4 good users who have placed a lot of tags but it appears to be due to them having done more missions. When overall tags are broken down to per mission or per pano, the R value would be close to 0 except for one bad user who used many tags.

Screen Shot 2019-08-02 at 1 08 33 PM

daotyl000 commented 5 years ago

These are the accuracy of each users specific to the given label type. There isn't a strong correlation between tags vs the accuracy for that given label type. Curb ramp tags are generally stable across all users so there isn't a correlation. Missing Curb ramp tags have a positive correlation mainly due to a few outliers who tagged a large amount of missing curb ramps. Obstacle tags surprisingly has a negative correlation mainly due to three users placing many tags but doing poorly in that category. Surface problem labels tags has mostly good users when placing atleast 125 tags except for a few.

This does not seem like a viable feature with the current correlations.

Screen Shot 2019-08-06 at 2 33 15 PM Screen Shot 2019-08-06 at 2 33 19 PM Screen Shot 2019-08-06 at 2 33 23 PM Screen Shot 2019-08-06 at 2 33 28 PM