jonfroehlich
commented
5 years ago @daotyl000 and I discussed this in person and I asked him to make these visualizations. This is based on his initial qualitative analysis of the worst performing three crowd workers: he had found that most of the mistakes he analyzed were from placing labels where there were no sidewalks (which is quite common in the northern parts of Seattle due to historical reasons).
If this is the case, it's not particularly interesting in the scientific/academic sense and the mistakes make sense: we routed these minimally trained users along many streets without sidewalks. They thought, incorrectly, that they should label the sides of the street as if there was a walking path (there sometimes is but it's not really a sidewalk, it's often just a little gravel segment or nothing at all). Instead, they should have just applied 'No Sidewalk' labels.
We will obviously want to expand our analysis (and also include Newberg) with more user samples.
The three users with the lowest accuracies all have worked in regions with there are not a lot of sidewalks, as can been seen with the large amount of purple no sidewalk labels. Their false labels being placed could be due to them label things where there are no sidewalks instead of other factors.