Let's do a deep-dive exploration of our 40% and ~60% users

[jonfroehlich]

In our small exploratory dataset, we have two users who are doing particularly poorly--a ~40% user and a ~60% user. We'd like @daotyl000 to perform a deep-dive exploration of these users and investigate why they performed so poorly. Look at some example labels, look more deeply at their performance, etc.

[daotyl000]

After initially looking at approximately 25 labels of the three users with the lowest accuracy, all except for one label were obstructions or surface problem labels in areas where there is not a sidewalk (ex. Mailboxes, trash cans on grass patches or in driveways, gravel or grass areas not sidewalks, vehicles in parking spaces)

[jonfroehlich]

Can you take screenshots in a Google Slide deck for us to analyze together. Also, paste over metadata in the notes field so we know the source, etc.

Sent from my iPhone

On Jul 8, 2019, at 12:09 PM, daotyl000 notifications@github.com wrote:

After initially looking at approximately 25 labels of the three users with the lowest accuracy, all except for one label were obstructions or surface problem labels in areas where there is not a sidewalk (ex. Mailboxes, trash cans on grass patches or in driveways, gravel or grass areas not sidewalks, vehicles in parking spaces)

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[daotyl000]

Here is the link to the Google Slides I have made. There is about 10 slides for each of the 3 users with the lowest accuracy

https://docs.google.com/presentation/d/1S5inXslcBxQ6Efs7AZUzH6UHHmNoTkLgOuTutqF3GM8/edit?usp=sharing

[jonfroehlich]

Thanks. Can you make me an editor on the doc by inviting me (my cs address).

Also, I think we need ~5-10 examples of each label type for each user. On each slide, please provide an explanation/tag for why they were wrong (you did this on some slides but not all).

On Mon, Jul 8, 2019 at 4:32 PM daotyl000 notifications@github.com wrote:

Here is the link to the Google Slides I have made. There is about 10 slides for each of the 3 users with the lowest accuracy

https://docs.google.com/presentation/d/1S5inXslcBxQ6Efs7AZUzH6UHHmNoTkLgOuTutqF3GM8/edit?usp=sharing http://url

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ProjectSidewalk/sidewalk-quality-analysis/issues/19?email_source=notifications&email_token=AAML55PDJ5YARTAVLCVMVOLP6PE7ZA5CNFSM4H47TTB2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZOUQQI#issuecomment-509429825, or mute the thread https://github.com/notifications/unsubscribe-auth/AAML55LB2HIFIPOWTOMTGOTP6PE7ZANCNFSM4H47TTBQ .

-- Jon Froehlich Associate Professor Paul G. Allen School of Computer Science & Engineering University of Washington http://makeabilitylab.io @jonfroehlich https://twitter.com/jonfroehlich - Twitter Help make sidewalks more accessible: http://projectsidewalk.io

[daotyl000]

I have added you as an editor. For each label type, do you want only labels verified as incorrect?

[jonfroehlich]

The goal is to figure out why these users are struggling to label--so, yes, I want to focus our analysis on the mistakes they are making.

On Tue, Jul 9, 2019 at 10:58 AM daotyl000 notifications@github.com wrote:

I have added you as an editor. For each label type, do you want only labels verified as incorrect?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ProjectSidewalk/sidewalk-quality-analysis/issues/19?email_source=notifications&email_token=AAML55LIJKC5JFHZFA3XSOTP6TGULA5CNFSM4H47TTB2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZRBFVQ#issuecomment-509743830, or mute the thread https://github.com/notifications/unsubscribe-auth/AAML55NSZS37IGEKHGREPRDP6TGULANCNFSM4H47TTBQ .

-- Jon Froehlich Associate Professor Paul G. Allen School of Computer Science & Engineering University of Washington http://makeabilitylab.io @jonfroehlich https://twitter.com/jonfroehlich - Twitter Help make sidewalks more accessible: http://projectsidewalk.io

[daotyl000]

Of the 10 current bad users, 5 of their mistakes are largely due to errors in neighborhoods without sidewalks. 5 others are just bad users in general

Overall Bad: 1353d168-ab49-4474-ae8a-213eb2dafab5, 35872a6c-d171-40d9-8e66-9242b835ea71, 6809bd6e-605f-4861-bc49-32e52c88c675, 939b6faa-0b57-4160-bcc2-d11fd2b69d9f F5314ef9-3877-438c-ba65-ee2a2bbbf7f5

Errors mainly in region w/ no sidewalks: 54c77d0f-fc8f-4497-84d3-5e336047b17e, 86d26e9d-010f-4802-88ba-680ae0a8e20d, 8a471c0f-fa81-4c57-9b65-bd04a92c6a5e, Bca24c1a-a6b1-4625-ab8e-9ff8693022d7 ec15a589-dd14-4513-a43e-8c06e55f4c71

[jonfroehlich]

OK! I care more about the "overall bad" than the other category (which feels somewhat Seattle-specific though certainly there are lessons here that generalize, I think).

Can you do a qualitative analysis of "Overall Bad" and post your findings?

[daotyl000]

Common Errors: Obstacles not on the sidewalk: 5 / 5 Surface problem not on the sidewalk: 2 / 5 Driveways as curb ramps: 2 / 5 Missing curb ramp labels on no pedestrian path: 2 / 5

It seems like even though these are the users who were routed through neighborhoods with a sufficient amount of sidewalks, they were still tagging obstacles & surface problems that weren't on the sidewalk. This being both things on the road right next to the sidewalk, and the 2 feet of sidewalk from the road that is used for sign, fire hydrants, etc.

1353d168-ab49-4474-ae8a-213eb2dafab5: Main: labeled surface problems on roads/side part of sidewalks Lesser: Obstacles on side part of sidewalks, missing curb ramp when wasn’t needed 35872a6c-d171-40d9-8e66-9242b835ea71: Main: Labeled surface problem of narrow but wasn’t, cars parked/ things on side of road/sidewalk Lesser: Driveways as curb ramps 6809bd6e-605f-4861-bc49-32e52c88c675: Main: Missing curb ramp as places where they aren’t needed, cars parked/ things on side of road/sidewalk 939b6faa-0b57-4160-bcc2-d11fd2b69d9: Main: Obstacle on side part of sidewalk, F5314ef9-3877-438c-ba65-ee2a2bbbf7f: Main: Driveways as curb ramps, plants & grass next to sidewalk as surface problems, obstacles . next to the sidewalk but not on it