Large number of incorrect labels in a few panos due to a user testing out bugs instead of actually auditing

[daotyl000]

The label numbers 73286 - 73667, are from Naomi testing out different things out on the website. She said she was mass labeling on the same road to see if certain bugs or issues were happening. Most of the labels weren't verified but the ones that were were all verified as false. All of them are random labels placed around the pano of different types

Screen recording of the labels: https://youtu.be/zKeA54Z8aqI

[jonfroehlich]

This is a problem as we treat researcher labels as sacrosanct.

So, we cannot and should not use Naomi's labels for any type of analyses going forward (not for ground truth, not for quality inference). If Naomi wants to continue labeling/validating in PS--and I hope she does--then she needs to make another username and use that from now on. What do you think @misaugstad?

[misaugstad]

yeah I agree, best for her to make a new account

[jonfroehlich]

Just chatting with her now about this so she's aware.

On Fri, Jul 19, 2019 at 11:11 AM Mikey Saugstad notifications@github.com wrote:

yeah I agree, best for her to make a new account

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ProjectSidewalk/sidewalk-quality-analysis/issues/28?email_source=notifications&email_token=AAML55LNV5IYKDAWTXCQY3DQAH7WNA5CNFSM4IETLER2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2MLR6Q#issuecomment-513325306, or mute the thread https://github.com/notifications/unsubscribe-auth/AAML55KN4GWJALZ3TXW346TQAH7WNANCNFSM4IETLERQ .

-- Jon Froehlich Associate Professor Paul G. Allen School of Computer Science & Engineering University of Washington http://makeabilitylab.io @jonfroehlich https://twitter.com/jonfroehlich - Twitter Help make sidewalks more accessible: http://projectsidewalk.io

[misaugstad]

Thank you!

[jonfroehlich]

@misaugstad before we close this out, how should we deal with this on the backend so that we don't mess up our own analysis for the next Project Sidewalk paper?

[misaugstad]

I recently added a user_stat table, and it has a column where we can manually mark someone as a low/high quality user. I just manually marked that first user account as low quality. Another way to ensure we don't make any mistakes would be to mark her old account as a registered user account instead of a researcher account.

[jonfroehlich]

Great. Yes, let's also mark her old account as a registered user account.

On Thu, Aug 1, 2019 at 1:36 PM Mikey Saugstad notifications@github.com wrote:

I recently added a user_stat table, and it has a column where we can manually mark someone as a low/high quality user. I just manually marked that first user account as low quality. Another way to ensure we don't make any mistakes would be to mark her old account as a registered user account instead of a researcher account.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ProjectSidewalk/sidewalk-quality-analysis/issues/28?email_source=notifications&email_token=AAML55MLZPNKAJG5SDU2TWTQCNCOLA5CNFSM4IETLER2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3L2GOQ#issuecomment-517448506, or mute the thread https://github.com/notifications/unsubscribe-auth/AAML55IB3JRCUKSUI46S6XTQCNCOLANCNFSM4IETLERQ .

-- Jon Froehlich Associate Professor Paul G. Allen School of Computer Science & Engineering University of Washington http://makeabilitylab.io @jonfroehlich https://twitter.com/jonfroehlich - Twitter Help make sidewalks more accessible: http://projectsidewalk.io

[misaugstad]

done and done!

[daotyl000]

Unless there is more to be done, I think that we have solved this issue by marking her account and switched it to registered instead of researcher.