Restarting this with a fresh, clean list of input features for user classification

[jonfroehlich]

Columns I want to start out with:

• [x] User id (please also send me a table that links user id to username but we won't check this in)
• [x] User type (anon, turk, registered, researcher)
• [x] Time taken to complete tutorial
• [ ] Country of auditor based on ip address (reverse lookup). Possibly also distance from deployment city to auditor's location as determined by reverse lookup.
• [x] Error count for tutorial
• [ ] Time per stage in tutorial (where a stage is each instruction)
• [ ] Min, max, avg, median, stdev of time per pano visited
• [ ] Min, max, avg, median, stdev of labels per pano visited (both overall and broken down by label type)
• [x] Min, max, avg, median, stdev of severity ratings both overall and per label type
• [ ] Median of severity ratings both overall and per label type
• [x] Percentage of labels with severity ratings both overall and per label type
• [x] Percentage of labels with tags both overall and per label type (@daotyl000 had begun an initial analysis of this here: https://github.com/ProjectSidewalk/sidewalk-quality-analysis/issues/40)
• [x] Agree, disagree, unsure validations received overall and per label type—that is, validation counts of the labels the given user supplied (needs to be raw count so I can normalize; so I'll also need to know how many labels of each type were validated)
• [x] How often a user marks agree, disagree, unsure when validating (both overall and broken down by label type)
• [ ] Min, max, avg, median, stdev of mouse movements per pano
• [ ] Min, max, avg, median, stdev of keyboard interactions per pano
• [x] total keyboard interactions count (stand in for above until a more efficient query is made)
• [ ] Min, max, avg, median, stdev of pan interactions per pano
• [x] total pan interactions count (stand in for above until a more efficient query is made)
• [x] Num audit missions completed
• [x] Num validation missions completed
• [x] Total audit distance (called 'meters_audited' currently)
• [x] Overall number of panos visited
• [x] Overall num of panos visited where they supplied at least one label
• [x] Mikey's lil assessment of good/bad turk quality (not exactly sure how he makes this determination and how much is automated vs. manual)

Some other things I'm thinking about but don't need right away

• [ ] Min, max, avg, median, stdev of time per 100 meters (i.e., travel speed)
• [ ] Min, max, avg, median, stdev of time per validation
• [ ] Percentage of agree, disagree, unsure overall and per label type that this user selects when performing validations
• [ ] On panos with labels, min, max, avg, median, stdev of time to place first label (i.e., search time)
• [ ] Num key presses during tutorial
• [ ] Num logins
• [ ] Num times dashboard loaded
• [ ] Num times help page visited (https://github.com/ProjectSidewalk/sidewalk-quality-analysis/issues/1#issuecomment-517884671 and https://github.com/ProjectSidewalk/sidewalk-quality-analysis/issues/42)
• [ ] Meters traveled before placing first label ever.
• [ ] Country of origin based on ip reverse lookup (yes, could be noisy due to VPN usage)
• [ ] Whether user ever wrote in text in the open text field (Description) for a label. Could also do descriptive stats for this but I imagine will be small.
• [ ] Distance from city being audited based on ip reverse lookup
• [ ] Agreement stats for occasions where we have overlap in validations (that is, multiple users have validated same label). A simple count could be how often this user disagrees with the majority in these cases (for when overlapping validators >= 3 users). See https://github.com/ProjectSidewalk/sidewalk-quality-analysis/issues/48.

[jonfroehlich]

Ongoing list of features and their descriptions here: https://github.com/ProjectSidewalk/sidewalk-quality-analysis/blob/master/data/ml-codebook.csv

[jonfroehlich]

@misaugstad, just coming back to this now after a while. Some things I'd still like to check out:

• The time per pano would still be helpful (e.g., min, max, avg, median, stdev of time per pano visited)
• Frequency of labels with open-ended descriptions. We have something similar for frequency of using tags n_label_with_tag and severity n_label_with_severity but not open ended descriptions

[misaugstad]

In a few minutes I'll be committing new CSVs including the column n_label_with_description, n_curb_ramp_with_description, n_missing_curb_ramp_with_description, n_obstacle_with_description, n_surface_problem_with_description, and n_no_sidewalk_with_description. I was already going to upload new CSVs without the minimum validations requirement, and this was an easy set of columns to add.

Time per pano is more complicated to add, so I'm thinking that I should be spending that time on incorporating Dutch translations in Sidewalk. lmk if this should be higher priority at any point.

[jonfroehlich]

Thank you. Agree. I think interactions per pano will be a reasonable proxy for time per pano anyway.