jonfroehlich
commented
5 years ago - [ ] Analyze performance in terms of neighborhood not just label type. Are there geographic patterns to performance? If so, why? Update (7/11/19): still an interesting idea, perhaps push to its own Issue
- [ ] Train and test across neighborhoods (similar to train and test across cities but should be easier though not as high impact). Update (7/11/19): still an interesting idea, perhaps push to its own Issue
- [x] Qualitatively analyze results and taxonomize errors (we do this in our CHI’19 paper for manually labeling mistakes)
- [x] Project Sidewalk is about to deploy into at least two more cities (we are maybe a month away from pilots). Another cool idea—if we can get the datasets by the right time—would be to see how we would do by training in one city and testing in another. It’s this sort of thing that would push our research from exciting to REALLY EXCITING!!
- [x] With the sliding window implementation, another comparison point is simply to compare the performance of the computer vision algorithms against the different user groups that we had in project sidewalk. In fact, this is absolutely crucial. :) For fairness, we have to do this comparison with only a subset of the data (the ground truth data). In sum, across all user groups, we found ~63% recall and ~71% precision. See paper for more details.
- [ ] In addition to the above, we should also compare to Tohme baseline to get a sense of how we're doing (will be important for paper). Update (7/11/19): this turned out harder than expected because we couldn't track down the dataset sadly, see: https://github.com/galenweld/project_sidewalk_ml/issues/13
- [x] What is the effect of training size on performance? What is the effect of pano size on performance?