search

Chicago / food-inspections-evaluation

This repository contains the code to generate predictions of critical violations at food establishments in Chicago. It also contains the results of an evaluation of the effectiveness of those predictions.
http://chicago.github.io/food-inspections-evaluation/
Other
410 stars 130 forks source link

Why is there a crook in the "Best Possible" line? #69

Closed orborde closed 9 years ago

orborde commented 9 years ago

In the current version of the white paper (page 12), the chart "Comparing cumulative violations discovered with additional optimal path" has a comparison of the model performance to a "Best Possible" path.

The "Best Possible" path seems to be computed on the assumption that every inspection finds a critical violation until all of the critical violations in the evaluation set are found. If that's the case, why is there a crook in the line, instead of a straight line from the origin up to the maximum? This crook would seem to imply that even the prescient "Best Possible" inspection pattern that can magically see the inspection outcome ahead of time would somehow fail to prioritize the actual violations and waste a few inspections on non-violating businesses before going back to finding violations.

What is going on here?

orborde commented 9 years ago

I took a look at the internal data (specifically, the "comp" table computed in forecasting-restaurants-with-critical-violations-in-Chicago.Rmd), and it looks like the explanation is pretty mundane: the "crooks" in the Best-Possible and Worst-Possible lines are weekends. Inspections evidently only happen on weekdays. Even psychic inspectors will be slowed down in their violation-finding by having weekends off.

geneorama commented 9 years ago

Yes, weekends are part of it.

In this comparison we're looking at inspections / day, and critical violations found / day. We want to see how many critical violations were found, and how many would have been found.

I wanted to make sure that we were looking and assuming the exact same workload would have happened with optimized inspections.

After some head scratching, I opted to keep the original vector of dates and the historical results, and then join in the optimized results (both model optimized results, and "perfectly good" optimized results, and "perfectly bad" optimized results). The each optimized result is sorted by its score, but it's being joined to a matrix that is sorted chronologically.

See lines 119-120 in https://github.com/Chicago/food-inspections-evaluation/blob/master/REPORTS/forecasting-restaurants-with-critical-violations-in-Chicago.Rmd#L119-120

Perhaps if the inspectors had the perfect list they wouldn't have been able to visit as many restaurants on some days if the perfect list were more geographically scattered than their actual work, and likewise it could have worked in their favor on other days. So, this plot is not taking into account travel time / changes in travel time. However, it's the best that I could I could think of. I think that it makes the lines more comparable than if we had made assumptions about average visits per day.

Good question, you're obviously paying attention!