insight-lane / crash-model

Build a crash prediction modeling application that leverages multiple data sources to generate a set of dynamic predictions we can use to identify potential trouble spots and direct timely safety interventions.
https://insightlane.org
MIT License
113 stars 40 forks source link

Implement model consistency checks #170

Closed bpben closed 11 months ago

bpben commented 5 years ago

As a user of the predictions or the contributing factors, I want to be reasonably sure the predictions/important features don't shift around a great deal with minor changes to parameters. I also want clear alerts about poor model performance and suggestions on how to increase performance.

A few tasks here

bpben commented 11 months ago

This is more of a general issue, tracking model drift, it might make more sense for this to be on the user. We've implemented warnings, so we'll leave it at that for now.