Build a crash prediction modeling application that leverages multiple data sources to generate a set of dynamic predictions we can use to identify potential trouble spots and direct timely safety interventions.
As a user of the predictions or the contributing factors, I want to be reasonably sure the predictions/important features don't shift around a great deal with minor changes to parameters. I also want clear alerts about poor model performance and suggestions on how to increase performance.
A few tasks here
Implement model testing: Ensure that there's not huge variation in the features that are seen to be important at any stage in CV and training
Apply alerts throughout when model performance drops below AUC 0.5
This is more of a general issue, tracking model drift, it might make more sense for this to be on the user. We've implemented warnings, so we'll leave it at that for now.
As a user of the predictions or the contributing factors, I want to be reasonably sure the predictions/important features don't shift around a great deal with minor changes to parameters. I also want clear alerts about poor model performance and suggestions on how to increase performance.
A few tasks here