ccao-data / wiki

Handbook, how-tos, and other documentation
3 stars 1 forks source link

Create model selection SOP #1

Open dfsnow opened 1 year ago

dfsnow commented 1 year ago

We should write a Standard Operating Procedure (SOP) codifying how and why we select final valuation model runs. This is mostly about formalizing and documenting the best practices and making sure that we're implementing them internally.

dfsnow commented 1 year ago

We should also reconsider using our current method for bootstrapped CI calculations. We rarely use the CIs now and I'm not confident that the original implementation works as intended. See https://stats.stackexchange.com/questions/433866/bootstrapped-confidence-intervals-for-performance-metrics-of-predictive-models.