dssg / tyra

Prediction model evaluation
Other
4 stars 1 forks source link

Add model group #6

Closed thcrock closed 7 years ago

thcrock commented 7 years ago

Model groups! We have a schema:

The models table has a model_group_id. Each model_id within a given model_group_id has all of the same attributes except for the test_end_date. Each model_group_id in the database at one time is understood to have the same set of test_end_dates, though this is not enforced in the schema (procedure outlined below to work around this).

Model search table should:

  1. Work with model groups (display model_group_id instead of model_id), and
  2. Only show the results for the second to most recent test_end_date from each group, as this is the most relevant at a glance. (Later we may add this as a search option).

To compute the list of available model dates, select the most recent model_id (by runtime) and take all of the test_end_dates in the model's model_group. Then take the second-most-recent date, and only show the metrics for the models of that test_end_date.

thcrock commented 7 years ago

Note: Call out the fact (at the top of the graph, most likely) that the metrics are as of a certain year and are not necessarily indicative of model stability.

thcrock commented 7 years ago

https://github.com/dssg/tyra/pull/30/