ccao-data / model-res-avm

Automated valuation model for all class 200 residential properties in Cook County (except vacant land and condos)
GNU Affero General Public License v3.0
20 stars 3 forks source link

Add workflow/process for tagging models with `run_type` #180

Closed dfsnow closed 5 months ago

dfsnow commented 5 months ago

Model runs can have different purposes, such as testing a new feature, finalizing an idea, etc. We don't currently have a way to identify the run "type" for a given model. As such, it can be difficult to know which model runs are actually worthy of consideration given the volume of runs we produce. @ccao-jardine proposes the following model typology:

We should update our pipeline to use this typology. This would involve three distinct tasks:

  1. Update the existing pipeline workflow to include a run_type parameter in the manual workflow_dispatch. This can follow the same pattern as the feature flags, such as cv_enable, etc. The result would be each model run receiving one type from the types above.
  2. Create an additional workflow to update the type of an existing run. This can be used to change the type once we examine the results of a run. This would involve updating the metadata parquet file. This can follow a UX pattern similar to the delete-model-run workflow.
  3. Manually retagging the historical model runs from 2022 and 2023.

The model tags can be stored in the repurposed run_type field in the model.metadata table. This was previously used to differentiate "full" model runs (where everything ran) from "limited" runs (which only ran prediction on the test set, to enable runs on CI runners). The run_type was an input to some conditional logic that determined what exactly ran, but that logic is not longer used and it's not necessary to store the original type.

jeancochrane commented 5 months ago

@dfsnow I think steps 1 and 2 should only take a day or two, as long as we're only setting or updating a metadata parameter that isn't used anywhere in the pipeline itself. I have less of a sense of how long step 3 will take but I assume it should be on the scale of minutes once we know how we want to tag each model.