Make slim CI work better

[ian-r-rose]

We currently use "Slim CI" for this project, which (in theory), only runs models that are affected by a PR in CI. The intention here is that we can save time and money by just running the stuff that needs to be run. It works by comparing the code in the branch with that in the nightly production job, and identifying ones where there is a diff.

This is not working as intended because of the way we do some different logic depending upon whether we are "dev" or "prd". In particular, we select less data in dev for a lot of our larger models to improve the developer experience for those models. Unfortunately, this means that there is a real difference between dev and prod, even if the pre-rendered model has not changed, thus creating some false positives in our slim CI job.

I'm not sure of the best way to fix this right now.

[jkarpen]

Assigning to @kengodleskidot to review and see if he has any thoughts on how to move forward on this one.

[jkarpen]

We put this in the backlog until @ian-r-rose returns to get his input on how to handle this item.

[jkarpen]

Summer will tackle this research project, will assign once she is added to the right GitHub team.

[summer-mothwood]

Note: the dbt documentation page on state modified caveats have been updated a few times over the last few weeks, with some changes that might specifically address our issue -- but would require a version upgrade from dbt 1.7.18 (current Caltrans dbt version) to dbt 1.9+ or dbt Versionless.

• https://github.com/dbt-labs/docs.getdbt.com/pull/6207

Beginning in dbt Core 1.9, when you set the state_modified_compare_vars behavior flag to True and a model uses a var or env_var in its definition, dbt will identify that lineage in such a way that it will include the model in state:modified when the var or env_var value has changed.

• https://github.com/dbt-labs/docs.getdbt.com/commit/cb69cf640a0fa91b52488669fd28c3ecae860b1d

You need to build the state directory using dbt v1.9 or higher, or Versionless dbt Cloud, and you need to set state_modified_compare_more_unrendered_values to true within your dbt_project.yml. If the state directory was built with an older dbt version or if the state_modified_compare_more_unrendered_values behavior change flag was either not set or set to false, you need to rebuild the state directory to avoid false positives during state comparison with state:modified.

I'm still looking into this, but wanted to document these updates for now.

[ian-r-rose]

Ooh, these new flags seem like they might be helpful! I think we are feeling enough pain in CI that an upgrade to 1.9 would be worth it.

[summer-mothwood]

Ian and I talked about this today and agreed on the following:

• these new flags seem like they might exactly solve our slimci issues
• this is a big enough problem (using unnecessary resources!) that we think it's worth it to prioritize upgrading our dbt version in order to test/use
  • 1.7 entered end-of-life on Nov 1st 2024 anyway
• Before we upgrade our dbt version, I'm going to take a look at documentation on the move from 1.7+ -> Versionless and call out any red flags for the team, then develop a schedule / communication plan for the upgrade.

Once we're ready to upgrade our dbt version, the general steps to do so is:

• Update the pyproject.toml
  • this should trigger a CI check
• Dbt cloud environments (in this order)
  • Development
  • CI
  • Production

Once in Versionless, we can then start working on setting the state_modified_compare_more_unrendered_values flag to true (will need to rebuild the state first, as mentioned in the documentation). More reading here: https://docs.getdbt.com/reference/global-configs/behavior-changes#source-definitions-for-state

[summer-mothwood]

I created a new issue in the DSE repo for upgrading/testing dbt Versionless https://github.com/cagov/data-infrastructure/issues/423. Will do the work there before coming back to this issue to implement the new state modified flag to address slim CI false positives.

[jkarpen]

This is blocked until dbt versionless change is implemented.