cal-itp / data-infra

Cal-ITP data infrastructure
https://docs.calitp.org/data-infra
GNU Affero General Public License v3.0
48 stars 13 forks source link

Use dynamic years instead of fixed years on models used for NTD Validation #3489

Closed erikamov closed 1 month ago

erikamov commented 1 month ago

Description

Use dynamic years (this_year and last_year) instead of fixed years on models used for NTD Validation. I am using the same logic from fct_ntd_rr20_service_checks.sql that was already created using those dynamic years. So the variable this_year is based on the date that the DAG run started and the last_year is just a previous year.

I also turn int_ntd_rr20_service_1alldata into a materialize view as requested by previous Github Warehouse report.

This change also make the models ready for use on next years without having to change it again.

Type of change

How has this been tested?

Tested creating all models running locally:

poetry run dbt run --select "models/staging/ntd_validation/"
poetry run dbt run --select "models/intermediate/ntd_validation/"
poetry run dbt run --select "models/mart/ntd_validation/"

Those models were successfully created on cal-itp-data-infra-staging.erika_staging and cal-itp-data-infra-staging.erika_mart_ntd_validation and can be validated there.

Post-merge follow-ups

Once the next transform_warehouse runs we can validate the changes on those models in cal-itp-data-infra.

github-actions[bot] commented 1 month ago

Warehouse report 📦

DAG

Legend (in order of precedence)

Resource type Indicator Resolution
Large table-materialized model Orange Make the model incremental
Large model without partitioning or clustering Orange Add partitioning and/or clustering
View with more than one child Yellow Materialize as a table or incremental
Incremental Light green
Table Green
View White

erikamov commented 1 month ago

@KatrinaMKaiser This PR is ready for review. Let me know if you and Christian decided to wait for his return on Monday or if we can merge it today.