erikamov commented 1 week ago

Description

This PR updates mart_ntd.dim_monthly_ntd_ridership_with_adjustments and dependents tables to use the new NTD monthly ridership models processed in Airflow as requested on #3519.

The xlsx file link was updated to import new data (July, August, and September).

All documentation for NTD monthly ridership tables and columns were added.

As part of fixing the nomenclature on other NTD models (as requested by @tiffanychu90 on #3497), the main table is now mart_ntd.dim_monthly_ridership_with_adjustments, removing "ntd" from the name since it is part of the mart_ntd group.

Also to keep consistent with other NTD models, some column names where changed on mart_ntd.dim_monthly_ridership_with_adjustments:

uza_name is now primary_uza_name
uace_cd is now primary_uza_code
mode_full_name is now mode_name,
ts is now execution_ts
Removed year column since it was only part of the old scrape process
Added key column as a unique key generated from ntd_id, mode, tos, period_month, period_year

This PR also remove the deprecated tables:

external_ntd_data_products.monthly_ridership_with_adjustments_upt
external_ntd_data_products.monthly_ridership_with_adjustments_vrm
external_ntd_data_products.monthly_ridership_with_adjustments_vrh
external_ntd_data_products.monthly_ridership_with_adjustments_voms
staging.stg_ntd__monthly_ridership_with_adjustments_upt
staging.stg_ntd__monthly_ridership_with_adjustments_vrm
staging.stg_ntd__monthly_ridership_with_adjustments_vrh
staging.stg_ntd__monthly_ridership_with_adjustments_voms

Type of change

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] Documentation

How has this been tested?

Models tested and created on staging.

 ❯ poetry run dbt run -s +models/mart/ntd/dim_monthly_ridership_with_adjustments.sql
02:16:56  Running with dbt=1.5.1
02:16:59  Found 418 models, 979 tests, 0 snapshots, 0 analyses, 852 macros, 0 operations, 12 seed files, 170 sources, 4 exposures, 0 metrics, 0 groups
02:16:59
02:17:03  Concurrency: 8 threads (target='dev')
02:17:03
02:17:03  1 of 11 START sql table model erikap_staging.int_ntd__modes .................... [RUN]
02:17:03  2 of 11 START sql view model erikap_staging.stg_ntd_ridership_historical__complete_monthly_ridership_with_adjustments_and_estimates__upt  [RUN]
02:17:03  3 of 11 START sql view model erikap_staging.stg_ntd_ridership_historical__complete_monthly_ridership_with_adjustments_and_estimates__voms  [RUN]
02:17:03  4 of 11 START sql view model erikap_staging.stg_ntd_ridership_historical__complete_monthly_ridership_with_adjustments_and_estimates__vrh  [RUN]
02:17:03  5 of 11 START sql view model erikap_staging.stg_ntd_ridership_historical__complete_monthly_ridership_with_adjustments_and_estimates__vrm  [RUN]
02:17:04  5 of 11 OK created sql view model erikap_staging.stg_ntd_ridership_historical__complete_monthly_ridership_with_adjustments_and_estimates__vrm  [CREATE VIEW (0 processed) in 1.27s]
02:17:04  6 of 11 START sql table model erikap_staging.int_ntd__monthly_ridership_with_adjustments_vrm  [RUN]
02:17:04  3 of 11 OK created sql view model erikap_staging.stg_ntd_ridership_historical__complete_monthly_ridership_with_adjustments_and_estimates__voms  [CREATE VIEW (0 processed) in 1.31s]
02:17:04  7 of 11 START sql table model erikap_staging.int_ntd__monthly_ridership_with_adjustments_voms  [RUN]
02:17:04  4 of 11 OK created sql view model erikap_staging.stg_ntd_ridership_historical__complete_monthly_ridership_with_adjustments_and_estimates__vrh  [CREATE VIEW (0 processed) in 1.32s]
02:17:04  8 of 11 START sql table model erikap_staging.int_ntd__monthly_ridership_with_adjustments_vrh  [RUN]
02:17:04  2 of 11 OK created sql view model erikap_staging.stg_ntd_ridership_historical__complete_monthly_ridership_with_adjustments_and_estimates__upt  [CREATE VIEW (0 processed) in 1.34s]
02:17:04  9 of 11 START sql table model erikap_staging.int_ntd__monthly_ridership_with_adjustments_upt  [RUN]
02:17:05  1 of 11 OK created sql table model erikap_staging.int_ntd__modes ............... [CREATE TABLE (22.0 rows, 374.0 Bytes processed) in 2.37s]
02:17:45  8 of 11 OK created sql table model erikap_staging.int_ntd__monthly_ridership_with_adjustments_vrh  [CREATE TABLE (618.3k rows, 163.0 GiB processed) in 41.40s]
02:17:46  6 of 11 OK created sql table model erikap_staging.int_ntd__monthly_ridership_with_adjustments_vrm  [CREATE TABLE (618.3k rows, 169.2 GiB processed) in 42.44s]
02:17:48  7 of 11 OK created sql table model erikap_staging.int_ntd__monthly_ridership_with_adjustments_voms  [CREATE TABLE (618.3k rows, 151.3 GiB processed) in 44.00s]
02:17:49  9 of 11 OK created sql table model erikap_staging.int_ntd__monthly_ridership_with_adjustments_upt  [CREATE TABLE (618.3k rows, 167.5 GiB processed) in 44.55s]
02:17:49  10 of 11 START sql table model erikap_staging.int_ntd__monthly_ridership_with_adjustments_joined  [RUN]
02:17:54  10 of 11 OK created sql table model erikap_staging.int_ntd__monthly_ridership_with_adjustments_joined  [CREATE TABLE (618.3k rows, 253.8 MiB processed) in 5.20s]
02:17:54  11 of 11 START sql table model erikap_mart_ntd.dim_monthly_ridership_with_adjustments  [RUN]
02:18:00  11 of 11 OK created sql table model erikap_mart_ntd.dim_monthly_ridership_with_adjustments  [CREATE TABLE (618.3k rows, 99.4 MiB processed) in 6.03s]
02:18:00
02:18:00  Finished running 4 view models, 7 table models in 0 hours 1 minutes and 0.61 seconds (60.61s).
02:18:00
02:18:00  Completed successfully
02:18:00
02:18:00  Done. PASS=11 WARN=0 ERROR=0 SKIP=0 TOTAL=11

 ❯ poetry run dbt test -s +models/mart/ntd/dim_monthly_ridership_with_adjustments.sql
02:18:34  Running with dbt=1.5.1
02:18:36  Found 418 models, 979 tests, 0 snapshots, 0 analyses, 852 macros, 0 operations, 12 seed files, 170 sources, 4 exposures, 0 metrics, 0 groups
02:18:36
02:18:38  Concurrency: 8 threads (target='dev')
02:18:38
02:18:38  1 of 13 START test not_null_dim_monthly_ridership_with_adjustments_key ......... [RUN]
02:18:38  2 of 13 START test not_null_dim_monthly_ridership_with_adjustments_ntd_id ...... [RUN]
02:18:38  3 of 13 START test not_null_int_ntd__monthly_ridership_with_adjustments_joined_ntd_id  [RUN]
02:18:38  4 of 13 START test not_null_int_ntd__monthly_ridership_with_adjustments_upt_ntd_id  [RUN]
02:18:38  5 of 13 START test not_null_int_ntd__monthly_ridership_with_adjustments_voms_ntd_id  [RUN]
02:18:38  6 of 13 START test not_null_int_ntd__monthly_ridership_with_adjustments_vrh_ntd_id  [RUN]
02:18:38  7 of 13 START test not_null_int_ntd__monthly_ridership_with_adjustments_vrm_ntd_id  [RUN]
02:18:38  8 of 13 START test not_null_stg_ntd_ridership_historical__complete_monthly_ridership_with_adjustments_and_estimates__upt_ntd_id  [RUN]
02:18:38  7 of 13 PASS not_null_int_ntd__monthly_ridership_with_adjustments_vrm_ntd_id ... [PASS in 0.83s]
02:18:38  5 of 13 PASS not_null_int_ntd__monthly_ridership_with_adjustments_voms_ntd_id .. [PASS in 0.85s]
02:18:38  9 of 13 START test not_null_stg_ntd_ridership_historical__complete_monthly_ridership_with_adjustments_and_estimates__voms_ntd_id  [RUN]
02:18:38  10 of 13 START test not_null_stg_ntd_ridership_historical__complete_monthly_ridership_with_adjustments_and_estimates__vrh_ntd_id  [RUN]
02:18:38  6 of 13 PASS not_null_int_ntd__monthly_ridership_with_adjustments_vrh_ntd_id ... [PASS in 0.88s]
02:18:38  11 of 13 START test not_null_stg_ntd_ridership_historical__complete_monthly_ridership_with_adjustments_and_estimates__vrm_ntd_id  [RUN]
02:18:38  4 of 13 PASS not_null_int_ntd__monthly_ridership_with_adjustments_upt_ntd_id ... [PASS in 0.91s]
02:18:38  12 of 13 START test unique_dim_monthly_ridership_with_adjustments_key .......... [RUN]
02:18:39  2 of 13 PASS not_null_dim_monthly_ridership_with_adjustments_ntd_id ............ [PASS in 0.96s]
02:18:39  13 of 13 START test unique_ntd_modes_to_full_names_ntd_mode_abbreviation ....... [RUN]
02:18:39  3 of 13 PASS not_null_int_ntd__monthly_ridership_with_adjustments_joined_ntd_id  [PASS in 1.02s]
02:18:39  1 of 13 PASS not_null_dim_monthly_ridership_with_adjustments_key ............... [PASS in 1.08s]
02:18:39  13 of 13 PASS unique_ntd_modes_to_full_names_ntd_mode_abbreviation ............. [PASS in 0.70s]
02:18:40  12 of 13 PASS unique_dim_monthly_ridership_with_adjustments_key ................ [PASS in 1.41s]
02:18:40  8 of 13 PASS not_null_stg_ntd_ridership_historical__complete_monthly_ridership_with_adjustments_and_estimates__upt_ntd_id  [PASS in 2.41s]
02:18:41  9 of 13 PASS not_null_stg_ntd_ridership_historical__complete_monthly_ridership_with_adjustments_and_estimates__voms_ntd_id  [PASS in 3.01s]
02:18:41  10 of 13 PASS not_null_stg_ntd_ridership_historical__complete_monthly_ridership_with_adjustments_and_estimates__vrh_ntd_id  [PASS in 3.03s]
02:18:42  11 of 13 PASS not_null_stg_ntd_ridership_historical__complete_monthly_ridership_with_adjustments_and_estimates__vrm_ntd_id  [PASS in 3.20s]
02:18:42
02:18:42  Finished running 13 tests in 0 hours 0 minutes and 5.71 seconds (5.71s).
02:18:42
02:18:42  Completed successfully
02:18:42
02:18:42  Done. PASS=13 WARN=0 ERROR=0 SKIP=0 TOTAL=13

Post-merge follow-ups

[ ] No action required
[x] Actions required (specified below)
Check for the correct creation of the new model after the next DAG runs with data until September/2024.
Replace the use of mart_ntd.dim_monthly_ntd_ridership_with_adjustments by mart_ntd.dim_monthly_ridership_with_adjustments on Metabase and Jupyter notebooks.
Change the schedule of the DAG to weekly instead of daily

github-actions[bot] commented 1 week ago

Warehouse report 📦

Checks/potential follow-ups

Checks indicate the following action items may be necessary.

[ ] For new models, do they all have a surrogate primary key that is tested to be not-null and unique?

New models 🌱

calitp_warehouse.mart.ntd.dim_monthly_ridership_with_adjustments

DAG

Legend (in order of precedence)

Resource type	Indicator	Resolution
Large table-materialized model	Orange	Make the model incremental
Large model without partitioning or clustering	Orange	Add partitioning and/or clustering
View with more than one child	Yellow	Materialize as a table or incremental
Incremental	Light green
Table	Green
View	White

erikamov commented 6 days ago

Did a 1-1 to review this. A good update on the previous work. I assume we'll remove the outdated models in the next pr!

Thank you Vivek! I remove the outdated models as requested. :D

cal-itp / data-infra

Use new NTD monthly Ridership models for mart_ntd.dim_monthly_ntd_ridership_with_adjustments #3549