cal-itp / data-infra

Cal-ITP data infrastructure
https://docs.calitp.org/data-infra
GNU Affero General Public License v3.0
48 stars 13 forks source link

Creating an incremental table for vendor vehicle message age by day. #3477

Closed fsalemi closed 1 month ago

fsalemi commented 2 months ago

Creating a new incremental table for vendor vehicle position message age by day.

Resolves issue https://github.com/cal-itp/data-infra/issues/3478

Type of change

How has this been tested?

Yes, _poetry run dbt run -s +fct_daily_vendor_vehicle_positions_message_agesummary

20:17:15 Finished running 12 view models, 11 table models, 1 incremental model in 0 hours 4 minutes and 39.96 seconds (279.96s). 20:17:15 20:17:15 Completed successfully 20:17:15 20:17:15 Done. PASS=24 WARN=0 ERROR=0 SKIP=0 TOTAL=24

Post-merge follow-ups

Document any actions that must be taken post-merge to deploy or otherwise implement the changes in this PR (for example, running a full refresh of some incremental model in dbt). If these actions will take more than a few hours after the merge or if they will be completed by someone other than the PR author, please create a dedicated follow-up issue and link it here to track resolution.

github-actions[bot] commented 2 months ago

Warehouse report 📦

Checks/potential follow-ups

Checks indicate the following action items may be necessary.

New models 🌱

calitp_warehouse.mart.gtfs_quality.fct_daily_vendor_vehicle_positions_message_age_summary

DAG

Legend (in order of precedence)

Resource type Indicator Resolution
Large table-materialized model Orange Make the model incremental
Large model without partitioning or clustering Orange Add partitioning and/or clustering
View with more than one child Yellow Materialize as a table or incremental
Incremental Light green
Table Green
View White

vevetron commented 2 months ago

I'm helping @fsalemi install pre-commit hooks into his environment.

fsalemi commented 1 month ago

@evansiroky Although joining directly with bridge_organizations_x_gtfs_datasets_produced also works, but aggregating it would save some processing time since there are a lot of consecutive records for every vendor-to-agency association by date in this table. I incorporated your requested changes anyways.