cagov / caldata-mdsa-caltrans-pems

CalData's MDSA project with Caltrans on Performance Measurement System (PeMS) data

https://cagov.github.io/caldata-mdsa-caltrans-pems/

MIT License

7 stars 0 forks source link

update incrementality of 2 intermediate models #415

Closed JamesSLogan closed 1 month ago

JamesSLogan commented 1 month ago

408 introduced new joins to these models that affected incremental builds. More specifically, the models are generating too many rows on incremental builds, which affects performance (#414) and overwrites rows with nulls (#410).

To test these changes, I ran these models a couple times after a full refresh to verify that they don't change.

I will execute a full-refresh of these models to fix existing production data issues when this PR merges.

Fixes #410 Fixes #414

kengodleskidot commented 1 month ago

Thank you for making these updates @JamesSLogan. When I tried adding the incremental macros in the CTE's during my PR I ran into errors, so I am glad you were able to get these updated so quickly!