search

cagov / caldata-mdsa-caltrans-pems

CalData's MDSA project with Caltrans on Performance Measurement System (PeMS) data
https://cagov.github.io/caldata-mdsa-caltrans-pems/
MIT License
7 stars 0 forks source link

developed imputation QC model that will monitor daily sample balance for different imputation methods #438

Closed mmmiah closed 1 day ago

mmmiah commented 1 month ago

This PR is built based on issue #437 which counted the daily sample size by different imputation and check the balance of sample size between the daily observed and imputation for QC purpose

mmmiah commented 2 weeks ago

@kengodleskidot , would you be able to add three test on this imputation summary table in your previous test file. One for speed, one for volume and another for occupancy. Observed, imputed and observed_unimputed sum for each metric should be 100 or super minor less image

mmmiah commented 6 days ago

@JamesSLogan , Could you please look on this PR whenever you get time. It is failed due to dbt CI failure of 00:24:39 11 of 83 ERROR creating sql incremental model dbt_cloud_pr_15_438_diagnostics.int_diagnostics__constant_occupancy [ERROR in 1.70s]. I tried to drop the relevant one, but still failed. Thanks! image

summer-mothwood commented 3 days ago

@JamesSLogan this error in the constant occupancy code is happening because Mintu is working off a version of the code before the change I applied to remove station id from its output. I'm assuming the best way to address this is to rebase Mintu's branch to the current version of the code then trying again. Is there a good way to do that in dbt Cloud?

ian-r-rose commented 3 days ago

I learned the hard way that attempting to rebase with dbt Cloud is a recipe for heartache. If using cloud, it better to just merge from main.

mmmiah commented 1 day ago

Woow, It seems that it passed the dbt failure! If you do not have any comments, please go ahead and approve this PR, so that we can merge it to main!

mmmiah commented 1 day ago

Thanks Mintu! Has an issue been created for adding the tests discussed in this PR?

Ken is supposed to add test separately. No test in this PR