cagov / caldata-mdsa-caltrans-pems

CalData's MDSA project with Caltrans on Performance Measurement System (PeMS) data
https://cagov.github.io/caldata-mdsa-caltrans-pems/
MIT License
7 stars 0 forks source link

Debug Null Data in Clearinghouse 5-minute and Detector Status Models #410

Closed kengodleskidot closed 1 month ago

kengodleskidot commented 1 month ago

@JamesSLogan after merging #408 yesterday I did some QC checks this morning and there are too many Null values in the int_clearinghouse__detector_agg_five_minutes_with_missing_rows.sql and int_diagnostics__detector_status.sql models for past dates. The last few days data looks good, but we have too many Null values for the older data. A full resfresh may fix this but we can discuss this in more detail at the sprint meeting this morning 10/3/24.

kengodleskidot commented 1 month ago

As an example, the screenshot below shows no Good detectors on 9/26/24 and 9/27/24 but from 9/28/24 - 9/30/24 the statuses look good and are consistent with expected values (~25k Good detectors daily).

image image