[Feature] Extended Signals Detector Reporting

[atdservicebot]

What application are you using?

Data Tracker

Describe the problem.

New reporting feature

Describe the outcome you'd like to see when this feature is implemented.

Combine detector statuses for primary and secondary. A new column would have to be made for the reporting feature (e.g. "cabinet_detection_status") that merges all detector status. This would be seen in the "Signals Detector Report"

Describe any workarounds you currently have in place or alternative solutions you've considered.

My own python code pulling from the open data portal.

Requested By Joshil B.

Attachment (85.22kb)

Request ID: DTS21-101965

[amenity]

@dianamartin has a prototype going for this.

[amenity]

This may not be a 5 — depends on how far Joshil's existing code can get us. Once this is configured will need to put it on Airflow, etc.

[amenity]

Spoke with Joshil — He actually thought that you were working on it, @dianamartin, but I told him it's going to involve dev resources and would need to go on the next AMD agenda for prioritization.

He's going to send his Python code so that we can size this.

[dianamartin]

@amenity I told him I'd try to get a solution for them and see if I could

[amenity]

From Joshil: Here is my code. The way I combine primary and secondary signal detector data is in lines 130-164. Below is the basic method I use to do this:

1. Take the data of all signal (primary and secondary) detectors and combine the counts of each “Detector Status” per signal. This leaves only one row per intersection (both primary and secondary) that contains the total amount of “BROKEN,” “OK”, “REMOVED”, “INACTIVE”, “VERIFICATION NEEDED”, and “UNKNOWN” detectors.
2. Split the data table into a primary and secondary signals. This allows me to process the secondary signals further by tying them to their primary/parent signal. These get their detector statuses collapsed.
3. The primary and secondary signal data tables get merged for the final data table. This will show how many of each detector status there are per signal cabinet/controller. This then used to make the metric of how many signals are broken (“BROKEN” or “UNKNOWN STATUS”) and turned into a percentage.

[amenity]

Per @johnclary in Slack:

for an ad-hoc script like this, it would typically involve:

• creating an "API View" in Knack which exposes the data
• writing the script
• deploying the script to Airflow

these one-off Data Tracker tasks have historically lived in atd-data-publishing, but the goal is to deprecate that repository by migrating them to atd-knack-services or purpose-built repos (e.g. our finance data integration or KITS integration).the migration adds a bit overhead to a request like this, because we need to think through which repo the code will live in and possibly establish new design patterns. we've established the new patterns for ETLs to Socrata and AGOL, but not for one-off tasks like this one.TMI but that's why i'd estimate this as an 8

[amenity]

Brian is currently using the numbers Joshil puts together to check the Data Tracker reports. Won't completely solve detection reporting needs; other sources' numbers are also off.

Diana's report is working for now, but getting more of a logging capability in addition to current status.