Adding calibration information to SwzDeviceFeed

[j-d-b]

Discussed in https://github.com/usdot-jpo-ode/wzdx/discussions/262

^{Originally posted by #benafischer94 March 17, 2022} # Proposal Given that Smart Work Zone Devices, or more generically, ITS devices include some that require calibration for valid data, e.g.: detection, cameras, various sensors; adding fields for calibration would make sense. I would propose something along the lines of the following as an optional field collection for devices that regularly require calibration. ## Initial Proposal for Format Format is not in full schema formatting, but should provide a basis to kick off the conversation for further debate around what the fields, and potential acceptable values are. I would caution against being all encompassing initially, and instead allow a "good enough" approach to be acceptable as the format evolves and is further used it should naturally lead to full coverage. ```json { "calibration": { "last_calibrated" : "datetime+tz", "manual_calibration": "boolean", "calibrated_by": "Organization that is calibrating rather than individual within an organization", "calibration_standard": "url", "next_calibration_recommended": "datetime+tz", "device_malfunctions": { "malfunctions": [ { "malfunction_type": [ { "description": "a grouping of acceptable malfunction types should accompany e.g. 'pan-tilt':'boolean', 'detection':'boolean', 'led:'boolean'", "severity": "integer", } ], }, ], "functionality_impacted":[ { "name": "generalize name of functionality", "description": "one of an acceptable set of functionalities based on device. e.g: led_out, led_stuck_on, coupled with what is affected" } ], "total_severity": "integer" } } } ``` ## Rationale This will allow organizations to determine the reliability of data coming into system. e.g. an auto manufacturer may require a higher level of confidence in detection devices than a parking lot operator. Additionally, help assist in keeping operators honest about malfunctions in devices as well as initial deployment being verified. A growing number of OEMs are expressing concern of devices being improperly deployed to the field reducing the actual impact. This leads to safety theater, organizations implementing ITS devices with no accountability of proper deployment to effectively check a box with contract owners. It should hopefully help provide a structure for allowing adoption of requirements on calibration of devices to be properly certified. This could be a multi enforcement effort from OEMs, FHWA, USDOT, State DOTs, down to municipalities and private operators. This makes the inclusion of the calibration_standard an important part so that it is clear to installers and operators what the expected calibration standard is. This should also lead to transparency within contracts as to what the standards are that are being adhered to for third party validation. Additionally, this can help lead to further creation of automatic device health-checking capabilities. e.g: a higher confidence device can be used to provide baseline in corridor detection services. ## Field clarification * Last Calibrated: a timestamp of when the device was calibrated. For manual calibrations where technicians may not be entering the data immediately on the side of the road, the date with zeroed out hours should be acceptable. For machine based calibration, an exact timestamp makes sense. * Manual Calibration: A true false to determine if human calibrated or done by machine * Calibrated by: The organization doing the calibration and certification. It would be reasonable for organizations to track the individual doing it, but not to publish the exact individual for all. * Calibration standard: Manufacturer and organizations calibration standards vary, by publishing the exact standard followed, ambiguity can be eliminated. OEMs should make a linkable version for their devices that they recommend public for linking and best practice adhesion. * Device malfunctions: Some devices may have multiple capability e.g. a PDMS with detection. The detection is a secondary function, and could be malfunctioning while the PDMS is still considered good. so an array to support all possible malfunctions * Malfunction type: A generic set of lists that can be determined across a device of a given type. E.g. LEDs for PDMS, speed variance for detection. * Severity: A number that can be assigned to the impact of the individual malfunction. Working group could come up with standard based severity for malfunctions of primary, secondary, and tertiary function of components on a device of a given type. * Functionality impacted: Some standardized items impact of the specific malfunction to devices. This can be regardless of primary purpose of the device. An array of impacted functions * Name: Generic name of functionality e.g. message viewability, detection * Description: Generic descriptions to accompany this for reporting what it means to end-users. * Total severity: A total severity for all individual malfunctions. Organizations can make decisions and set thresholds for usability of data based on the total severity number. ## Call To Action Initial implementation would be optional for all devices to allow a period of adoption, either based on time or major version revision. This should reduce heartburn associated with making a requirement for calibration of devices. Given the high priority and push for devices to be deployed into work zones, making sure that they are reporting good actionable data should be accompanying this push. Along with the rapid development that the WZDX has demonstrated, a draft proposal should be possible within a three to six month period. ## Justification / Resources A few examples of systems not working due to failures in comms, calibration, etc. As well as existing data feeds use verification confidence. The quiet part not always acknowledged out-loud is that the majority of devices work well when calibrated, but if no one is watching, it gets skipped occasionally. Redeployment due to changing work zones can exasperate this problem. These sources aren't exhaustive, and there admittedly is a lot of anecdotal evidence as well. Though there is no shortage of documentation to support improperly calibrated/deployed devices if individuals request additional it can be presented. * [England smart system failure](https://www.tech-gate.org/usa/2021/08/29/smart-motorway-system-is-nicknamed-die-now-by-horrified-staff-following-computer-crashes/) * [TPIMS Data Feed specification](https://trucksparkhere.com/wp-content/uploads/2019/01/TPIMS_TruckParking_Data_Interface_App_Developers_V1.1.pdf) * [Data sources used in RCA](https://fdotwww.blob.core.windows.net/sitefinity/docs/default-source/research/reports/fdot-bd544-26-rpt.pdf?sfvrsn=c1bf8089_2)

[tfoster-vm]

I think this proposal has good intent, but is drilling too deep and not really accommodating most agency users. We need to walk before we run and in my opinion most systems and SWZ specs already require an acceptable level of accuracy for traffic sensors. If devices are not calibrated or out of service or other...they should either not be in the feed or be presented with an error or unknown condition/status. In addition, this would likely have to be manually entered by the provider or agency to populate the feed for this proposal. We believe that as much as possible needs to be automated to allow for success and widespread adoption. If traffic sensor are not providing good data 24/7, then we have a much bigger issue and should not even be providing that associated data. We either trust the date or the spec if not being satisfied and needs to be corrected ASAP. Garbage in-garbage out. Most agencies we do business with do not really care about the maintenance and calibration issues, they just want good data and system performance 24/7 and they have ways to enforce that outside of the WZDx and SWZ Device feeds. We are open to this overall discussion but need to not get caught in the weeds. We are not saying that sensors are perfect all the time, they are not. But is the sensor data representative of the current conditions to an acceptable level? If yes, then let the traffic data flow. If not than let's discuss how to address that in as automated a way as possible. The data must represent the conditions and have integrity, but not be a roadblock to the larger purpose. My two cents.

There’s also another issue that is trying to solve essentially the same problem (verify data integrity) but differently, with a timestamp per value instead of manual checkups: https://github.com/usdot-jpo-ode/wzdx/issues/253. This shows that there’s some kind of need and consumers of the data want to be confident in the data one way or another. I am guessing most data consumers just want to ensure the data is valid, but timestamps alone do not eliminate that potential of invalid data. I know internally we do many other things to ensure data integrity, but I cannot assure you that all data suppliers have the same levels of ensuring data integrity upon deployment and during the entire time in the field. If devices are calibrated and deployed correctly and accepted by an agency, why would we need to include a confidence/calibration value every minute in such a feed? I look forward to the dialogue (especially from active agencies). This may or may not be something we solve this development cycle.

[benafischer94]

@tfoster-vm , could you elaborate how it fails to accommodate agency users? Coming from that background, having standardized data formats around data calibration provides more benefits than negatives that I can think of. Having the spec marked as optional for a release or more, allows them to adjust systems to begin using before mandatory status.

The calibration itself, could yes, have to be input manually. Some devices are adding the ability to self-calibrate via implementations of ML algos and the like. Other devices without such capability, sure they'll need an interface to update.

Regardless, this doesn't stop the feed from automatically updating. Additionally, this gives manufacturers the ability to add business rules such as if device moved more than threshold distance -> mark for calibration. As @rdsheckler mentioned in https://github.com/usdot-jpo-ode/wzdx/discussions/262#discussioncomment-2382057_ this is something others are looking to do regardless. The agencies and researchers I've worked with have always mentioned that more filtering ability is a positive. This gives them an additional criteria for filtering with more granularity than Source->Score applied to all in feed.

Additionally, by the device in the field providing a score that can be used at ingestion time, it provides the OEM the ability to help the Agency develop rules for automatically flagging devices for calibration. We've all been a part of a workzone where a device may be deployed initially and calibrated, then it's moved/turned by a worker for whatever reason. Maybe it gets put back correctly, maybe not. This allows business rules to be developed to help flag this so the agency can charge the device maintainer, or agency staff to return for calibration.

Depending on the TMS exporting the data, they may choose to include all devices in the system. So you end up with both WZ specific devices, as well all permanent ITS devices that exist within a given WZ, or a mixture of both. Ideally the system should be able to track the reliability/calibration of the data. They may also choose to pass through directly the data as presented from the device, or apply additional business rules prior to publishing and giving their mark of approval. This can help reduce the need for Agency staff to login and toggle specific devices as publish/don't publish by generating additional business rules around publish scores.

253 looks to solve a similar issue, I agree. But there's a difference between a system lagging with good data, and the determination of good/bad data. I foresee the need to support both.

[tfoster-vm]

@benafischer94 in general I think the more manual entry there is the less likely it will be utilized. I think most of what you are proposing is manual entry. If I am wrong, please let me know. I suggest we see if there is wider support for this, since the only comments are from the two of us. I will solicit comments at our next subgroup meeting and you can feel to add context or specifics during that meeting.

[j-d-b]

Closed based on SWZ co-chairs discussion on 2022-09-09:

• High complexity
• Significant manual entry to populate