Configuration Diagnostics (Direction Switching)

[kengodleskidot]

In the previous technical committee meeting (5/30/24) one of the members brough up this topic which is discussed in more detail below:

The main reason for this configuration diagnostic is to identify detectors that have incorrect direction configuration information. There are a number of cases where the detector is on the opposite side of the freeway which is in conflict with the configuration information provided by the district. The algorithm described below attempts to automatically identify detectors which are on the opposite side of the road. There may be an opportunity to use geospatial functions to perform this diagnostic so I recommend further discussion on what geospatial tools may be available before diving into creating a model using the algorithm described below:

Each detector in the system is assigned a score between 0 and 100, with zero meaning that it's highly likely that this detector is on the opposite side of the freeway. Since a high number indicates that the detector is very similar to its neighbors in the same direction, this score is usually referred to as Neighbor Affinity.

Conceptually we are looking for detectors that have an hourly flow pattern over a day that doesn't look like the hourly flow pattern from detectors that are upstream and downstream in the same direction. If a pattern is found for a particular detector that more closely matches the detectors on the opposite side of the freeway then a low score is assigned. The details of assigning a score to one particular "target" detector are as follows:

The algorithm is run on a weekday when the flow profiles have strong AM and PM peak direction patterns. Only data during the hours of 5am until 10pm is used (at 3am the flow is similar everywhere). For the target detector all of its neighboring detectors on both sides of the freeway are selected for 5 miles in each direction. Only detectors diagnosed as GOOD are used. For the purposes of this calculation, in order for a detector to be GOOD we require that

1. More than 75% of the individual lane detectors have to be judged good by the daily diagnostic routines
2. None of the 5-minute flows for any of the lanes during the time period were purely imputed, and
3. At least 80% of the resulting hourly aggregate data points are good (not imputed).

For each neighbor that has been judged to be good the distance between the target and the neighbor is determined. For distance the mean of the absolute differences between the normalized hourly flow patterns of the two detectors for the day is used. Note that this is done on the aggregate flow across all lanes and that this test is only performed for mainline (ML) detectors.

For the top 10 closest neighbors, as computed by the distance in the preceding step, we count the number of detectors that are in the same direction and we call that K. The resulting score for this detector is just K / 10. If the top 10 closest neighbors are all on the other side of the freeway then the resulting score will be zero. This is then a pretty good indication that the detector is switched.

If we don't have 10 neighbors with good data then we use whatever we have. In some cases, we might not have any neighbors with good data. In that case we don't compute a score at all. Currently this direction switching algorithm in PeMS is done every week on Tuesday night. If the data feed is down for that particular day, then it's possible that we won't be able to compute any scores for any detectors in the system.

[kengodleskidot]

The GIS data mentioned in #233 could offer a solution to handling this configuration diagnostic issue using geospatial tools.

[mmmiah]

@kengodleskidot @britt-allen @thehanggit The current PeMS station locations are mostly accurate, however, some of the stations Geo-location are incorrectly configured. Some of the examples are below-

a. Although stations labeled properly but its geometry is far way from its defined location as shown below for D10 ML 580 freeway- b. In some cases, although it should be dual alignment of the road, however, State Highway system used single poly line geometry to visualize both direction of road which makes complexity to use spatial analysis to diagnosis the direction. One of the example is shown below for SH 132 at D10 ML-

c. Some of the lat and long are incorrectly configured, example, D4 ML Freeway 80

d. PeMS stations label or lat/long was not updated for New Alignment of Highway, Example D10 ML 99 e. Big Deviation of Station location, example Freeway 108 ML D10 as shown below-

d. Either wrong location or wrong label, example, D3 ML Freeway 89 e. Overall, at least 727 PeMS station geo-location needs to be fixed as shown below-

based on my Observation, It is important to request each individual district to collect the lat and long for each station. I am going to prepare the inconsistency list that needs to be corrected

[kengodleskidot]

Great work so far @mmmiah! Let me know if you want to discuss what the list contains so we can hand it off to district staff so they can update their station configuration data as needed. I will also bring this up in my next meeting with Iteris to make sure this is not something caused by any changes/automations they are performing in PeMS.

[mmmiah]

Based on further delve of the data. Here is the summary @kengodleskidot @ZhenyuZhu-Caltrans @britt-allen @thehanggit

1. 727 Stations lat and long are highly incorrect list of PeMS Station that needs lat and long corrections.csv
2. Another 1403 stations need to manual edit its location to match the directions and route properly. Here is the list of those 1403 stations- incorrect lat and long to match the direction and route properly.csv
3. Remaining 20525 stations lat,long, direction and route are correct. Here are those list- Good Quality PeMS stations list.csv

[mmmiah]

@kengodleskidot , I found that we have state postmile/absolute postmile for each PeMS station, on the other hand we have beginning and ending postmile in the SHS file (https://gisdata-caltrans.opendata.arcgis.com/datasets/77f2d7ba94e040a78bfbe36feb6279da/explore). I think that for SHS system postmile begin from county boundary lines. Do you know how PeMS defined its state postmile/State Absolute postmile???

[mmmiah]

@kengodleskidot ,Visualized the error in online map. Here is the link

[mmmiah]

Closed as completed

[kengodleskidot]

@kengodleskidot , I found that we have state postmile/absolute postmile for each PeMS station, on the other hand we have beginning and ending postmile in the SHS file (https://gisdata-caltrans.opendata.arcgis.com/datasets/77f2d7ba94e040a78bfbe36feb6279da/explore). I think that for SHS system postmile begin from county boundary lines. Do you know how PeMS defined its state postmile/State Absolute postmile???

@mmmiah my understanding is that the postmile data in PeMS is brought in with the configuration file data provided by districts. I will share this information with district staff to see the best way to get updates made.

[mmmiah]

@kengodleskidot, Updated the Analysis just for ML and HV with Active stations. Here is the summary-

1. Out 12287 PeMS stations, 11,500 PeMS stations locations are correct which located within 50ft distance from SHS and matched the route and direction.

Good Quality PeMS stations list.csv

3. 414 Stations that is within 50 ft have direction and route mismatch, more breakdown of this 414 stations are : 39 stations did not match with direction and route, 364 direction mismatch only while route is matched, 11 stations route mismatch while direction is right

incorrect lat and long to match the direction and route properly.csv

4. Another 337 stations are beyond 50 ft threshold from SHS, needs immediate attention

list of PeMS Station that needs lat and long corrections.csv

6. All good and bad data compiled in the below file with error status all_PeMS_Station_data.csv

Here is the updated map link