Import unsignalized intersections

[GoogleCodeExporter]

instead of using the default...actually look up unsignalized intersections from 
city's GIS file.

Original issue reported on code.google.com by elizab...@sfcta.org on 25 Feb 2012 at 3:09

[GoogleCodeExporter]

Thoughts from 2/27 Checkin:

There are two basic methodologies we could implement:
(1) Easist: match 4-way stops based on closest node (and employing various 
other smart things like making sure its not a virtual node, looking at number 
of incoming links, etc); and then infer 2 way stops for all intersections based 
on a hierarchy of functional types, lanes, etc.

(2) Hard: match GIS file based on names 

Resolution: lets try #1 first to get something working sooner rather than later.

Original comment by elizab...@sfcta.org on 28 Feb 2012 at 1:52

[GoogleCodeExporter]

Original comment by elizab...@sfcta.org on 8 Mar 2012 at 1:59

• Changed state: Accepted

[GoogleCodeExporter]

No work was done on this topic the week of 3/5/2012.  

Original comment by elizab...@sfcta.org on 12 Mar 2012 at 4:21

[GoogleCodeExporter]

Transferring this to Renee

Original comment by elizab...@sfcta.org on 16 Apr 2012 at 4:00

[GoogleCodeExporter]

all-way stop sign file: Q:\GIS\Road\StopSigns\allway_stops.shp

We also have Q:\GIS\Road\StopSigns\stops_signs.shp
which looks like it tells which direction the stop sign faces; it has STREET, 
X_STREET, and ST_FACING (NB,SB,etc)

Original comment by lisa.zorn@sfcta.org on 12 Jun 2012 at 12:58

[GoogleCodeExporter]

Re-classifying as high priority.  This will be one of the next items on Renee's 
list.

Original comment by greg.erh...@ucl.ac.uk on 12 Jun 2012 at 3:59

• Added labels: Priority-High
• Removed labels: Priority-Low

[GoogleCodeExporter]

Ignore the last push I made for this.  I need to make some other changes to the 
code to make this work well.  I think I'll need to create a class of "Stop 
Nodes" that can then match to Road Nodes.  It shouldn't take long to get that 
fix done tomorrow.

Original comment by alsu...@pbworld.com on 13 Jun 2012 at 9:34

[GoogleCodeExporter]

I haven't read over the code in detail but I'm curious what problem you're 
having with it?

Original comment by lisa.zorn@sfcta.org on 13 Jun 2012 at 9:37

[GoogleCodeExporter]

For some reason it keeps trying to write out the stop nodes being read in as 
part of the Dynameq network.  It then has a problem because I've changed their 
labels to be equal to the intersection name.  I'll look at this more closely 
tomorrow.

Original comment by alsu...@pbworld.com on 13 Jun 2012 at 9:50

[GoogleCodeExporter]

Figured out what was going wrong here.  I was setting the priority for some of 
the stops as a string instead of a number.  Fixed it, and now it finds 3767 
nodes with stop signs and updates their priorities.  The next two steps on this 
will be (1) make sure that node doesn't have a signal plan and (2) check 
two-way stops to see if both roads have the same functional type, and set the 
priorities if they do.  Rev 093c8f7f36f9 has the preliminary version of this 
import code, and I'll push the updates that check for signals and conflicts 
later today. 

Original comment by alsu...@pbworld.com on 14 Jun 2012 at 1:50

[GoogleCodeExporter]

It looks like a lot of the two-way stops where both links have facility type 7, 
so Dynameq won't be able to handle these right now.  I'm going to have to use 
the shapefile to select the link that gets priority and then create a custom 
priority file to set those priorities.  For now, we can use the four-way 
intersections, but the two-way ones will take some time before I can get the 
priority file set up.

Original comment by alsu...@pbworld.com on 14 Jun 2012 at 2:57

[GoogleCodeExporter]

In that state that it's in now, the stop_signs.shp file has a lot of 
intersections that are actually four-way, but only three of the stop signs are 
included in the shapefile.  These are also intersections that are not included 
in the allway_stops.shp file.  I'm wondering if, at least for now, it would be 
ok to just change the code so that if the number of stop signs is more than 
half of the number of incoming links, the node is assigned as an all-way stop.  
With more time, I can look at these each individually and add them to the 
shapefile, but as a quick fix, if you guys are ok with it, I think making these 
ones (that have 3 of 4 incoming links tagged with stop signs) all-way stops 
might be a good solution.

Original comment by alsu...@pbworld.com on 14 Jun 2012 at 8:59

[GoogleCodeExporter]

Regarding your last two comments, I agree that it is appropriate to simplify.  
There are thousands of intersections with stop signs in our network, so it 
could take a while if we set out trying to get each one right.  

We should keep in mind that there are two important things in putting stop 
signs in the network:

1. Slow down the lower-class facilities so people can't zip through on local 
streets and avoid the arterials
2. Make sure we don't put a stop sign in the middle of a big congestion street 
because doing so might result in gridlock.  

As for the rest of them, if they are on low class facilities (facility type 7 
is local) those streets shouldn't have a lot of volume, so the specifics won't 
matter much as long as we slow the locals down enough that they don't get 
overloaded.  I think you're on the right track, but I would recommend the 
process as:

1. Read the allway_stops layer and set those as all-way stops.  
2. Read the stop_signs layer, and identify any all-way stops based on the 
number of approaches.  (As we looked at, the stop_signs layer includes a 
mish-mash of different stuff, some all-way, some two-way, and some that seem to 
have data missing.)  
3. Set the remaining stops in the stop_signs layer as two-way stops, and let 
Dynameq pick the approaches to put the stop-sign on as the lower-class 
approaches.
4. For those that Dynameq can't pick--where two FT 7 links cross--just treat 
them as all-way stops for now.

Then, we run an assignment, look at the results, and decide what to do next.  
#4 is obviously not right, but I am more interested in getting another 
assignment out to see if the stop signs we do have push more traffic onto the 
arterials.  

Original comment by greg.erh...@ucl.ac.uk on 15 Jun 2012 at 6:04

[GoogleCodeExporter]

I've updated the code with the changes that Greg suggested.  I also added a 
check so that it wouldn't update a node if it was already designated as 
signal-controlled.  This took away about 30 of the nodes that it was originally 
updating, and when I looked at the intersection names, I saw that the new ones 
that don't find a match now really are signalized intersections.  I'm not sure 
why they're included in these shapefiles if they are signalized, but that's an 
error that we can look at later.  For now, the check should be enough.  I'll 
keep working to get the stops that don't match to a node right now to do so, 
but the version of the code in rev 1379e487c650  should be good enough to use 
for the next round of Dynameq testing as-is (with two-way stops with facility 
type conflicts set to all-way).  It must be run AFTER the import excel signals 
code, though, so that it can check to see if the node is signalized.  

Original comment by alsu...@pbworld.com on 15 Jun 2012 at 12:57

[GoogleCodeExporter]

I've been looking at the output more clearly, and I'm finding an error that I'm 
not sure how we should handle.  It looks like on quite a few of the stop signs 
in stop_signs have coordinates and street names that do not match.  The 
coordinates match with a different intersection than is indicated by the STREET 
and X_STREET fields.  I'm wondering which of those we trust more, becuase if we 
trust the street names, I'll need to either change those or check in the code 
if the street names match, and not use those coordinates if they don't.  If 
anyone knows which measure is better/more reliable in those files, I'll change 
the code to use that one.

Original comment by alsu...@pbworld.com on 15 Jun 2012 at 6:12

[GoogleCodeExporter]

Renee - is there an example intersection that you can give us so that we can 
follow it through and see what it is supposed to be doing?

Original comment by elizab...@sfcta.org on 15 Jun 2012 at 6:28

[GoogleCodeExporter]

There is a stop that is located at the Madera St & Wisconsin St intersection in 
the shape layer, but the street names are Felton St & Madison St. Similarly, an 
intersection located at 20th & Wisconsin has street names 23rd & Wisconsin.  
What makes this problem even more confusing is that there are other stop signs 
at 20th & Wisconsin that have street names 20th & Wisconsin, and there are also 
stops at 23rd & Wisconsin that have the right street names. It's just the one 
node that was either wrongly placed or wrongly labeled.  This isn't just 
happening on Wisconsin either.  Those were just the first two examples that 
were up on my screen.

Original comment by alsu...@pbworld.com on 15 Jun 2012 at 6:35

[GoogleCodeExporter]

One quick note I forgot to include last week: I had to add an ID field to the 
stops_signs shapefile because there was previously no field that it could read 
in as and Id.  I also had to delete the CNN field because it was causing errors 
in reading in the shapefile.  I moved that updated version of the shapefile 
into \\files\GIS\GIS\Road\StopSigns this morning, so that one should work with 
the current version of the code.

Original comment by alsu...@pbworld.com on 18 Jun 2012 at 12:41

[GoogleCodeExporter]

Hi Renee --
Q:\GIS (aka \\Files\GIS) is our agency GIS data repository and so it would be 
preferable if you don't delete fields from a dataset.  The CNN field might be 
useful to someone using this GIS data file for another purpose, where CNN is 
meaningful.

Similarly, I don't think adding meaningless fields is a good idea; the ID field 
really doesn't actually mean anything here, so I wouldn't add it.

Finally, if you *must* manipulate the GIS datasets in a non-trivial way, using 
git or something to revision the previous versions is a good idea.  (For 
example, Q:\GIS\Road\SFCLINES\AttachToCube is a git repository since we make 
enough changes to it that it's good to see why we changed things and also to be 
able to revert, if our changes breaks working code.)

That said, this morning, Chris gave us a new stop sign shape file.  Looking at 
it in GIS, this shapefile includes all-way stops and individual stops, and it's 
much more recent than the other files we previously used (which look to be 2005 
and 2008).  So it makes sense to just use this file.  I rearranged things as 
follows:

Q:\GIS\Road\StopSigns\2012 is a shortcut to 
Q:\GIS\CityGIS\TrafficControl\StopSigns, containing the shapefile 
stops_signs.shp that Chris sent this morning.  I made a single modification, 
which is to put "0" into a blank CNN field in the dbf, documented in the 
readme.txt.

Q:\GIS\Road\StopSigns\2008\stops_signs.shp contains the older (2008) version of 
this file, which I reverted to it's original 2008 version (with CNN, without ID)

Q:\GIS\Road\StopSigns\2005 contains allway_stops.shp, which we don't need to 
use any more

Original comment by lisa.zorn@sfcta.org on 18 Jun 2012 at 10:56

[GoogleCodeExporter]

I did keep a copy of the old *.dbf file with the CNN field included, but that 
field can't be read correctly by the shapefile python code, so it crashed it 
every time until I deleted it.  I checked the new file, though, and that CNN 
field doesn't have any problems, so it's still there.  The reason I added the 
Id field was for debugging so that I could have a number to reference which 
stop signs were having what problems.  Again, I kept a copy of the *.dbf file 
without the Id field and just added _old to the name so that it could still be 
used.   

Instead of changing those files, it might work better if I just added the Id 
field to the copy on my computer so that I can debug without changing the files 
on the server.  Are you sure that all of the allway_stops.shp stops are in the 
stops_signs.shp file?  There were about 5 that weren't in the last one, which 
was why I set up the code to read both files.

Original comment by alsu...@pbworld.com on 18 Jun 2012 at 11:10

[GoogleCodeExporter]

Lisa- Thanks for the guidance.  We got stuck on the shapefile.py error you get 
when there is a blank in a numerical field.  Converting the blank to a zero 
does the trick too.  

Regarding the IDs, it doesn't completely help if it works on a local copy for 
Renee, but not on the network (which is why I asked her to upload the modified 
files, but your point regarding version control is valid).  Maybe the thing to 
do is to make it an optional field in the script, so you can use it locally for 
debugging if needed, but the script will still run if it's not there.  Either 
that, or we just print the street names instead of an ID.  The important thing 
being that it runs with available inputs. 

Original comment by greg.erh...@ucl.ac.uk on 18 Jun 2012 at 11:49

[GoogleCodeExporter]

I didn't see the old dbf file -- in which case, nice, you already read my mind. 
 :)

I am updating the script now for the new file.  I actually was noticing that 
the CNN is useful because it's unique per intersection, and looping through all 
the roadnodes to find the appropriate intersection is slow enough that it helps 
to do it once per unique intersection.

I will check in my updates reasonably soon.

Original comment by lisa.zorn@sfcta.org on 19 Jun 2012 at 12:59

[GoogleCodeExporter]

Ok, I ran out of time to truly finish this but I just checked in some updates, 
primarily in Rev 84a7194b453f.  I streamlined the code a lot so it's much 
shorter.  We can talk about the differences.  There are still some todos: I 
don't check that the streetnames from the shapefile match the streetnames from 
the network, but the debug file indicates that things are pretty good.  Also 
the two-way issue, more on that below.

Ran the full network creation in 
X:\Projects\ModelDev\dtaAnyway\validation2010.3\input
Summary is
Read 3814 stop-sign intersections
  10  : Failed to map
  31  : Ignored because they're signalized
  1830: Setting as allway-stop (including 121 questionable, with more stop signs than incoming links)
  1943: Setting as twoway-stop

Dynameq handles the twoway stop signs with matching facility types by ignoring 
them, and reporting them as no priorities set.  I think the correct way to 
handle this is to make the custom priority file, which explicitly gives 
priority to the incoming links without the stop sign.  Should be do-able using 
the streetnames/direction info in the stop file, so that's the other TODO here.

Original comment by lisa.zorn@sfcta.org on 19 Jun 2012 at 2:11

[GoogleCodeExporter]

Will mark as all-way stops until custom priorities are in.
Also report on intersections that are not controlled.

Original comment by lisa.zorn@sfcta.org on 5 Jul 2012 at 6:32

[GoogleCodeExporter]

In Revision 73f5ed8d90fe : for two way stops with ambiguity (more than one 
through movement with matching incoming link facility types), the code now 
falls back to all-way stops.

There is a slight discrepancy; importUnsignalizedIntersections.DEBUG.log 
reports:
2012-07-05 16:58 INFO Read 3839 stop-sign intersections
2012-07-05 16:58 INFO   10  : Failed to map
2012-07-05 16:58 INFO   27  : Ignored because they're signalized
2012-07-05 16:58 INFO   1828: Setting as allway-stop (including 120 
questionable, with more stop signs than incoming links)
2012-07-05 16:58 INFO   689 : Setting as allway-stop in lieu of custom 
priorities
2012-07-05 16:58 INFO   1285: Setting as twoway-stop

But when I import it into Dynameq, Dynameq reports (via Scenario -> 
Properties..., Category=Network)
Intersection control
Signalized: 1096
Unsignalized: 10529

Priority Template
AWSC: 2504
TWSC: 1222
None: 7899

It's not huge but there are 63 two-way stops that Dynameq doesn't like that we 
don't expect, and 13 all-way stops.
Not sure why, since Dynameq doesn't warn about it or give us any other 
information...

Original comment by lisa.zorn@sfcta.org on 6 Jul 2012 at 12:13

[GoogleCodeExporter]

Original comment by alsu...@pbworld.com on 13 Aug 2012 at 5:00

• Changed state: Fixed