Create Testing Scenarios and Inputs

[colinsheppard]

GanttStart: 2017-08-14 GanttDue: 2017-08-22

Create a simple scenario for testing. This will consist of a simple 5x5 Manhattan grid for the roads with 2 public transit agencies (a bus and a rail). In R5 terms, let's assume the rail is the SUBWAY mode. Here's a graphic of the network with ID labeling defined. The left shows the roads (which we label with L for Link and N for Node) and the right shows the transit lines (S for station, R for Rail, B for Bus).

We need to specify this road network in OSM format, let's assume each link is 1 km in length. More specs to come for other attributes of the OSM ways. Let's assume the lat/lon coordinate for N20 is (0,0) and that each link is 0.01 degrees lat in length, so the coordinate for N5 would be (0.05, 0.05). This makes each link ~1117 m in length.

The transit will be specified using GTFS standard. For a relatively simple example of a GTFS spec, see the Caltrain data: https://transitfeeds.com/p/caltrain/122

[colinsheppard]

FYI - after seeing the naming scheme used in the matsim example transit network, I'm definitely open to following that same scheme:

https://github.com/LBNL-UCB-STI/beam/commit/fa3b630ad736ea34f321632136788291cfb5f802

I do however, think we should continue with the slightly more complicated network topology specified above to enable the ability to test multimodal routes (i.e. walk to bus to train to bus, etc.).

[colinsheppard]

Note, see Andrew's preliminary work creating the roads for this test case:

https://github.com/LBNL-UCB-STI/beam/issues/53#issuecomment-322354506

[neverforgit]

I somehow missed this thread. FYI, the link and node IDs in the spec above won't work for OSM. Although OSM files do not have an official DTD schema defined, the API's DTD seems to be the default. It requires integers for IDs.

https://wiki.openstreetmap.org/wiki/API_v0.6/DTD

In the file I shared #53 (comment), I set the coordinates of the center of the map to be the UC Berkeley Campanile. Each block is about 100 meters.

@colinsheppard in the spec you suggested, I guess you want the lower left node to be at (0,0). I can redo my test network with that spec and the longer block length.s

[colinsheppard]

Thanks Andrew. After you adapt to this spec let me know and I'll correct the labeling.

[neverforgit]

Here is the latest. Updates:

1) Lower left corner is (0,0) 2) Blocks are 0.01 degrees long 3) Reverse direction ways for the same block no longer overlap -instead of single nodes for intersections, we have 4 nodes. -a single block is represented by 2 reverse-directed ways 4) Streets are now named (row_0, row_1... col_0, col_1...)

Here is the download link: https://drive.google.com/open?id=0B5DYEo0XCyF6ajEwdWcxbHlpaU0

[colinsheppard]

Thanks @neverforgit. Is there any utility (for flexibility in testing) in turning some of these into higher capacity links? E.g. rows 2 and 4 could be 4 or 6 lanes...

@fdariasm since the road network is nearing completion, please focus on producing the GTFS data for the 2 transit agencies.

[neverforgit]

@colinsheppard it's a bit of Rumsfeldian dilemma; I don't know what I don't know about what is critical for testing. There are scores of other OSM elements, keys and attributes I could experiment with. A few I have thought of:

1 - Have 2-directional non-separated links. ("oneway"="no") 2 - Add non-motorized ways (e.g. foot paths) 3 - Incorporate other modes (sidewalks, cyclepaths) 4 - Include relations. Relations are ordered sets of ways. They are used for things like local routes and highways (e.g. defining Route 66, or defining bus routes).

There's a combinatorial limit to what we can test. I'm just not sure what is critical. From my experience, I item (1) is probably the most relevant. It certainly caused me some problems when generating the MATSim network from R5.

[neverforgit]

@fdariasm do you already have a branch going for working on this? I wrote the code to generate the OSM street network under the router-r5-extractor branch, but this really issue belongs under its own branch I guess.

[neverforgit]

@fdariasm I cut a new branch router-test-network. The class to generate the street network is here:

https://github.com/LBNL-UCB-STI/beam/tree/router-test-network/src/main/java/beam/utils/testNetwork

[neverforgit]

@colinsheppard, I am going to add some "oneway"="no" links and change the lanes and speeds a bit. Then I will submit a pull request back to master.

[sfwatergit]

Of interest may be this paper: https://svn.vsp.tu-berlin.de/repos/public-svn/publications/vspwp/2016/16-02/KickhoeferEtAl2016MatsimSantiago.pdf, which describes a recent implementation of an OSM->MATSim network conversion that has some rationale regarding what to keep and what to discard. You might also look at how the JOSM extension is tested: https://github.com/matsim-org/josm-matsim-plugin/tree/master/src/test/java/org/matsim/contrib/josm as well as OSM readers elsewhere: https://github.com/matsim-org/matsim/blob/10d7cd179f3c23a8c1a978a9e8d18fc0775fa7fd/matsim/src/test/java/org/matsim/core/utils/io/OsmNetworkReaderTest.java

On Thu, Aug 17, 2017 at 3:00 PM, Andrew notifications@github.com wrote:

@colinsheppard https://github.com/colinsheppard, I am going to add some "oneway"="no" links and change the lanes and speeds a bit. Then I will submit a pull request back to master.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/LBNL-UCB-STI/beam/issues/60#issuecomment-323206309, or mute the thread https://github.com/notifications/unsubscribe-auth/ABiGtZUGhnZNMZvJ_aRug20fu9uiAVW5ks5sZLgGgaJpZM4O09iY .

[fdariasm]

GTFS data from issue description: https://www.dropbox.com/s/zhhjifwr2gx0y2f/gtfs.zip?dl=0

Some considerations:

• zone_id = 1 given it's the same zone

stop_time.txt:

• All bus and rail transportations starts at 6:00, and it's only one travel route.
• Bus has an approximately duration of 20 min for station change and waits 5 min in each stop.
• Rail has an approximately duration of 10 min for station change and waits 10 min in each stop.

@colinsheppard data was verified with https://github.com/google/transitfeed/wiki/FeedValidator, and OTP. Please check and let me know if any change is needed.

[sfwatergit]

Might be useful to create the rail network along the diagonal of the grid so that it's more clear whether rail and bus/car use separate links.

On Aug 18, 2017 8:28 AM, "fdariasm" notifications@github.com wrote:

GTFS data from issue description: https://www.dropbox.com/s/ zhhjifwr2gx0y2f/gtfs.zip?dl=0

Some considerations:

• zone_id = 1 given it's the same zone

stop_time.txt:

• All bus and rail transportations starts at 6:00, and it's only one travel route.
• Bus has an approximately duration of 20 min for station change and waits 5 min in each stop.
• Rail has an approximately duration of 10 min for station change and waits 10 min in each stop.

@colinsheppard https://github.com/colinsheppard data was verified with https://github.com/google/transitfeed/wiki/FeedValidator, and OTP. Please check and let me know if any change is needed.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/LBNL-UCB-STI/beam/issues/60#issuecomment-323384901, or mute the thread https://github.com/notifications/unsubscribe-auth/ABiGtSZpEmnLKTLTWKOHjFc7ocXEIbD9ks5sZa2VgaJpZM4O09iY .

[colinsheppard]

@fdariasm

See updated schematic below. Let's make the following adjustments.

In the OSM file that Andrew provided, please snap all coordinates to no more than 5 decimals, e.g. 0.029949999999999997 should be 0.02995.

Please turn row_2 into a highway with intersection only at col_1 and col_5. Give it a maxspeed of 65 mph. Note that you will need to remove the existing intersections between row_2 and cols 2-4 and reduce what was 2 ways to one. (E.g. col_2 was 4 segments long but now it will only be 3 segments with the first one being twice as long as the rest).

In the GTFS data, please break up the TRAIN and BUS into 2 separate agencies, so there will be two sets of GTFS files in total. Note the slight renaming of the stops in the figure below.

It would only take a bus moving ~30 mph about 3 minutes to travel the 2.2km between stops. So let's make trip time (i.e. from departure to arrival) be 3 minutes and then make the next departure be 2 minutes later. Repeat the pattern every 10 minutes, i.e. so B1 will start an eastbound trip, let's call it B1-EAST-1 at 6:00 and it should reach S3 at 6:08 and be done. But then a new trip on route B1 should departure from S1 at 6:10, call it B1-EAST-2. Repeat this 6 times to cover the hour and also make as many WEST trips.

Finally, please create a feature branch off of agentsim, e.g. agentsim-test-data and commit to the directory test/input/beam/beamville/r5/ (you'll have to create the last two sub-directories). Inside here put the GTFS directories and the OSM network.

[colinsheppard]

Also, let's assume the train is a 4 minute trip from R1 to R2 and the train departs every 15 minutes in both directions.

[fdariasm]

@colinsheppard, do you mean highway intersection would be at col_1 and col_5? also, I understand cols 2-4 will have three segments, but what do you mean by reducing what was 2 ways to one?

[colinsheppard]

before there was one way connecting row 1 to row 2 via col 2 and a second way connecting row 2 to row 3 via col 2.... but after removing the intersection there will be only one way connecting row 1 to row 3 via col2.

[colinsheppard]

imagine that those roads now have a bypass that goes over the freeway with no more intersection or on/off ramps.

[colinsheppard]

@fdariasm

A loose end you could work on (feel free to talk with @neverforgit if you need help), is to create the highway. This you can do directly in model-inputs/beamville/r5/beamville.osm without worrying about changing data that others need for tests... this is because the model doesn't directly use the .osm file, I'll handle testing and integrating the result when I return late next week.

[neverforgit]

@fdariasm, have you worked on adding the highway link?

[fdariasm]

@neverforgit, just resuming taks right now (long weekend here, sorry). I'm going to start working on it

[fdariasm]

@colinsheppard, just updated osm with highway. Sorry for the delay, I was figuring out how to use JOSM to modify file. Is Rail already added?

[neverforgit]

@fdariasm the new highway looks good, but I think there is a little confusion in our naming syntax. In my original network, the rows are named row_0:row_4, counting upwards. In Colin's example picture, the names are row_1:row_5, increasing downwards from the top row.

So, per Colin's previous comment, row_2 is now a highway. But it now overlaps the middle bus route. I guess this is problematic for the middle bus stops.

[colinsheppard]

Sorry for the confusion. Please make Row1 per Andrew's convention be the highway I will update the schematic later this week.

[colinsheppard]

And just to avoid further confusion. The intension was that the highway would be in the row between the rows with the bus routes.

It might be interesting at some point (not right now) to move a bus through the highway, but if we want that, we'll add a new bus route.

[fdariasm]

Ok got it, I'll make changes.

[fdariasm]

Just to confirm, in osm file per @neverforgit convention row_0 is bottom row while @colinsheppard's picture row_1 is top one. Should I change OSM file to adapt to picture or modify row_3 in OSM file? in osm file rows start from bottom to top but picture is the other way around.

[colinsheppard]

You can choose, but please let us know with another comment which convention so I can update the diagram if necessary.

[fdariasm]

I kept Andrew's convention but modified row_3, second from top to bottom. So first row starts on index 0 and is bottom row increasing to top. File is updated in google drive folder.

[colinsheppard]

Thanks to all who helped with this. They're working great.

[neverforgit]

R5 has a problem processing the test network with the freeway. It skips the 'motorway' ways and the street ways that cross it. It must be due to the fact that the network has been modified in josm.

Since OSM wants keep a wikipedia-like canonical set of edits, with the option to rollback, the modified links and nodes are represented differently: 1) they have negative ids, 2) the use the action='modify' tag.

This is the MATSim network, created from the R5 network, created from beamville.osm:

[neverforgit]

I took my original test network, with only city streets, and added a big diagonal freeway. In the xml, those two ways look like all the others, only the some of the tags have been changed ('motorway' instead of 'trunk', 65 mph, and 2-lanes).

This time the R5 and MATSim networks have all the links.

[colinsheppard]

Ok, so is the final solution to manually edit the OSM instead of using JOSM? If so, can you do that to add the freeway the way we originally planned?

[neverforgit]

I'm not sure. Are these the same conventions we see in the real OSM data? If so, R5 should be handling them w/out issue. Or is this just a JOSM thing?

[colinsheppard]

Don't know, but I can bring this up with Conveyal.

Just so I'm clear though, when we added the highway, it was saved to OSM with a negative id for the way and a 'motorway' type with action "modified" tag and the result is a broken network? Is it clear that the network inside of R5 is missing the highway and it's not somehow the extractor you wrote?

[neverforgit]

Ok, this seems to be the convention for submitting changes to the canonical OSM servers. The beamville.osm file looks like what you would submit (like a pull request) to the OSM servers. The negative IDs are placeholders that the server then swaps for the canonical IDs. The action attributes tell it to replace existing links and nodes.

Maybe there is a quick way to "accept" the changes in JOSM so that the new freeway looks like it would once the OSM servers have processed it? @fdariasm

[neverforgit]

@colinsheppard to answer your question, yes. I am logging the value of the 'highway' tag for every way in the R5 network. There are no 'motorway' values. When I run it on my version, with the diagonal freeway, we do see the two 'motorway' values.

I am pretty sure that the problem is that beamville.osm does not look like an actual canonical data file as provided by the OSM servers. It looks like a pull request to the servers for a map edit.

[colinsheppard]

Ok, yeah, let's see if we can figure out how to get JSOM to "commit" the change or whatever needs to happen before we bring up with R5 folks.

[neverforgit]

Ok, I'd like to focus on building the MATSim network. I can use my version w/ the diagonal for now. Maybe @fdariasm can find a quick solution since he has used JOSM before.

[fdariasm]

The only option I see on JOSM is commiting file to OpenStreet servers, already tried but got an error. Will check tomorrow and let you know

[fdariasm]

@neverforgit, can you please provide me steps to generate R5 and Matsim networks out of OSM file? also the OSM file you created with diagonal freeway, I'm running some tests for this. Thanks

[neverforgit]

I shared the OSM file w/ the diagonal freeway with you in Glip.

Generating the R5 files is very simple. Under the router-r5-extractor branch, you can see my class beam.agentsim.r5Extractor.MakeR5Net

You just point R5 at the directory w/ the OSM file:
TransportNetwork tNet = TransportNetwork.fromDirectory(inputDir);

If the mapdb and network.dat files do not already exist, it will find the *.osm.pbf file and build everything from that.

NOTE: you need to convert that OSM file I shared in Glip to a pbf using Osmosis.

[fdariasm]

@neverforgit I followed this process: created input folder with map.osm and map.pbf files, run r5Main pointing to this input folder to generate network.dat file. Then, run r5Main with --graphs /path/to/input/files and access server on http://localhost:8080/debug.html. These are what I can see on both your beamville.osm file and the one that's on repository with freeway as described here:

Your file:

Beamville.osm with freeway:

Can you please confirm this is not the expected output?

[colinsheppard]

@fdariasm just to be clear, please do work toward making the freeway be as originally specified (across the second row from the top instead of diagonal).

also, can you please update the bus GTFS data so that the buses stop at every intersection? it turns out that multi-stop trips was causing issues that we weren't adequately testing with Beamville. Thanks!

[neverforgit]

Weird. I can think of two reasons the modified links don't make it into the MATSim network

1) When I iterate over the all the edges, EdgeStore.Edge.advance(), I never see the modified links. This is what I seem to remember.

2) The modified links are not added to the mapdb data store.

Can you select individual links and see their attributes? Do the modified links look correct in your file?

[fdariasm]

@colinsheppard, gtfs data updated, please check PR https://github.com/LBNL-UCB-STI/beam/pull/102.

[colinsheppard]

Thank you @neverforgit and @fdariasm for your work on this!

There is more work to be done on Beamville, but this will all get specified in new issues.