Create reproducible example for Edinburgh

[dabreegster]

For https://raw.githubusercontent.com/nptscot/npt/main/data-raw/od_subset.csv, what should the origins and destinations be? All buildings to... schools? Workplaces?

[Robinlovelace]

Homes + workplaces as per message on Slack. Happy to take a look at results and I need to generate test outputs..

[dabreegster]

How do you generate subpoints for each of those today? For homes, all buildings, or do you filter for anything? How about for workplaces?

There's a few bugs with the Edinburgh example I'm working through, will let you know when it's ready for a look. For reference, the routing step took about 2s for the 138k requests from the OD csv

[Robinlovelace]

Today we use these for origins and destinations: https://github.com/nptscot/npt/blob/main/data-raw/oas.geojson

[Robinlovelace]

For the open data build..

[dabreegster]

Alright, fixed.

git clone https://github.com/dabreegster/routing-engines
cd routing-engines/examples/edinburgh
python3 setup.py
cargo run --release config.json

Load output/output.geojson with https://dabreegster.github.io/routing-engines/, or check out output/counts.csv.

You can also get full detail for some routes: cargo run --release config.json --detailed-routes=50. Check output/route*.geojson.

Not counting the time for cargo build, the first run to generate counts from 140k routes (- 5k errors) takes 10s total for me. If you run it again (using some cached files in intermediate, it's just 5s.

[Robinlovelace]

For the full build we also use OA centroids for origins but switch to OS POIs for destinations in the 'full' build as per https://github.com/nptscot/npt/blob/main/_targets.R#L136

@mem48 may have more info on how open/not the destination dataset is but seems closed from that code. You could approximate it pretty well with workplaces on OSM.

[dabreegster]

The LTS categories are still broken, working on it.

[Robinlovelace]

These timings are impressive! As it happens Malcolm and I were just talking about build times with units of hours not seconds :laughing: :snail: :turtle:

[dabreegster]

For the full build we also use OA centroids for origins but switch to OS POIs for destinations in the 'full' build

Thanks! For now, both origins and destinations in this are just all building centroids. setup.py creates em. We can get fancier with tag filtering to detect workplaces.

[dabreegster]

I'm getting the total of 140k requests from the all column in od_subset.csv. Do you have anything bigger? ;)

[Robinlovelace]

If you share the .geojson version we may be able to do a quick benchmark. Heads-up @juanfonsecaLS1 who is looking at the validation side of things.

[Robinlovelace]

I'm getting the total of 140k requests from the all column in od_subset.csv. Do you have anything bigger? ;)

I do but may take some time to generate.

[dabreegster]

Outputs at https://www.dropbox.com/scl/fi/rlvyw21q9y9xp7afxhoii/edinburgh_rnet.geojson?rlkey=iwg1e6frgerz5j95y9v5eyww1&dl=0 and https://www.dropbox.com/scl/fi/m1o5s5xchh5byl8elnuj4/edinburgh_counts.csv?rlkey=i5e4y2h9648q8h0fqg8op2u5v&dl=0

[dabreegster]

Oh yeah and this is not using an uptake model, modify config.json for that

[Robinlovelace]

It's using bicycle column for now I guess.. will check.

[Robinlovelace]

No sign of which value in the OD data it's using for the n. routes: https://github.com/dabreegster/routing-engines/blob/main/examples/edinburgh/config.json

[Robinlovelace]

OK: all should be replaced with bicycle here if you want an estimate of cycling levels in Edinburgh today (or 2011 which wasn't too different although cycling has defo grown since then): https://github.com/dabreegster/routing-engines/blob/main/examples/edinburgh/setup.py#L52

[dabreegster]

See setup.py. It's using all, you could change it there depending what we want to do -- look at converting driver trips or estimating current cycling.

(I'm still working on docs, but I'm making some different choices about config vs convention here. Input files need to have a very well-defined format. Less config for things like columns. Need to write little data prep scripts anyway to download input files, might as well do cleaning there too.)

[Robinlovelace]

Looking good and glad it's a useful example. Yeah from looking at setup.py it's worth replacing all with bicycle to compare with cycle counters and with baseline results generated by NPT.

[temospena]

Back to this reproducible example... (is there any other? I was just trying to run something to make sure I have everything needed before proceed!)

I did this

cd examples/edinburgh
python3 setup.py
cargo run --release config.json

It made a lot of operations, but ended with an error.

    Finished release [optimized] target(s) in 0.04s
     Running `/media/rosa/Dados/GIS/od2net/target/release/od2net config.json`
Using config from config.json:
{
  "requests": {
    "description": "Existing home to school cycling trips in Edinburgh, from NPT data. The origins and destinations are both building centroids, since not every zone has a school.",
    "pattern": {
      "BetweenZones": {
        "zones_path": "zones.geojson",
        "csv_path": "od.csv"
      }
    },
    "origins_path": "buildings.geojson",
    "destinations_path": "buildings.geojson"
  },
  "cost": "Distance",
  "uptake": "GoDutchPCT",
  "lts": "BikeOttawa"
}

## everything
#### Load network
Trying to load network from /media/rosa/Dados/GIS/od2net/examples/edinburgh/intermediate/network.bin
#### Load network took 87.06607ms
#### Loading or generating requests
###### Loading origins
Loading points from /media/rosa/Dados/GIS/od2net/examples/edinburgh/input/buildings.geojson
###### Loading origins took 198.550525ms
###### Loading destinations
Loading points from /media/rosa/Dados/GIS/od2net/examples/edinburgh/input/buildings.geojson
###### Loading destinations took 197.017365ms
Got 207,989 origins and 207,989 destination
###### Loading zones from /media/rosa/Dados/GIS/od2net/examples/edinburgh/input/zones.geojson
###### Loading zones from /media/rosa/Dados/GIS/od2net/examples/edinburgh/input/zones.geojson took 775.214µs
###### Matching points to zones
0 zones have no matching origin points. Using centroid instead.
0 zones have no matching destination points. Using centroid instead.
###### Matching points to zones took 67.494409ms
###### Generating requests from /media/rosa/Dados/GIS/od2net/examples/edinburgh/input/od.csv
###### Generating requests from /media/rosa/Dados/GIS/od2net/examples/edinburgh/input/od.csv took 1.010092ms
Got 8,015 requests
#### Loading or generating requests took 464.92421ms
#### Routing
Trying to load CH from /media/rosa/Dados/GIS/od2net/examples/edinburgh/intermediate/ch.bin
###### Building RTree for matching request points to OSM nodes
###### Building RTree for matching request points to OSM nodes took 16.586681ms
Got counts for 56,408 edges
7,444 succeeded, and 571 failed
#### Routing took 70.970182ms
#### Writing output CSV
Skipped 0 edges (started/ended mid-edge)
#### Writing output CSV took 28.151276ms
#### Writing output GJ
Skipped 0 edges (started/ended mid-edge)
#### Writing output GJ took 133.135579ms
#### Converting to pmtiles for rendering
Running: "tippecanoe" "/media/rosa/Dados/GIS/od2net/examples/edinburgh/output/output.geojson" "-o" "/media/rosa/Dados/GIS/od2net/examples/edinburgh/output/rnet.pmtiles" "--force" "-l" "rnet" "-zg" "--drop-fraction-as-needed" "--extend-zooms-if-still-dropping" "--description" "{\"config\":{\"requests\":{\"description\":\"Existing home to school cycling trips in Edinburgh, from NPT data. The origins and destinations are both building centroids, since not every zone has a school.\",\"pattern\":{\"BetweenZones\":{\"zones_path\":\"zones.geojson\",\"csv_path\":\"od.csv\"}},\"origins_path\":\"buildings.geojson\",\"destinations_path\":\"buildings.geojson\"},\"cost\":\"Distance\",\"uptake\":\"GoDutchPCT\",\"lts\":\"BikeOttawa\"},\"num_origins\":7125,\"num_destinations\":6442,\"num_requests\":8015,\"num_succeeded_requests\":7444,\"num_failed_requests\":571,\"num_edges_with_count\":56408,\"routing_time_seconds\":0.07096822,\"total_meters_not_allowed\":2706.1546407716487,\"total_meters_lts1\":7060655.421423288,\"total_meters_lts2\":7214196.108788052,\"total_meters_lts3\":16797463.27973969,\"total_meters_lts4\":2840260.3781559914,\"total_time_seconds\":null,\"tippecanoe_time_seconds\":null}"
#### Converting to pmtiles for rendering took 71.908611ms
WARNING: Dropping timer during block everything. Probably crashing.
Error: No such file or directory (os error 2)

I am not sure what I am supposed to get, but there is actually a output.geogson file with 26MB in the output folder. I tried to load this file to od2net.org but nothing is shown. Any guidance on this will be very helpful :)

Next: Create an example for Lisbon! @partavogen already forked this repo and is trying to make that, but also facing some issues.

[dabreegster]

I think you're missing tippecanoe. https://github.com/Urban-Analytics-Technology-Platform/od2net/blob/main/docs/tutorial_examples.md#setup has all the prerequisites. https://github.com/felt/tippecanoe#installation or there's also instructions to run with docker

@partavogen, feel free to ask about any problems you hit

[temospena]

Thank you, that was it indeed! I missed that tutorial link. Now I'm ready :kite:

## everything took 38.60656751s

Summary:
      - Scrape OSM data took 1.870924485s
      - Split into edges took 399.138367ms
      - Building RTree for matching amenities to edges took 653.335348ms
      - Match amenities to closest edge took 47.117938ms
      - Calculate LTS for all edges took 200.903644ms
      - Calculate cost for all edges took 22.633144ms
    - Make Network from xml or pbf took 3.194178353s
    - Saving to /home/rosa/GIS/od2net/examples/edinburgh/intermediate/network.bin took 112.743049ms
  - Load network took 3.333523759s
    - Loading origins took 459.723639ms
    - Loading destinations took 443.483893ms
    - Loading zones from /home/rosa/GIS/od2net/examples/edinburgh/input/zones.geojson took 2.225474ms
    - Matching points to zones took 182.770747ms
    - Generating requests from /home/rosa/GIS/od2net/examples/edinburgh/input/od.csv took 2.949707ms
  - Loading or generating requests took 1.091331384s
    - Building InputGraph took 84.030618ms
    - Preparing the CH took 3.14475253s
    - Building RTree for matching request points to OSM nodes took 42.819263ms
  - Routing took 3.457530768s
  - Writing output CSV took 44.245049ms
  - Writing output GJ took 272.523907ms
  - Converting to pmtiles for rendering took 30.407325297s
- everything took 38.60656751s

Input: Existing home to school cycling trips in Edinburgh, from NPT data. The origins and destinations are both building centroids, since not every zone has a school.
- Origins: 7,125
- Destinations: 6,442
- Requests: 8,015
- Requests (succeeded): 7,444
- Requests (failed): 571
- Edges with a count: 56,180
- Total distance on not allowed roads: 3.0 km
- Total distance on LTS 1 roads: 7155.3 km
- Total distance on LTS 2 roads: 7199.3 km
- Total distance on LTS 3 roads: 16712.8 km
- Total distance on LTS 4 roads: 2842.6 km

[Robinlovelace]

Heads-up @dabreegster I'm giving this a spin now..

[Robinlovelace]

Still hitting this:

python3 setup.py 
Traceback (most recent call last):
  File "/home/robin/github/nptscot/od2net/examples/edinburgh/setup.py", line 83, in <module>
    run(["mkdir", "-p", "input"])
NameError: name 'run' is not defined

[dabreegster]

It's because of the sys.path.append("..") hack, probably. I had an old branch trying to set up Python modules properly, but I didn't get it working. Not sure why you're hitting this problem though. I'll look into it eventually :\

[Robinlovelace]

I should be able to debug it... I tried with

sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))

but that also didn't work. I'll try moving the contents of utils.py into setup.py to see if that works..

[Robinlovelace]

That did it, things are happening!

[Robinlovelace]

[Robinlovelace]

This is done! Great work on it Dustin, it's got big potential.