Interface to batch routing API

[Robinlovelace]

It would be good to have an interface to the batch routing API, enabling upload of .csv files and download of .geojson files (or output in other formats).

This issue can track development of this functionality.

```r # Aim: test batch routing library(stplanr) library(cyclestreets) # input data ?odrust::odr_jitter library(odrust) od = readr::read_csv("https://github.com/dabreegster/odjitter/raw/main/data/od.csv") zones = sf::read_sf("https://github.com/dabreegster/odjitter/raw/main/data/zones.geojson") road_network = sf::read_sf("https://github.com/dabreegster/odjitter/raw/main/data/road_network.geojson") set.seed(42) od_jittered = odr_jitter( od, zones, subpoints = road_network, disaggregation_threshold = 50 ) od_geo = od_jittered[1:3, 0] coord_example = od::od_coordinates(od_geo) readr::write_csv(as.data.frame(coord_example), "coord_example.csv") sf::write_sf(od_geo, "coord_example.geojson") coord_example_sf = sf::st_sf(coord_example, geometry = od_geo$geometry) coord_example_cs = route(coord_example_sf, route_fun = journey) sf::write_sf(coord_example_cs, "batch_output_example_route_stplanr.geojson") ```

[Robinlovelace]

Example inputs and envisioned outputs can be found here: https://github.com/cyclestreets/cyclestreets-r/tree/master/data-raw

Function on the R side could be something like:

journey_batch(od_geo, plan = "balanced")

on the R side.

[Robinlovelace]

Results from quick test below.

# starting from outputs
library(tmap)
tmap_mode("view")
#> tmap mode set to interactive viewing
piggyback::pb_list()
#> Error in libgit2::git_repository_discover: could not find repository from '/tmp/RtmppYpY4d/reprex-71d0d42538d59-flat-ram'
u1 = "https://github.com/cyclestreets/cyclestreets-r/releases/download/v0.5.3/batchtesting.-.with.Include.full.JSON.API.response.option.geojson"
u2 = "https://github.com/cyclestreets/cyclestreets-r/releases/download/v0.5.3/batchtesting.-.with.Include.route.string.column.geojson"
u3 = "https://github.com/cyclestreets/cyclestreets-r/releases/download/v0.5.3/batchtesting.-.with.neither.option.geojson"
route1 = sf::read_sf(u1)
route2 = sf::read_sf(u2)
route3 = sf::read_sf(u3)
route1
#> Simple feature collection with 2 features and 43 fields
#> Geometry type: LINESTRING
#> Dimension:     XY
#> Bounding box:  xmin: -0.24156 ymin: 51.56121 xmax: 0.04791 ymax: 51.58567
#> Geodetic CRS:  WGS 84
#> # A tibble: 2 × 44
#>   start_id start_longitude_request start_latitude_re… finish_id finish_longitud…
#>   <chr>                      <dbl>              <dbl> <chr>                <dbl>
#> 1 0_0_0                     -0.241               51.6 0_0_3              -0.0242
#> 2 0_0_0                     -0.241               51.6 0_0_4               0.0481
#> # … with 39 more variables: finish_latitude_request <dbl>, start <chr>,
#> #   finish <chr>, startBearing <chr>, startSpeed <chr>, start_longitude <chr>,
#> #   start_latitude <chr>, finish_longitude <chr>, finish_latitude <chr>,
#> #   crow_fly_distance <chr>, event <chr>, whence <chr>, speed <chr>,
#> #   itinerary <chr>, clientRouteId <chr>, plan <chr>, note <chr>, length <chr>,
#> #   time <chr>, busynance <chr>, quietness <chr>, signalledJunctions <chr>,
#> #   signalledCrossings <chr>, west <chr>, south <chr>, east <chr>, …
route2
#> Simple feature collection with 2 features and 10 fields
#> Geometry type: LINESTRING
#> Dimension:     XY
#> Bounding box:  xmin: -0.25569 ymin: 51.58447 xmax: -0.03854 ymax: 51.59919
#> Geodetic CRS:  WGS 84
#> # A tibble: 2 × 11
#>   start_id start_longitude_request start_latitude_re… finish_id finish_longitud…
#>   <chr>                      <dbl>              <dbl> <chr>                <dbl>
#> 1 0_0_0                     -0.256               51.6 0_0_4              -0.0820
#> 2 0_0_0                     -0.256               51.6 0_0_5              -0.0386
#> # … with 6 more variables: finish_latitude_request <dbl>, strategy <chr>,
#> #   distance <chr>, time_seconds <chr>, calories <chr>,
#> #   geometry <LINESTRING [°]>
route3
#> Simple feature collection with 2 features and 10 fields
#> Geometry type: LINESTRING
#> Dimension:     XY
#> Bounding box:  xmin: -0.24124 ymin: 51.58359 xmax: 0.04808 ymax: 51.58359
#> Geodetic CRS:  WGS 84
#> # A tibble: 2 × 11
#>   start_id start_longitude_request start_latitude_re… finish_id finish_longitud…
#>   <chr>                      <dbl>              <dbl> <chr>                <dbl>
#> 1 0_0_0                     -0.241               51.6 0_0_3              -0.0242
#> 2 0_0_0                     -0.241               51.6 0_0_4               0.0481
#> # … with 6 more variables: finish_latitude_request <dbl>, strategy <chr>,
#> #   distance <chr>, time_seconds <chr>, calories <chr>,
#> #   geometry <LINESTRING [°]>
qtm(route1, col = names(route1)[1]) # it's a long route!

plot(route1$geometry[1], lwd = 5, col = "red")
plot(route1$geometry[2], add = TRUE)

# Let's take a look at segment level data
route1$distances
#> [1] "15,18,11,12,12,22,15,54,3,48,7,49,27,4,19,3,6,10,9,179,21,51,7,9,18,29,14,18,19,31,15,10,13,47,5,58,20,23,10,4,3,9,12,4,17,11,13,81,26,23,27,31,42,51,1,118,15,41,12,30,33,70,16,20,37,108,28,66,31,5,24,15,10,15,46,6,42,9,5,6,6,3,5,2,13,6,6,5,5,6,2,7,7,7,8,5,5,4,7,7,6,7,13,6,6,7,7,6,7,6,7,5,6,40,32,42,4,38,9,7,9,7,8,10,14,44,10,11,9,8,7,11,12,21,15,8,90,66,10,142,10,31,13,6,7,4,5,14,4,45,234,84,9,14,21,80,5,9,12,42,24,45,82,9,11,37,24,15,37,2,5,2,3,1,4,7,18,6,101,114,3,116,25,15,46,21,103,40,66,48,19,44,293,10,95,44,33,41,21,16,14,20,40,70,75,6,11,18,12,13,37,20,2,24,11,16,10,8,7,14,14,15,36,39,28,23,33,17,27,8,9,3,4,14,16,30,68,32,24,27,44,30,23,20,26,14,10,10,31,20,27,22,11,14,4,3,5,30,35,24,48,15,15,13,11,12,10,9,13,2,53,11,6,8,14,2,6,16,13,3,1,3,4,3,3,3,6,9,9,20,14,23,7,3,12,16,51,50,26,46,21,16,24,62,39,27,33,24,14,19,14,32,19,11,27,30,30,23,31,29,10,20,108,91,56,33,85,30,36,7,17,22,22,4,40,26,14,14,43,16,7,25,36,37,56,48,36,32,27,40,40,7,6,50,49,44,14,24,35,28,7,6,21,41,41,15,4,7,9,10,36,34,60,4,4,13,24,19,3,15,11,11,39,7,10,60,6,15,19,35,53,9,9,21,32,26,3,14,7,8,67,11,8,79,38,23,8,8,55,24,38,12,51,53,21,25,39,27,106,15,14,38,9,56,49,56,10,17,30,11,1,10,8,4,32,32,24,28,24,43,19,17,15,18,18,15,15,12,11,39,26,12,23,8,54,70,71,9,58,17,44,16,58,43,20,25,29,29,4,7,19,26,24,34,5,4,17,22,3,4,4,4,58,4,4,10,1,8,3,5,31,197,20,21,9,27,8,23,34,97,53,21,12,13,20,4,2,4,3,3,5,7,3,4,2,3,8,37,21,19,71,7,34,42,72,75,73,76,16,73,10,20,9,4,3,29,13,12,60,3,99,6,5,4,4,4,5,19,43,3,3,3,4,4,2,2,2,2,4,7,10,6,12,9,8,78,24,105,19,22,25,10,8,12,41,8,9,8,8,27,9,9,8,8,4,3,3,2,3,3,34,38,13,16,26,16,10,5,4,4,10,9,2,15,20,5,6,4,3,9,22,6,34,26,24,14,16,15,11,14,100,80,45,9,7,9,64,7,8,5,10,9,7,4,4,9,12,36,31,12,17,41,44,10,27,146,47,29,9,10,23,32,48,17,33,74,60,51,10,6,7,49,16,44,7,31,13,13,4,14,20,33,14,19,14,21,17,44,77,16,14,28,3,29,62,11,12,6,22,7,16,13,5,5,19,43,33,13,26,41,10,7,6,10,17,36,24,4,2,5,13,87,16,3,4,4,3,7,9,7,16,50,4,4,8,18,43,24,65,25,282,67,36,109,27,2,2,5,33,8,5,3,14,8,21,15,24,8,2,7,10,9,11,10,16,4,3,4,4,4,13,4,13,8,7,5,6,6,4,5,3,15,13,11,14,12,18,21,17,10,3,3,8,10,6,5,4,10,9,4,5,5,5,2,13,99,6,64,4,26,19,14,28,35,21,45,15,8,8,17,52,69,40,7,85,8,59,65,74,75,11,36,13,20,38,58,8,60,13,16,29,42,25,53,8,6,9,7,174,18,92,38,10,21,5,16,8,21,11,7,8,11,37,50,42,39,20,13,23,22,9,4,3,2,3,2,12,13,3,5,17,63,29,23,25,1,14,7,5,9,46,6,5,4,7,39,4,47,3"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
#> [2] "15,18,11,12,12,22,15,54,3,48,7,49,27,4,19,3,6,10,9,179,21,51,7,9,18,29,14,18,19,31,15,10,13,47,5,58,20,23,10,4,3,9,12,4,17,11,13,81,26,23,27,31,42,51,1,118,15,41,12,30,33,70,16,20,37,108,28,66,31,5,24,15,10,15,46,6,42,9,5,6,6,3,5,2,13,6,6,5,5,6,2,7,7,7,8,5,5,4,7,7,6,7,13,6,6,7,7,6,7,6,7,5,6,40,32,42,4,38,9,7,9,7,8,10,14,44,10,11,9,8,7,11,12,21,15,8,90,66,10,142,10,31,13,6,7,4,5,14,4,45,234,84,9,14,21,80,5,9,12,42,24,45,82,9,11,37,24,15,37,2,5,2,3,1,4,7,18,6,101,114,3,116,25,15,46,21,103,40,66,48,19,44,293,10,95,44,33,41,21,16,14,20,40,70,75,6,11,18,12,13,37,20,2,24,11,16,10,8,7,14,14,15,36,39,28,23,33,17,27,8,9,3,4,14,16,30,68,32,24,27,44,30,23,20,26,14,10,10,31,20,27,22,11,14,4,3,5,30,35,24,48,15,15,13,11,12,10,9,13,2,53,11,6,8,14,2,6,16,13,3,1,3,4,3,3,3,6,9,9,20,14,23,7,3,12,16,51,50,26,46,21,16,24,62,39,27,33,24,14,19,14,32,19,11,27,30,30,23,31,29,10,20,108,91,56,33,85,30,36,7,17,22,22,4,40,26,14,14,43,16,7,25,36,37,56,48,36,32,27,40,40,7,6,50,49,44,14,24,35,28,7,6,21,41,41,15,4,7,9,10,36,34,60,4,4,13,24,19,3,15,11,11,39,7,10,60,6,15,19,35,53,9,9,21,32,26,3,14,7,8,67,11,8,79,38,23,8,8,55,24,38,12,51,53,21,25,39,27,106,15,14,38,9,56,49,56,10,17,30,11,1,10,8,4,32,32,24,28,24,43,19,17,15,18,18,15,15,12,11,39,26,12,23,8,54,70,71,9,58,17,44,16,58,43,20,25,29,29,4,7,19,26,24,34,5,4,17,22,3,4,4,4,58,4,4,10,1,8,3,5,31,197,20,21,9,27,8,23,34,97,53,21,12,13,20,4,2,4,3,3,5,7,3,4,2,3,8,37,21,19,71,7,34,42,72,75,73,76,16,73,10,20,9,4,3,29,13,12,60,3,99,6,5,4,4,4,5,19,43,3,3,3,4,4,2,2,2,2,4,7,10,6,12,9,8,78,24,105,19,22,25,10,8,12,41,8,9,8,8,27,9,9,8,8,4,3,3,2,3,3,34,38,13,16,26,16,10,5,4,4,10,9,2,15,20,5,6,4,3,9,22,6,34,26,24,14,16,15,11,14,100,80,45,9,7,9,64,7,8,5,10,9,7,4,4,9,12,36,31,12,17,41,44,10,27,146,47,29,9,10,23,32,48,17,33,74,60,51,10,6,7,49,16,44,7,31,13,13,4,14,20,33,14,19,14,21,17,44,77,16,14,28,3,29,62,11,12,6,22,7,16,13,5,5,19,43,33,13,26,41,10,7,6,10,17,36,24,4,2,5,13,87,16,3,4,4,3,7,9,7,16,50,4,4,8,18,43,24,65,25,282,67,36,109,27,2,2,5,33,8,5,3,14,8,21,15,24,8,2,7,10,9,11,10,16,4,3,4,4,4,13,4,13,8,7,5,6,6,4,5,3,15,13,11,14,12,18,21,17,10,3,3,8,10,6,5,4,10,9,4,5,5,5,2,13,99,6,64,4,26,19,14,28,35,21,45,15,8,8,17,52,69,40,7,85,8,59,65,74,75,11,36,13,20,38,58,8,60,13,16,29,42,25,53,8,6,9,7,174,18,92,38,10,21,5,16,8,21,11,7,8,11,37,50,42,39,20,13,23,22,9,4,3,2,3,2,5,6,15,3,6,30,110,23,58,31,66,84,11,51,16,8,28,61,8,15,3,4,4,5,9,11,5,3,2,5,4,2,15,40,92,11,8,58,12,11,27,32,21,29,16,19,28,28,4,90,3,9,11,26,32,4,4,3,3,25,10,6,41,4,6,16,14,41,17,60,33,14,36,10,8,10,9,11,11,14,9,9,7,4,15,56,32,12,9,15,8,5,4,4,36,4,65,72,62,28,5,43,22,2,6,4,4,5,5,6,6,4,32,54,26,10,46,82,18,34,6,27,6,5,13,7,12,16,12,39,20,10,12,8,4,2,4,8,7,8,9,7,9,9,11,8,22,7,15,120,83,7,10,4,4,40,94,6,180,26,12,21,37,28,97,14,21,18,25,37,3,4,16,17,2,19,6,19,12,16,10,30,15,63,41,25,33,4,1,27,22,38,2,5,9,22,34,6,20,11,7,21,28,125,108,61,141,95,131,78,29,8,36,70,16,23,11,29,53,74,21,53,15,7,6,19,21,4,8,11,27,15,8,14,6,26,84,17,30,20,20,8,66,40,65,11,11,63,110,14"

^{Created on 2022-03-22 by the reprex package (v2.0.1)}

[Robinlovelace]

Plan: create a function called batch_journey() that will take arguments TBC in collaboration with @mvl22. Good news: we have example outputs so I can work on this before our meeting next week. See attached. batchtesting-data.csv.gz

[Robinlovelace]

Planning to work on this today. I may try using httr2 instead of httr: https://github.com/r-lib/httr2

Talk soon @mvl22 !

[Robinlovelace]

Heads-up @mvl22 this is the output from the {geojsonsf} package that I'd like to pass to the batch routing API:

{"type":"LineString","coordinates":[[-1.5187101220052267,53.792230071570809],[-1.5460238195984054,53.79586646502788]]} 
{"type":"LineString","coordinates":[[-1.5460238195984054,53.79586646502788],[-1.5460238195984054,53.79586646502788]]} 

Currently it's failing with the following message:

Response [https://api.cyclestreets.net/v2/batchroutes.createjob?key=xxx]
  Date: 2022-05-14 00:05
  Status: 200
  Content-Type: application/json; charset=UTF-8
  Size: 77 B
{
    "error": "The GeoJSON structure does not contain a features section."

Can you enable it to accept the IMO fairly uncontroversial input above?

For reference the tested example is a featurecollection with an unconventional (as far as I know) ID definition:

{"type": "FeatureCollection", "features": [
      {"type": "Feature", "id": 1, "properties": {}, "geometry": {"type": "Point", "coordinates": [0.14187, 52.20303]}},
      {"type": "Feature", "id": "a", "properties": {}, "geometry": {"type": "Point", "coordinates": [0.14711, 52.20061]}},
      {"type": "Feature", "id": 56, "properties": {}, "geometry": {"type": "Point", "coordinates": [0.11638, 52.20360]}}
    ]}

That's the current blocker on this ATM.

[Robinlovelace]

Another GeoJSON representation that would be ideal:

{
"type": "FeatureCollection",
"name": "coord_example2",
"crs": { "type": "name", "properties": { "name": "urn:ogc:def:crs:OGC:1.3:CRS84" } },
"features": [
{ "type": "Feature", "properties": { "id": "S02001616S02001616" }, "geometry": { "type": "LineString", "coordinates": [ [ -3.2235272, 55.9298953 ], [ -3.2101257, 55.9334341 ] ] } },
{ "type": "Feature", "properties": { "id": "S02001616S02001616" }, "geometry": { "type": "LineString", "coordinates": [ [ -3.2101018, 55.9346315 ], [ -3.2203651, 55.930717 ] ] } },
{ "type": "Feature", "properties": { "id": "S02001616S02001620" }, "geometry": { "type": "LineString", "coordinates": [ [ -3.2100342, 55.932279 ], [ -3.2064407, 55.9484088 ] ] } }
]
}

[mvl22]

It does already expect a FeatureCollection with LineStrings.

You will recall I changed this from the original prototype using CSV input to the API release using GeoJSON with LineStrings.

Another GeoJSON representation that would be ideal:

Yes, that is what is now supported, except for the id position which I am working through.

For reference the tested example is a featurecollection with an unconventional (as far as I know) ID definition:

That contained points, so was failing.

[Robinlovelace]

Yes, that is what is now supported, except for the id position which I am working through.

Great to hear, many thanks Martin! Just to clarify, this should now (or soon) work?:

{
"type": "FeatureCollection",
"name": "coord_example2",
"crs": { "type": "name", "properties": { "name": "urn:ogc:def:crs:OGC:1.3:CRS84" } },
"features": [
{ "type": "Feature", "properties": { "id": "S02001616S02001616" }, "geometry": { "type": "LineString", "coordinates": [ [ -3.2235272, 55.9298953 ], [ -3.2101257, 55.9334341 ] ] } },
{ "type": "Feature", "properties": { "id": "S02001616S02001616" }, "geometry": { "type": "LineString", "coordinates": [ [ -3.2101018, 55.9346315 ], [ -3.2203651, 55.930717 ] ] } },
{ "type": "Feature", "properties": { "id": "S02001616S02001620" }, "geometry": { "type": "LineString", "coordinates": [ [ -3.2100342, 55.932279 ], [ -3.2064407, 55.9484088 ] ] } }
]
}

[mvl22]

The real download URLs in the output is now implemented.

[mvl22]

Great to hear, many thanks Martin! Just to clarify, this should now (or soon) work?:

Yes, use of IDs in the properties is now supported.

[Robinlovelace]

Great news, it now works! Heads-up @mvl22 (and many thanks for implementing this) and @temospena (another potential 'early adopter' of this function) :tada:

[Robinlovelace]

Edge case bug in attached zip. test-data-batch.csv.gz

[Robinlovelace]

Reproducible error:

routes = batch_read(file = "test-data-batch.csv.gz")
Reading in the following file:
test-data-batch.csv
NA values detected
NA values detected
NA values detected
NA values detected
NA values detected
NA values detected
NA values detected
Error in `[[<-.data.frame`(`*tmp*`, i, value = c("1", "1", "1", "1", "1",  : 
  replacement has 5912 rows, data has 5940
In addition: Warning messages:
1: In data.frame(..., check.names = FALSE) :
  row names were found from a short variable and have been discarded
2: In data.frame(..., check.names = FALSE) :
  row names were found from a short variable and have been discarded
Called from: `[[<-.data.frame`(`*tmp*`, i, value = c("1", "1", "1", "1", "1", 
"1", "1", "1", "2", "2", "2", "2", "2", "2", "2", "2", "2", "2", 
"2", "2", "2", "2", "2", "2", "2", "2", "2", "2", "2", "2", "2", 
"2", "2", "2", "2", "2", "2", "2", "3", "3", "3", "3", "3", "3", 
"3", "4", "4", "5", "5", "5", "5", "5", "5", "5", "5", "5", "6", 
"6", "7", "7", "7", "7", "8", "8", "8", "8", "8", "8", "8", "8", 
"8", "8", "8", "8", "8", "8", "9", "9", "9", "9", "9", "9", "9", 
"10", "10", "10", "10", "10", "10", "10", "10", "11", "11", "11", 
"12", "12", "12", "12", "13", "13", "13", "13", "13", "13", "13", 
"13", "13", "14", "14", "15", "15", "15", "15", "16", "16", "16", 
"16", "16", "16", "17", "17", "17", "17", "17", "17", "17", "17", 
"18", "18", "19", "19", "19", "19", "19", "20", "20", "20", "20", 
"20", "20", "20", "20", "20", "20", "21", "21", "21", "21", "21", 
"22", "23", "23", "23", "23", "23", "23", "23", "23", "24", "24", 
"24", "24", "24", "24", "24", "24", "24", "24", "25", "25", "25", 
"25", "25", "25", "25", "25", "25", "25", "26", "26", "26", "26", 
"26", "26", "26", "26", "26", "26", "26", "27", "28", "28", "28", 
"28", "28", "28", "29", "29", "29", "29", "29", "29", "29", "29", 
"30", "30", "30", "30", "30", "30", "30", "30", "30", "30", "30", 
"30", "30", "31", "31", "31", "31", "31", "31", "32", "32", "32", 

[Robinlovelace]

This is the failing line:

  res_df$id = elev_df$id

[Robinlovelace]

This is now fixed!

[Robinlovelace]

Code below.

[Robinlovelace]

To set the username try changing...

[Robinlovelace]

You can set the username option in the batch() function as follows:

https://github.com/cyclestreets/cyclestreets-r/blob/4cbc639a0d01566c8e7413cbf283b8438ed3c18f/R/batch.R#L45