Elevation data

[dabreegster]

This is a big project, but with small steps. Roughly:

1) Get better elevation data, either fixing the bugs in the srtm.rs impl or preferably using the better LIDAR data around Seattle. I have some pointers for that in an email somewhere; remind me to post them

2) Figure out how elevation change affects the standard bicyclist speed. Incorporate that in the simulation and pathfinding cost function

3) Render the elevation layer much more understandably. Isochrones are the thing people are familiar with, probably should give that a shot

I can give pointers to relevant code when needed

[dabreegster]

http://brouter.de/brouter/elevation.html for reference

[dabreegster]

All those references to better lidar data for washington, thanks to Michael Patrick from CUGOS:

• presentation: https://www.dnr.wa.gov/publications/ger_presentations_wa_gis_day_2016_gleason.pdf?ko7ffe
• downloading as multiple tiles: https://lidarportal.dnr.wa.gov/#47.49215:-122.29706:9
• As a single file:
  • NOAA Portal https://coast.noaa.gov/htdata/raster2/elevation/WA_King_DEM_2016_8589/
  • Tile Index: https://coast.noaa.gov/htdata/raster2/elevation/WA_King_DEM_2016_8589/tileindex_WA_King_DEM_2016.zip
  • ftp://ftp.coast.noaa.gov/pub/DigitalCoast/raster2/elevation/WA_King_DEM_2016_8589/
  • a 5GB geotiff: ftp://ftp.coast.noaa.gov/pub/DigitalCoast/raster2/elevation/WA_King_DEM_2016_8589/kingcounty_delivery1_be.tif

[dabreegster]

Seattle only and old data, but looks detailed: https://data-seattlecitygis.opendata.arcgis.com/datasets/contour-lines-1993

[Robinlovelace]

There's an R package that takes OSM data and calculates gradients in the surrounding area that can provide ideas/inspiration for this: https://github.com/ITSLeeds/slopes The main computational tasks are quite generally applicable so I would suggest splitting this out as a separate cargo crate, either within this repo (or perhaps better?) as a separate repo. Vested interest: I want the functionality in the slopes package to be faster. Thinking of eventually using the Rust implementation as a back-end via the libR-sys crate https://github.com/extendr/libR-sys . Another advantage of this approach: Rust is a solid and future-proof language with interfaces to many lanuages so a Rust implementation could help make the awesome functionality in slopes available for more communities, beyond the small (but highly active and rapidly growing #rspatial community).

[Robinlovelace]

In terms of Rust/R integration it's early days but I just ran the following commands from the extendr/libR-sys repo and it works fine:

libR-sys on  master via 𝗥 v1.48.0 
➜ cargo build                          
    Updating crates.io index
   Compiling libR-sys v0.2.1 (/mnt/57982e2a-2874-4246-a6fe-115c199bc6bd/orgs/extendr/libR-sys)
    Finished dev [unoptimized + debuginfo] target(s) in 12.26s

libR-sys on  master is 📦 v0.2.1 via 𝗥 v1.48.0 took 12s 
➜ cargo test -- --nocapture --test-threads=1

[Robinlovelace]

Good news: it seems the approach is fast. Faster than R's very mature C++ bindings according to a small benchmark here: https://github.com/extendr/rextendr/issues/11#issuecomment-755119346

[Robinlovelace]

Reproducible example on a machine that has R and Rust installed. Not 100% sure what's going on here, imagine @dabreegster you'll have a better idea if you can reproduce it...

library(rextendr)

# create a Rust function
rust_function("fn add(a:f64, b:f64) -> f64 { a + b }")
#> build directory: /tmp/RtmphEpi9F/file6a284aec7f99

# call it from R
add(2.5, 4.7)
#> [1] 7.2

^{Created on 2021-01-09 by the reprex package (v0.3.0)}

[Robinlovelace]

And with regard to the question of which elevation data source to use... SRTM has been superseded I think, some countries (and cities) have higher res data but for the purposes of this package (which should be able to use custom DEM data) I think globally available datasets as a default will be sufficient and most useful.

In the slopes package as far as I can recall we use the https://docs.mapbox.com/help/troubleshooting/access-elevation-data/ via the ceramic package, source code of which can be found here: https://github.com/hypertidy/ceramic/blob/7980c9f4134c0770ab3de351049e7dd511aac002/R/tile-providers.R

[dabreegster]

The Mapbox API is quite convenient. Did you hit any licensing or API limit issues with caching data from them?

For a general purpose crate, having a method that winds up making a Mapbox request would be very useful. For the A/B Street case, we likely want to use some kind of bulk API or to find and operate on a DEM file. I'd be a little surprised if Mapbox just lets you dump elevation data for a huge reason, cache it on your end, redistribute it, etc.

[Robinlovelace]

Did you hit any licensing or API limit issues with caching data from them?

Not yet, I have an unpaid mapbox API key that is quite generous and not even sure if you need one for the terrain data.

[dabreegster]

Huh, https://www.mapbox.com/legal/tos/ doesn't mention anything against redistributing data from their APIs. Nice!

[Robinlovelace]

But to be fair haven't been bulk downloading data in a way that I have been doing for OSM/STATS19 data. I'm sure lack of DEM data will not be a limiting factor for this issue in any case!

[dabreegster]

https://valhalla.readthedocs.io/en/latest/sif/elevation_costing/ reference for how to use elevation data for cycle routing

[PayasR]

SRTM has been superseded by NASADEM: https://earthdata.nasa.gov/esds/competitive-programs/measures/nasadem

[dabreegster]

Thanks, good to know! We could try hooking up NASDEM with https://github.com/eldang/elevation_lookups#raster-elevation-data pretty easily. If GDAL can read it, should be easy.

[dabreegster]

Summarizing how things work currently.

• We use https://github.com/eldang/elevation_lookups to deal with LIDAR data for Seattle, and with SRTM for other cities. This Python library has an API for elevation at a point and, more importantly, plugging in a polyline representing a street and getting the total incline.
  • Because this depends on GDAL and other stuff, we actually run it all in a Docker container: https://github.com/a-b-street/abstreet/blob/master/convert_osm/src/elevation.rs. I was running Ubuntu 18 for a while and for the life of me, couldn't get modern GDAL installed without breaking 10 other things.
• The A/B Street importer calls out to this library during the first step of importing a map. We just store elevation per intersection, currently. Everything downstream just uses that; we're not keeping around the raw elevation datasets.
• Walking and biking account for incline by costing pathfinding and in the simulation. Walking speed is here, based on Tobler's hiking function. Biking is here, loosely based on Valhalla -- I was unable to find a simple formula here.
• We have a few dataviz things hooked up, some https://github.com/a-b-street/abstreet/blob/master/game/src/layer/elevation.rs. Contour map, showing steep streets, and per-trip elevation profiles:

Future work / help needed:

• Pure Rust solutions without heavy GDAL dependencies would be nice. Especially to parallelize querying; because of some mutable caching layer, it's expensive to spawn multiple threads to query a large vector or raster input source.
• I've only carefully checked the end-to-end elevation results for Seattle, which uses LIDAR. Everywhere else uses SRTM, and the results look quite suspect -- as a quick sanity test, the contour map will jump from totally flat to 100m without anything in between. Not sure where the issue is yet.
• If NASDEM has worldwide coverage and is better than SRTM, we should switch to it.
• Things break near bridges, because I think the input dataset gives the value for the water/ground below. From the abst importer, we know when we are querying near a bridge, so there were some ideas to just ask for the polyline's endpoints, instead of sampling every few meters.

[dabreegster]

Keeping my eye on https://github.com/pka/georaster as another pure-Rust geotiff reader. Instead of memmap or similar for handling massive files, we could split the big UK and Seattle files into tiles (or possibly even download them that way to begin with)

[dabreegster]

https://opendem.info/