Support map vector tile?

[sckott]

I think the canonical proto file is here

https://github.com/mapbox/vector-tile-spec/blob/master/2.1/vector_tile.proto

not sure if you want to include any more here ...

[jeroen]

Is there an existing R class or json schema that has a corresponding one-on-one mapping into this structure? Can you elaborate a bit how/where we would use this?

[sckott]

There's a PR to rgbif pkg that i maintain https://github.com/ropensci/rgbif/pull/284 integrating with GBIF API for maps https://www.gbif.org/developer/maps and one format option they give back is MVT, map vector tiles, which is a binary protobuf format. We'd want a way to allow users of rgbif to read in this mvt format and use it to make maps in R - it seems we might have another option to use the GDAL stuff, but i'd rather avoid that if we can

a good reference doc: https://www.mapbox.com/vector-tiles/specification/

looking into conversions/mappings to other formats

[sckott]

You can get GBIF mvt data e.g, like

install.packages("rgbif")
library(rgbif)
x <- map_fetch(taxonKey = 2480498, year = 2010, format = ".mvt")
is.raw(x)

[jeroen]

Maybe here is more demo data: https://github.com/mapbox/vector-tile-base/tree/master/tests/data

[jeroen]

I have started with an implementation in the mapbox branch:

remotes::install_github("jeroen/protolite@mapbox")
?read_mvt

This parses the protobuf and extracts features and attributes but I am unsure how to treat the geometry vector. Currently it just returns the vector of integers from the data, I suppose we should transform that as described in section 4.3 of the spec?

[sckott]

there looks to be some c++ stuff in here for parsing mvt https://github.com/mapbox/awesome-vector-tiles - though i imagine it's easy enough for you to do from scratch

[sckott]

there's some good examples here https://github.com/NYCPlanning/labs-geojson2mvt/tree/master/example - run npm install in the root of the repo, then node script.js within the example dir, and it generates a bunch of .mvt files in a the tiles dir, some of which are empty files, but many have data. e.g.,

protolite::read_mvt("labs-geojson2mvt/example/tiles/11/604/768.mvt")
[[1]]
[[1]]$version
[1] 1

[[1]]$name
[1] "layer0"

[[1]]$extent
[1] 4096

[[1]]$features
[[1]]$features[[1]]
[[1]]$features[[1]]$id
[1] 0

[[1]]$features[[1]]$type
[1] 2

[[1]]$features[[1]]$attributes
[[1]]$features[[1]]$attributes$color
[1] "FAA61A"

[[1]]$features[[1]]$attributes$id
[1] 479

[[1]]$features[[1]]$attributes$route_id
[1] "Q50"

[[1]]$features[[1]]$attributes$route_long
[1] "Co-Op City/Pelham Bay - Flushing"

[[1]]$features[[1]]$geometry
  [1]    9  140 8320   10    9   27    9    0    0   10    7   25    9    0    0   10    7   29    9    0    0   10    7   19    9    0
 [27]    0   10    1    3    9    0    0   34   35   83   13   69    2   77   18   71    9    0    0   10    2    5    9    0    0   10
 [53]   10   45    9    0    0   10    6   25    9    0    0   10    4    9    9    0    0   10    2    7    9    0    0   10   11    3
 [79]    9    0    0   10    3    1    9    3    0   10    7    1    9    0    0   10    5   20    9    0    4   10    9   38    9    0
[105]    0   18    9   38   20    2    9    0    0   10   10    2    9   24   95   66   20   41    0   35   29   81   13  113   27  115
[131]   27   71   97  175   47  111    9    0  819  202   20   16   56   14  258  103   28    0   22   24    5   58   57  104   23   76
[157]    0  206   17   52   47   60   15   54    2   10  190   42   42   30   14   22    0   40   47   12   23   18  161  244   10   20
[183]   60   32   36   86   52   23    7   19    9  145  898   34   31   53   25   81    0   89   18   79    9    0    0   10   22   97

// .... cutoff

[jeroen]

I have merged an initial implementation in master: https://github.com/jeroen/protolite/pull/13

Hopefully you can test this and see if there are any issues.

[sckott]

sweet, trying it out now

[sckott]

This isn't working as expected:

read_mvt_data("../labs-geojson2mvt/example/tiles/17/38589/49274.mvt")[[1]]$features[[1]]$geometry
#>      [,1] [,2] [,3]
#> [1,] 8100 1958    1

Using this node cli tool https://github.com/mapbox/vt2geojson

vt2geojson example/tiles/17/38589/49274.mvt

{
  "type": "FeatureCollection",
  "features": [
    {
      "type": "Feature",
      "geometry": {
        "type": "Point",
        "coordinates": [
          -74.00943025946617,
          40.725948208350815
        ]
      },
      "properties": {
        "stop_id": "903079",
        "stop_name": "SPRING ST/GREENWICH ST"
      }
    }
  ]
}

[sckott]

Ah, i guess all of them turn out like above. I assume there's another step required to convert the matrix to coordinates?

[sckott]

while testing this, seems like the read_mvt methods should allow query parameters beyond the .mvt ending, e.g.,

read_mvt_sf('https://api.gbif.org/v2/map/occurrence/density/9/81/183.mvt')
#> works fine
read_mvt_sf('https://api.gbif.org/v2/map/occurrence/density/9/81/183.mvt?year=2010')
#> Error in parse_mvt_params(data) :
#>   Input path does have standard /{z}/{x}/{y}.mvt format so you must provide zxy parameters manually

this doesn't affect the rgbif usage, but it would be good for this to work more generally, and not just with urls without query params

[sckott]

other than the above about query params with urls, i think it's good to go.

Did some rough comparisons, comparing read_mvt_sf converted to geojson to the cli tool vt2geojson on the same mvt tiles created in the labs-geojson2mvt repo. Spot checked many of the same files visually and the coordinates and I couldn't see any differences.

[jeroen]

OK I added something to strip the URL params