Support for ESRI ProtocolBuffer format

[internoot]

This library is a dream to work with, thanks for making this available.

I was wondering if there is any plan to support ESRI's PBF format to provide an option other than JSON for the underlying data retrieval (or interest in a PR for this)?

Some ArcGIS REST endpoints seem to have issues (consistent HTTP 500 errors) generating JSON/geoJSON for very large features but seem capable of returning these features in the more efficient PBF format. For example, some features in this layer are not possible to reliably retrieve using JSON.

[CalebM1987]

There are not currently any plans for supporting the pbf format, however, I think that would be incredibly useful. We would certainly be open to any pull requests that would implement this functionality.

I would be interested in taking a stab at it, but I don't know when I would find the time to do so. @philnagel would you have any interest in helping with this as well? I definitely think this would be great feature to support.

[philnagel]

I agree, this would be a great addition to this package. I do not have any experience with ESRI's PBF format. Although I was able to compile a basic parser based on ESRI's published spec for this, it did not seem like everything was necessarily working as smoothly as it should. @internoot if you have worked with this format before, we would really appreciate any contribution you can make in terms of adding support, even just building a basic parser that could then be further tied in with the rest of the package. Otherwise, I am in the same boat as Caleb. I would certainly be interested in working on this, but I am not sure when I would be able to get around to it.

[internoot]

I don't have any experience with protobuf either, but I had a little play and was able to decode some FeatureCollections

The geometries are structurally quite different and require decoding from a quantized form & scaling to return coordinates in the query CRS

geometry {
  lengths: 5
  coords: 5268789779552354
  coords: -3867244056744994
  coords: 5271878
  coords: 184069004
  coords: -183249925
  coords: 5301663
  coords: -5271880
  coords: -184083895
  coords: 183249927
  coords: -5286772
}

...

scale {
  xScale: 6.714873101998366e-09
  yScale: 6.714873101998366e-09
  mScale: 0.0001
  zScale: 0.0001
}
translate {
  xTranslate: -20037700.0
  yTranslate: -30241100.0
  mTranslate: -100000.0
  zTranslate: -100000.0
}

Docs are pretty thin but I found some helpful notes in this issue and will have a crack at decoding the PBF back to a typical geometry

[internoot]

geometry: {
  coords: [256, 256, 2, 0, 0, 2, -2, 0, 0, -2]
  lengths: [5]
}

Each coordinate pair is relative to the pair preceding it within groups of lengths[], they can be unpacked along the lines of:

i = 1
while i < geometry.lengths[0]:
    geometry.coords[2 * i] += geometry.coords[2 * (i - 1)]
    geometry.coords[2 * i + 1] += geometry.coords[2 * (i - 1) + 1]
    i = i + 1

For output:

[256, 256, 258, 256, 258, 258, 256, 258, 256, 256]

Where lengths contains multiple elements in the context of polygons these would unpack to multiple rings:

geometry: {
  coords: [256, 256, 2, 0, 0, 2, -2, 0, 0, -2, 56, 56, 2, 0, 0, 2, -2, 0, 0, -2]
  lengths: [5, 5]
}

...

Polygon {
  rings: [ 
    [ [256, 256], [258, 256], [258, 258], [256, 258], [256, 256] ], 
    [ [56, 56], [58, 56], [58, 58], [56, 58], [56, 56] ] 
  ]
}

Once unpacked the coordinates need to be scaled & translated:

if transform.scale:
    x *= transform.scale.xScale
    y *= transform.scale.yScale

if transform.translate:
    x += transform.translate.xTranslate
    y += transform.translate.yTranslate

There's a bit of work to implement all of the various ESRI geometry types but writing a parser which returns functionally equivalent objects to f=json looks doable

Edit: There's a partial JS implementation of just that here: https://github.com/rowanwins/arcgis-pbf-parser