search

mapbox / vector-tile-spec

Mapbox Vector Tile specification
https://www.mapbox.com/vector-tiles/specification/
Other
900 stars 211 forks source link

FlatBuffer support #98

Closed gertcuykens closed 7 years ago

gertcuykens commented 7 years ago

Probably been asked before but didn't find it. Can we have flatbuffers instead?

https://google.github.io/flatbuffers/flatbuffers_benchmarks.html

flippmoke commented 7 years ago

My initial reaction to this is no. Granted I don't have experience with flatbuffers, but here are the reasons.

gertcuykens commented 7 years ago

As benchmark suggest it's not going to be smaller that's correct. But looking at the numbers it's worth investing the time to make a vector tile benchmark. My knowledge how to do so is limited. But because this specification is aiming for global adoption the best solution is required to prevent others to implement their own and end up with a mess of different specifications not maintained by the open source community.

flippmoke commented 7 years ago

Are you concerned mostly about reading and writing speed from the files themselves @gertcuykens? We developed https://github.com/mapbox/protozero to make read/write of protobuffers much faster at Mapbox.

Additionally, imagine we did allow flatbuffers -- does that mean we end protobuffer support? If not there would be two very different file encodings for vector tiles, which would be very confusing. The only other option would be to do so on a new version and even then to support legacy vector tile types you would have to multiple types of encoding/decoding libraries just within your vector tile library.

gertcuykens commented 7 years ago

For me generating a tile buffer in golang as quickly as possible is important and it becomes more important over time while maps get bigger and contain more details. If the loss of bandwidth is in reason in exchange for a much higher speed increase encoding and decoding tiles on the backend when map updates occur then in my eyes it's worth considering creating a specification draft based on the current one and see if it is possible to make a field that specifies the encoding used.

Note that flatbuffer also has many native libraries for many different languages, if protozero can get to the speed of flatbuffer which I didn't see numbers for yet, we the community have to create native libraries also if we really want to get the most out of it.

flippmoke commented 7 years ago

@gertcuykens is your performance bottleneck really buffer creation at this point? Decoding the geometry object already will require the creation of a buffer, the only real benefit is reading of properties.

Additionally, in vector tile encoding there are a lot of other steps that are significantly slower then the buffer creation for the tile, such as:

gertcuykens commented 7 years ago

I agree probably not, but still should the specification not allow to have a handshake mechanism for deciding the encoding used? If it would be in the specs then it would be more future proof if let say the encoding or decoding turns out one day to be a bottleneck.

flippmoke commented 7 years ago

@gertcuykens the best way I can explain this is as an analogy.

Would geojson still be geojson if the formatting was allowed to be the same in XML? I do not think the specification should be this generalized, it would only fragment users of the data and cause more confusion. Closing this issue as I do not see this as ever being a part of v3 or any future version of the specification.

gertcuykens commented 7 years ago

I see it more like a http Content-Encoding: gzip analogy :P but still like you pointed out the bottle neck right now is not going to be protobuf. But also I am pretty sure if we turned back time and both buffers where on the table while designing the specification, flatbuffer may have been the better pick else protozero wouldn't be needed. I will agree the cost is too high to break current specification.

gertcuykens commented 6 years ago

FYI: Turns out protobuf is faster then flatbuffers using gogo protobuf https://github.com/alecthomas/go_serialization_benchmarks So right now protobuf is the way to go either way for me XD