2.0 spec

[pnorman]

Fixes #43 Fixes #44 Fixes #31 Fixes #28 Closes #47 (imo)

This is a draft of a 2.0 spec which addresses tile formats and compression.

Unaddressed are spec versioning (#3), tileset versioning (#31), and tileset type (#28) since no decision was made on if those changes should be accepted or not.

For a diff of 1.2 to 2.0, see https://github.com/mapbox/mbtiles-spec/compare/14942c649e4cff48d3b0c393f856d7681890594a...pnorman:2.0

CC @hallahan @blair1618 @systemed

[hallahan]

I like that you included o5m. Other formats that I would like to use are OSM XML and OSM PBF. How would we declare those? Perhaps we should allow any data format for the contents of a tile and have a good way of describing what it is?

[tmcw]

Some comments:

• Since MBTiles is a container format, it's certainly chill to store whatever in it. But also since it's a container format, having MBTiles filled with more exotic formats will inevitably reduce the expectation of "this will work" from MBTiles consumers. If the spec starts pushing for more formats of vector data sources, I think it should also explicitly acknowledge that it does not require implementors to include support for all formats, and potentially should make clear what implementors should do when an unknown format is encountered.
• Listing explicit formats seems like a slope to shoutouts to everyone. Could we say, instead, that if a format is a mime-type, then it is treated as that type?

[pnorman]

Listing explicit formats seems like a slope to shoutouts to everyone. Could we say, instead, that if a format is a mime-type, then it is treated as that type?

I was trying to preserve compatibility with jpg and png so a valid mbtiles 1.2 is a valid 2.0. If breaking backwards compatibility is an option, I'd go with MIME type.

Since MBTiles is a container format, it's certainly chill to store whatever in it. But also since it's a container format, having MBTiles filled with more exotic formats will inevitably reduce the expectation of "this will work" from MBTiles consumers.

2.0 makes it clear that it is a container format that can store any tiled data, which is not possible on 1.x. Does this require modifications to the prose or README.md?

If the spec starts pushing for more formats of vector data sources,

Well, support for any vector data sources.

I think it should also explicitly acknowledge that it does not require implementors to include support for all formats, and potentially should make clear what implementors should do when an unknown format is encountered.

Right now the spec places no requirements on implementations, just on the mbtiles files. I see the role of mbtiles here similar to that of .tar - it defines a way to store content/files, and it's up to something else to make use of that content.

I agree that right now any implementation is likely to have the same support for png and jpg and the situation will change with arbitrary types, so I'll try to come up with appropriate wording that makes it clear there's no requirement on what readers do with the contained tile data. fwiw, this is theoretically possible in 1.2 if you had a reader that supported jpeg but not png.

[pnorman]

Perhaps we should allow any data format for the contents of a tile and have a good way of describing what it is?

Any data format is allowed, it's just a question of describing it.

[hallahan]

One thing to make note of is that the MIME type of MVT is application/vnd.mapbox-vector-tile. Looking at how VND MIME types work, can we basically put whatever we want in there? I've seen vnd.android.cursor.dir where android would be the IANA recognized vendor name. Having the vendor name isn't required though...

I'd say we really do not want to break backwards compatibility with the 1.x specs. Part of the motivation to continue using MBTiles is that it has been a well known format for 5+ years.

[pnorman]

Looking at how VND MIME types work, can we basically put whatever we want in there?

You register MIME types, but it's easy. See mapbox/vector-tile-spec#48 for background. I'm talking to some people about doing this for OSM types.

I'd say we really do not want to break backwards compatibility with the 1.x specs. Part of the motivation to continue using MBTiles is that it has been a well known format for 5+ years.

Is there much software which relies on png/jpg to know what to do with tile data?

[hallahan]

Looking at Mapbox Android SDK Legacy (DEPRECATED), it looks like the MBTiles support doesn't look at format at all. It simply assumes that it is PNG, since it uses Android's BitmapFactory to decode an InputStream. BitmapFactory won't handle JPEG decoding.

https://github.com/AmericanRedCross/OpenMapKitAndroid/blob/5df03841475b805141c9b8441512653cbd8672c9/MapboxAndroidSDK/src/main/java/com/mapbox/mapboxsdk/tileprovider/tilesource/MBTilesLayer.java#L199

https://github.com/AmericanRedCross/OpenMapKitAndroid/blob/5df03841475b805141c9b8441512653cbd8672c9/MapboxAndroidSDK/src/main/java/com/mapbox/mapboxsdk/tileprovider/MapTileCache.java#L102

Looking at WhirlyGlobe-Maply (Android), it looks like it does the same thing.

https://github.com/mousebird/WhirlyGlobe/blob/develop_3_0/WhirlyGlobeSrc/Android/src/com/mousebird/maply/MBTiles.java#L249

https://github.com/mousebird/WhirlyGlobe/blob/develop_3_0/WhirlyGlobeSrc/Android/src/com/mousebird/maply/MBTilesImageSource.java#L68

However, WhirlyGlobe does check to see if it is jpg or png, so that suggests that jpg support may come in the future.

https://github.com/mousebird/WhirlyGlobe/blob/develop_3_0/WhirlyGlobeSrc/Android/src/com/mousebird/maply/MBTiles.java#L188

However, that is referencing the Android implementation. The iOS implementation can handle either, because under the hood, it uses UIImage which figures out the format from they blob for you.

https://github.com/mousebird/WhirlyGlobe/blob/8e7b1693d95c32be643f55aff4d41bf4bc296753/WhirlyGlobeSrc/WhirlyGlobeLib/src/LoadedTile.mm#L91

Another good one would be Nutiteq, though I don't have access to the source. For the Android, my best guess would be that they do the same thing and only support PNG.

Phew! So, I think the answer would be: no. The implementation either assumes it's PNG or discerns if it is PNG or JPG from the data itself.

[pnorman]

Phew! So, I think the answer would be: no. The implementation either assumes it's PNG or discerns if it is PNG or JPG from the data itself.

That means that we can break backwards compatibility but there is a subset of 2.0 tilesets which will work the same on most 1.2 readers.

I'm surprised by the lack of JPEG support, are there not many JPEG tilesets in the wild?

[hallahan]

From my experience, it's always been PNG unless MapQuest made it. I think Mapbox does JPG too, but people usually use the PNG endpoints?

[pnorman]

From my experience, it's always been PNG unless MapQuest made it. I think Mapbox does JPG too, but people usually use the PNG endpoints?

The main reason for this is that PNG is a better format for maps unless you have hillshading, imagery, or some other raster data.

I'll edit to use MIME type if we think this is acceptable. For OSM MIME types, I'm going to be starting a discussion on registering them - some details still need working out.

[pnorman]

The roadmap has "The format will switch tile ordering to the XYZ schema popularized by OpenStreetMap and away from the Tile Map Service specification." Do we want to do this in 2.0, put it off to a later version, or drop it from the roadmap?

[pnorman]

The roadmap has "The format will switch tile ordering to the XYZ schema popularized by OpenStreetMap and away from the Tile Map Service specification." Do we want to do this in 2.0, put it off to a later version, or drop it from the roadmap?

fwiw, I'm in favour of dropping it. The conversion between the two is technically trivial and I think the pain from breaking compatibility in this way is worse than any potential gains.

[hallahan]

👍 . Support for Google TMS would just make things more confusing. Perhaps give this hint in the spec:

y = (1 << z) - 1 - y;

[pnorman]

Couple of changes following some gitter discussion and the discussion here

• Keep format as REQUIRED.
• Require all data in a mbtiles to have the same compression, even if compression is not specified. A tileset of mixed formats or compressions would require detecting information from magic numbers of tile data, which we want to avoid.
• Use MIME type
• Adjust prose in the readme that mbtiles is a container format

Remaining issues

• [x] Is more prose explaining the implications of mbtiles being a container format and a specific mbtiles might not be useful to a specific piece of software needed?
• [x] The spec guarantees interoperability in reading mbtiles, but with views there is no guarantee that two tilesets will be written the same way, or that an implementation can add tiles to a tileset. Should this be made explicit?
• [x] Review that "tileset" is being used everywhere that mbtiles files are mentioned

[e-n-f]

BTW, I had failed to see this PR and just made my own attempt (https://github.com/mapbox/mbtiles-spec/pull/47) to document the current de facto standard for mbtiles, which I should now attempt to reconcile with these changes.

[pnorman]

I think the only changes left are informative ones to the prose of the text to make some aspects clearer. Does anyone see anything else?

[e-n-f]

@pnorman I know I am very late to this, but I would suggest (based on what I did in https://github.com/mapbox/mbtiles-spec/pull/47):

• Recommend creating an index for metadata
• Recommend creating an index for tiles
• Document existing png, jpg, mvt, and pbf as aliases for MIME types in format
• Add recommended center, minzoom, and maxzoom to metadata
• Document what version and type are for. (As far as I know nothing actually cares, so maybe they should be removed instead.)
• Document json field, which is required for Mapbox Vector Tiles
• Include the formula for converting between TMS Y axis and tile URL Y axis
• Maybe make all the metadata except name, format, and json optional, since existing implementations will infer the rest

[pnorman]

I view indexes as an implantation detail because views are allowed, they don't need to be tables.

More on other comments when I get home.

[pnorman]

Document existing png, jpg, mvt, and pbf as aliases for MIME types in format

With the previous discussion of support in the wild fo the format key, I don't think we need this. Existing implementations auto-detect the image type or otherwise ignore the format key.

Document json field, which is required for Mapbox Vector Tiles

As including what?

Document what version and type are for. (As far as I know nothing actually cares, so maybe they should be removed instead.)

Can we verify this? I'm not opposed to making them OPTIONAL after looking at existing implementations.

[pnorman]

I was thinking of adding a section along these lines

Compatibility

This section is informative and does not add requirements to implementations

Because views may be used to produce the MBTiles schema two implementations may store tiles with different internal details, meaning one implementation may not be able to add to an existing file.

As a container format, MBTiles can store any tiled data, so data can be stored that an implementation cannot do anything with.

[e-n-f]

@pnorman The only tool I know that insists on a particular format is tile-reduce, which wants pbf, but that should be easy to fix. Given that, I'm fine with requiring a MIME type going forward.

The wording that I came up with for the json row is in my pull request if you'd like to use it.

I am pretty sure type is genuinely optional, because Tippecanoe doesn't produce it, and neither does Mapbox Studio Classic, and as far as I know no one has complained. Mapbox Studio Classic does not produce version, but Tippecanoe arbitrarily sets version to 2. I am OK with leaving either or both of them required as if anyone wants them, but if so, it would be useful to document some some sort of semantics for version. (Version of the mbtiles spec? Revision of the tileset itself? Something else?)

[pnorman]

@pnorman The only tool I know that insists on a particular format is tile-reduce, which wants pbf, but that should be easy to fix. Given that, I'm fine with requiring a MIME type going forward.

I could see grandfathering jpg and png, but never pbf which no conformant mbtiles implementations would have produced. pbf also has the problem that there's lots of different pbf encodings for geodata.

The wording that I came up with for the json row is in my pull request if you'd like to use it.

I looked at it, and I'm 👎 on using it because it puts lots of format-specific data into the spec, and does so in the wrong way. If we wanted to add additional information we should do so with additional metadata keys or additional tables.

[e-n-f]

@pnorman I'm ok with putting the json definition in some other document and just referring to it here, but it has been a de facto requirement for producing vector tilesets that the Mapbox Unpacker will accept for more than two years, and I think it would be negligent to revise the spec without documenting it.

[pnorman]

• Made some metadata OPTIONAL
• Added a section on implementation compatibility
• Added wording about TMS tile_row != XYZ y
• Added myself to authors list

json is implementation specific and not all implementations require or produce this.

I am happy with the state of the PR, although another pass of copyediting would be welcome.

[pnorman]

I am happy with the state of the PR, although another pass of copyediting would be welcome.

What else needs doing here?

[ImreSamu]

Thank you for working on this ..

It is possible to modify the spec to

• language neutral

  • now: name: The plain-english name of the tileset.
    • [a-z][a-z0-9_]* or utf8 ( https://sqlite.org/c3ref/column_name.html ) ?
    • spanish, german, hungarian , words allowed ?

• and add multi-language support ( like in OpenStreetMap )

  • attribution_: An attribution string, which explains in (and HTML) the sources of data and/or style for the map.
  • description_: A description ( in ) of the layer as plain utf text

[pnorman]

language neutral

Yes, I can fix this to be non-discriminatory. The encoding of strings is part of sqlite, and not something that belongs here, and I don't even think you have a good way to see what the encoding is.

[pnorman]

I believe all outstanding issues are solved.

Do the maintainers see anything else that needs changing?

[TimSC]

I support the suggestion by @ericfischer that version could do with clarification. "Version of the mbtiles spec? Revision of the tileset itself? Something else?"

[pnorman]

Pinging @flippmoke on this, since he said mapbox is resuming work on mbtiles

[flippmoke]

/cc @ericfischer

[pnorman]

ping @ericfischer

[flippmoke]

After some discussion with @ericfischer and @springmeyer on this issue last week we have decided to hold off on this until after #47 is complete. Our initial goal is to have the spec represent what is currently existing in the wild and then from there work towards how we want the spec to actually be. I believe this PR has a lot of potential in it for the next version 2.0 of the spec after 1.3.

If there are elements of this PR that should be moved into 1.3 (if they represent what is currently existing in the wild), please note and we will do our best to update it there.

That being said, I am somewhat concerned about the rebase of this branch after we update to 1.3 as we will need to "rebase" the spec here.

[pnorman]

Our initial goal is to have the spec represent what is currently existing in the wild and then from there work towards how we want the spec to actually be.

That's the goal with this 2.0 work too. Anything in the wild should work with it. The format key has changed, but existing software ignores that.

If there are elements of this PR that should be moved into 1.3 (if they represent what is currently existing in the wild), please note and we will do our best to update it there.

https://github.com/mapbox/mbtiles-spec/compare/14942c649e4cff48d3b0c393f856d7681890594a...pnorman:2.0 is the best thing to look at.

Off hand, I'd add

• all the grammatical changes
• use of RFC 2119
• TMS vs. XYZ wording
• "The tile_data blob column contains raw tile data in binary." although compression needs to be clarified

That being said, I am somewhat concerned about the rebase of this branch after we update to 1.3 as we will need to "rebase" the spec here.

The git side is easy, because they're all different documents. The problem is applying any changes made in 1.3 to this.

I still feel the 2.0 route is a better one to go with. In any case, I do not expect to allocate any time to it, because I mentally gave up on the mbtiles and tilejson specs a few months ago