CLI: `pmtiles convert INPUT.mbtiles OUTPUT.pmtiles` fails silently for MBTiles v 1.0

mizmay commented 3 months ago

Problem

When trying to convert MBTiles created based on the 1.0 spec, such as in this repo, PMTiles are generated but won't display.

Explanation

MBTiles 1.0 spec did not include format among the top-level metadata. PMTiles relies on this field to determine whether the data is vector or raster, since this version only supported raster (jpg/png).

Potential fixes

Can be fixed manually by adding name/value format/png to the metadata table of the MBTiles, convert operation will work as expected.
Raising an error from the CLI and/or JS library to warn users the required metadata is missing to aid debugging
Inferring format from other available metadata or the contents of the MBTiles

Additional Information

Repo that currently contains a bunch of MBTiles 1.0: https://github.com/mizmay/BerkeleyElectionResults

Sample (detailed) metadata for MBTiles 1.0 before adding the required format tag (note there is other non-required metadata available that could be used to resolve the value):

id: ElectionBerkeleyMeasureS
tile_info:
  format: png
  encoding: ''
tilejson:
  tilejson: 3.0.0
  tiles: []
  attribution: Stephanie May ~ mizmay.com
  bounds:
  - -122.3263
  - 37.8448
  - -122.2012
  - 37.9063
  center:
  - -122.2747
  - 37.8731
  - 13
  description: 'MEASURE S: "Shall an ordinance prohibiting sitting on sidewalks in commercial districts from 7:00 am to 10:00 pm, with exceptions for: (a) medical emergencies; (b) wheelchairs and similar mobility devices; (c) bus benches; (d) street events; (e) other furniture placed on the sidewalk pursuant to a permit; requiring the City to ensure that it is applied in a constitutional manner and requiring a warning prior to citation, be approved?" '
  legend: "<div class='my-legend'>\n<div class='legend-title'>Measure S - Berkeley, CA</div>\n<div class='legend-scale'>\n<div class='legend-subtitle'>Yes Votes</div>\n  <ul class='legend-labels'>\n    <li><span style='background:#FFFFB2;'></span>7 - 30%</li>\n    <li><span style='background:#FECC5C;'></span>31 - 40%</li>\n    <li><span style='background:#FD8D3C;'></span>41 - 50%</li>\n    <li><span style='background:#F03B20;'></span>51 - 60%</li>\n    <li><span style='background:#BD0026;'></span>61 - 100%</li>\n  </ul>\n<div class='legend-source'>Source: <a href=\"http://www.acgov.org/rov/current.htm\">Alameda County Registrar of Voters</a></div>\n</div>\n\n<style type='text/css'>\n  .wax-legends { \n    top:5px !important;\n    bottom: auto !important;\n    }\n  .wax-legend .legend-title {\n    text-align: left;\n    margin-bottom: 8px;\n    font-weight: bold;\n    font-size: 100%;\n    }\n  .wax-legend .legend-subtitle {\n    text-align: left;\n    margin-bottom: 3px;\n    font-weight: bold;\n    font-size: 85%;\n    }\n  .wax-legend .legend-text {\n    text-align: center;\n    margin-bottom: 6px;\n    font-size: 80%;\n    }\n  .wax-legend .legend-scale ul {\n    margin: 0;\n    padding: 0;\n    float: left;\n    list-style: none;\n    }\n  .wax-legend .legend-scale ul li {\n    display: block;\n    float: left;\n    width: 50px;\n    text-align: center;\n    font-size: 80%;\n    list-style: none;\n    }\n  .wax-legend ul.legend-labels li span {\n    display: block;\n    float: left;\n    height: 15px;\n    width: 50px;\n    }\n  .wax-legend .legend-source {\n    font-size: 70%;\n    color: #999;\n    clear: both;\n    }\n  .wax-legend a {\n    color: #777;\n    }\n</style"
  maxzoom: 16
  minzoom: 13
  name: Measure S - November 2012 Election - Berkeley, CA
  template: |-
    {{#__location__}}{{/__location__}}{{#__teaser__}}Precinct: <strong>{{{CPrecinct}}}</strong><br/>
    Yes Votes: <strong>{{{S_pct}}}%</strong><br/>
    <strong> {{{S_turnout}}}%</strong> of registered voters voted on Measure S.<br/>
    <style>
    .wax-tooltip {
      top: auto !important;
      bottom: 150px !important;
    }
    </style>{{/__teaser__}}{{#__full__}}Precinct: <strong>{{{CPrecinct}}}</strong><br/>
    Yes Votes: <strong>{{{S_pct}}}%</strong><br/>
    Registered Votors: <strong>{{{Registrati}}}</strong> <br/>
    Ballots Cast: <strong>{{{t_BallotsC}}}</strong><br/>
    Mail-in Votes: <strong>{{{m_MSVoted}}}</strong> <br/>
    Votes at Poll: <strong> {{{p_MSVoted}}} </strong><br/>
    Abstained on S: <strong>{{{t_MSNoVote}}}</strong><br/>
    <strong> {{{NoS_pct}}}%</strong> of voters skipped Measure S.<br/>

    <style>
    .wax-tooltip {
      top: auto !important;
      bottom: 150px !important;
    }
    </style>{{/__full__}}
  version: 1.0.0

bdon commented 3 months ago

MBTiles spec 1.1 and later have the format key: https://github.com/mapbox/mbtiles-spec/blob/master/1.1/spec.md

Because this is isolated to only 1.0, I am tempted to just detect version = 1.0 and exit with a suggestion of adding the metadata manually.

Other alternatives:

Use the tile_info key to derive png/jpg. If around the year 2011, the only program making MBTiles was TileMill, and it always added this key, maybe this is good enough?
Inspect the first tile in the tileset and sniff the PNG magic bytes to distinguish between jpeg/png - more annoying to implement

prusswan commented 3 months ago

Option to specify the format is better (out of all the supported image formats). In reality the spec could work with any format as long as the data is organized in the expected schema in a sqlite database.

bdon commented 3 months ago

Added a log message if no format metadata is detected in the MBTiles archive

bdon commented 3 months ago

Updated in 1.19.2

protomaps / go-pmtiles