CLI tool to convert raster to pmtiles

[daniel-j-h]

The protomaps format allows for raster tile hierarchies.

We've been working with bundling up raster tiles recently and ended up using the python library from https://github.com/protomaps/PMTiles to do so (see also example in https://github.com/protomaps/PMTiles/blob/e1228f5df1a4ac852e0117dc08c8813089a5c8af/python/examples/create_raster_example.py).

What would have made that workflow simpler is a cli tool like pmtiles convert but instead of converting mbtiles into pmtiles it would take e.g. a slippy map tile hierarchy as in z/x/y.ext or a GeoTIF raster and turn it into a pmtiles file.

Should the convert sub-command be extended to support raster, too?

[bdon]

It's a good feature request - anything that interacts with GeoTIFF or raster formats will probably depend on GDAL, which is out of scope for go-pmtiles. I've heard the best outputs come from rasterio and rio-mbtiles here: https://docs.protomaps.com/pmtiles/create#geotiff after which you can use go-pmtiles convert

so maybe we just need to make a rio-pmtiles equivalent?

[bmcbride]

A rio-pmtiles equivalent to directly convert GeoTIFF to PMTiles would be awesome, especially if the functionality was able to be added to go-pmtiles, giving the user the option to convert either MBTiles or GeoTIFF.

[bdon]

It's unlikely this would be part of go-pmtiles because it would need to depend on GDAL. A python package installed through pip run via a rasterio command e.g. rio pmtiles ... I think would be the easiest path.

[bdon]

issue transferred to main PMTiles repository

[mabhub]

What would have made that workflow simpler is a cli tool like pmtiles convert but instead of converting mbtiles into pmtiles it would take e.g. a slippy map tile hierarchy as in z/x/y.ext or a GeoTIF raster and turn it into a pmtiles file.

Isn't it already kinda simple by the use of mb-util which allow to convert tiles file tree to mbtiles, then converting it to pmtiles with pmtiles convert?

$ mb-util my-tiles-root-directory my_tiles.mbtiles
$ pmtiles convert my_tiles.mbtiles my_tiles.pmtiles

[neodescis]

What would have made that workflow simpler is a cli tool like pmtiles convert but instead of converting mbtiles into pmtiles it would take e.g. a slippy map tile hierarchy as in z/x/y.ext or a GeoTIF raster and turn it into a pmtiles file.

Isn't it already kinda simple by the use of mb-util which allow to convert tiles file tree to mbtiles, then converting it to pmtiles with pmtiles convert?

$ mb-util my-tiles-root-directory my_tiles.mbtiles
$ pmtiles convert my_tiles.mbtiles my_tiles.pmtiles

This is the route I went to convert some terrain tiles. It is indeed pretty simple, but:

1. This is not an obvious path to take for someone who doesn't know anything about mbtiles
2. It seems bad for pmtiles to depend on mb-util like this to get archives created

[larsmaxfield]

@bdon what would you think of PR which implements a solution like how @neodescis and @mabhub describe above?

Specifically, converting raster to .pmtiles with a temporary .mbtiles file which is automatically created and deleted? I'm doing a similar approach in a script — I could look into writing a function in PMTiles.

[bdon]

@JesseCrocker contributed this script: https://gist.github.com/JesseCrocker/4fee23a262cdd454d14e95f2fb25137f

I would prefer a CLI that does not need an intermediate mbtiles because that will be slower for large archives. Could you try the Gist above?

[larsmaxfield]

@bdon thanks for the feedback. The gist wasn't applicable for me because my need is to convert a z/x/y.ext directory of tiles (a.k.a. "file tree" or "slippy map tile hierarchy").

I created a disk_to_pmtiles method which directly reads the tiles and writes them to PMTiles — here's the fork diff: https://github.com/protomaps/PMTiles/compare/main...larsmaxfield:PMTiles:feature/disk_to_pmtiles.

The disk_to_pmtiles method uses elements of the original disk_to_mbtiles from mbutil combined with your mbtiles_to_pmtiles. It supports different tiling schemes like disk_to_mbtiles does, and I added an 'auto' feature for maxzoom whereby the maximum zoom is determined from the directory.

Let me know if you're OK with a PR for this.

[bdon]

I originally stopped development on the python CLI functions because https://github.com/protomaps/go-pmtiles replaces it and should be much faster for large tilesets. disk_to_pmtiles sounds useful if you have a small to medium size tileset, but likely doesn't work well for tens of millions of tiles. Does that match your use case?

[larsmaxfield]

Ah I see. The Python functions fit nice into my Python-based pipeline.

Indeed my use case is for tilesets at most up to z=10. And indeed disk_to_pmtiles is not appropriate for large tilesets. It takes my computer roughly 7 minutes at z=9, which extrapolates to a few hundred hours for z=15.

[bdon]

Well I think it will be useful for the small use cases like that. Can you open the PR?

[larsmaxfield]

Nice. I agree.

[Fil]

Hello,

sorry this might be slightly out of scope, but I came here because I'm trying to make a better pipeline for https://madmeg.org/delizie/ and I figured pmtiles would allow me to go fully static — and get rid of the php script that serves the images from mbtiles.

The files are not georeferenced, but I slice them to fit them into a square "world" view and I can use any of leaflet, maplibre openlayers etc. to view them. (The current iteration is a custom script based on d3-tile, but I'll probably opt for maplibre in the future.)

I've tried a few things, but failed on all of them:

• I couldn't find the disk_to_pmtiles method (which has been merged it seems?).

• When I try Jesse's script I get an error: rasterio.errors.CRSError: CRS is invalid: None

• I've also tried using pmtiles convert to migrate my existing mbtiles to pmtiles, and I'm not sure if the result is correct or not. When I try to open the file in a browser with maplibregl, I get the following error: The source image could not be decoded. (I don't know if pmtiles.io supports raster files. And I haven't been able to find a manageable (small size) raster pmtiles sample file.)

Any hint re: any of these questions would help :)

thanks!

[larsmaxfield]

@Fil disk_to_pmtiles is in convert.py in the python package:

import pmtiles.convert

convert.disk_to_pmtiles(...)

[Fil]

Almost there.

First, use gdal_translate to create a fake geo reference on the file:

gdal_translate -a_srs EPSG:3857 \
  -a_ullr -20026376.39 18503050.6 20026376.39 -18503050.6 \
  ./source.tiff ./georef.tiff

the fake coordinates try to span the whole globe in webmercator, taking into account the original tiff's aspect ratio.

Then I use Jesse’s script:

uv run ./rio-to-pmtiles.py --min_zoom 0 --max_zoom 5 --resampling_method lanczos --image_type PNG ./georef.tiff out.pmtiles

with max-zoom computed as ceil(log2(max(width, height) / 512)) - 1.

I still have an issue with the source tiff when it's not square, with errors such as rio_tiler.errors.TileOutsideBounds: Tile(x=0, y=0, z=5) is outside bounds. Adding a try/except seems to be ok (the missing tiles are shown in grey in the viewer—probably transparent?).

It would be nice if the viewer could be set to use 512 tiles as x2 tiles on retina screens. Currently it's not, and I can't reach the quality I'm seeing in my custom system. But I guess this is only this particular viewer, not an issue with the format.