WebP support for charts

[quantenschaum]

I just tried the following

1. take an mbtiles chart file
2. convert the binary blobs containing the image data to lossless WebP
3. drop this file into the charts folder of avnav
4. use the chart

:tada: And it works out of the box!

The webserver serves the files with png extension and as Content-Type: image/png but it still works (at least with Chrome and Firefox).

Using WebP results in a massive reduction of the file size of the chart files without any loss in quality (490MB -> 92MB, 19%).

It even kept working after converting the mbtiles to gemf (it's just binary blobs, the decoding happens in the browser).

The creation of the mbtiles files its actually not in the scope of avnav itself, but nonetheless I think this is worth mentioning. Unfortunately it does not seem to be possible to instruct mapproxy to write mbtiles containing WebP images.

[wellenvogel]

That's cool. For the mapproxy part: Maybe we could extent the plugin handling that I already have. For some sources you need a bit of python code to work properly...

[quantenschaum]

Since MapProxy is written in Python it might be possible. Encoding WebP requires more CPU power, but once it's in the cache that doesn't matter any longer. Decoding happens on the client anyways.

WebP GEMF files do even work on the android app (at least on my phone).

Another thing with MapProxy is, that it creates non-standard mbtiles with the y-axis inverted. According to the spec mbtiles use the TMS scheme.
This is always confusing. (for the record: TMS: y-axis up, sw origin, XYZ: y-axis down, nw origin)

WebP is a supported format im mbtiles according to the spec. OpenCPN does not seem to like it.

To be able to use mbtiles created by MP with other software, I added the -y option to my converter.

Trying to run your convert_mbtiles directly on mbtiles created by MP does not work (neither tms nor xyz mode)

[free-x]

for reading of xyz mbtiles you can choose corresponding option

[quantenschaum]

This the mbtiles-to-gemf-converter? OK, so that basically inverts the y axis. I used the command line script and it failed.

[free-x]

You need to go to Download->Chart->Click on Chart and see this option

[quantenschaum]

It's in the web ui. So, this is not (only) for downloading the mbtiles file, it is there to set the tiling scheme, such that avnav knows how to read the tiles.

Does it only set a flag or does it actually convert the file?

[free-x]

Only set option how to read mbtiles

[quantenschaum]

That's good to know. But I was also interested in creating mbtiles to be used in other software. Then you need to actually flip y in the file.

[free-x]

I have used for example https://github.com/mapbox/mbutil

mb-util xyz.mbtiles xyz mb-util --scheme=tms xyz tms.mbtiles

[quantenschaum]

Thanks, more tools. - The process is fairly simple, I cooked my own converter for now.

The issue was converting mapproxy mbtiles -> gemf, that did not work. (for use on android)

To my converter I also added

• reencoding images to another format
• discarding fully transparent tiles (then tiles from lower zoom levels get displayed in avnav)
• discarding small tiles like 1x1px placeholders (")

Depending on the data source that reduces the file size quite a lot.

[wellenvogel]

2 thoughts: (1) Maybe it could be nice to create a plugin with the converter so that it would be integrated with the new chart importer. It can be build to win against the internal mbtiles->gemf converter. (2) Maybe a PR for mapproxy to allow for switching the scheme of the mbtiles cache. And writing something to the metadata about the used scheme... Or again hooking into the mapproxy code.

[quantenschaum]

Extending MP for use in avnav which may also result in a PR for MP itself sounds good.

Interesting fact: If MP gets fed with WebP images from an upstream tile source it actually does store these WebP blobs in the mbtiles db and also serves them but with a wrong png extension and mime-type (the data actually is WebP). This works as long as the tiles are just passed through in MP's internal processing chain, converting from TMS->WMS, the images gets reencoded as png. So, the image handling lib (PIL) used by MP does support WebP.

Setting

services:
  wms:
    image_formats:
      - image/png
      - image/webp

causes MP to server WebP via WMS (but the demo web ui does not show this option). This does not work for TMS, though.

[quantenschaum]

There is the image option on the cache which allows to change the image format.

caches:
  c_foo:
    sources: [ s_foo,s_foo ]
    grids: [ GLOBAL_WEBMERCATOR ]
    cache:
      type: mbtiles
    image:
      format: webp

This allows the use of the webp file extension in the tile url. But the image is not converted from png to webp (using a png source upstream).

The demo web ui displays image format webp, but the layer in the viewer still uses png, so no tiles are displayed. It does not work avnav either because it also seems to use png hardcoded in the openlayers config.

You can force MP to reencode the tiles by setting multiple sources on the cache. Then MP has to merge the tiles and then it actually writes webp tiles to the cache (file as well as mbtiles). You can add the same source twice (downloading the tile two times). And there currently seems to be no way to enable lossless encoding.

[quantenschaum]

As workaround you can define a dummy source pointing to MP itself, with an invalid url always returning 404 and mapping this to a transparent tile.

sources:
  dummy:
    type: tile
    grid: GLOBAL_WEBMERCATOR
    url: "http://localhost:8004/dummy"
    on_error:
      404:
        response: transparent
caches:
  c_foo:
    sources: [ dummy,s_foo ]
    grids: [ GLOBAL_WEBMERCATOR ]
    cache:
      type: mbtiles
    image:
      format: webp