Tileserver-gl endpoints | Maputnik compatibility

[enkicoma]

• Clear description of the problem: The Tileserver-gl endpoint for Glyphs is not compatible with the maputnik editor, even if you use maputnik online which points to: https://api.maptiler.com/fonts/{fontstack}/{range}.pbf?key={key}, you get different results when hosting the same style locally. To keep things consistent I wanted to run Tilesetver-gl locally and host the Glyphs. Locally you can't use the URL of Tileserver-gl htts://localhost:8080/fonts/{fontstack}/{start}-{end}.pbf because maputnik will complain of range key missing.

• Version of the TileServer GL software you have used Latest (4.4.0)

• Version and name of the operating system you use or other details of your setup Ventura 13.1

• Information about your used config / styles / vector tiles Styles generated from maputnik online editor.

https://github.com/maptiler/tileserver-gl/blob/bb0cd60e6493e093703f0ec94dfb65c3a526ef52/docs/endpoints.rst#styles https://github.com/maptiler/tileserver-gl/blob/a7af45ee3f61032c82b339174ddc9c31c0be2f8c/src/serve_font.js#L41

I am not familiar with the entire source code, but will it be possible to create an endpoint that will look the same as the MapTiler team has implemented?: https://api.maptiler.com/fonts/{fontstack}/{range}.pbf

Endpoints I tried:

Data http://localhost:8080/styles/positron-gl-style.json http://localhost:8080/data/v3/{z}/{x}/{y}.pbf http://localhost:8080/data/v3.json

Glyphs / fonts . Pdf http://localhost:8080/styles/positron-gl-style/style.json http://localhost:8080/fonts/{fontstack}/{start}-{end}.pbf http://localhost:8080/fonts/{fontstack}/{range}.pbf http://localhost:8080/fonts.json http://localhost:8080/fonts/Metropolis Light/0-255.pbf

Sprites: 
http://localhost:8080/styles/positron-gl-style/sprite@2x.json http://localhost:8080/styles/positron-gl-style/sprite.json

These 2 tools are the only left Opensource Alternative... And hope they will remain compatible and alive!

[acalcutt]

If I try my style at ( https://tiles.wifidb.net/styles/WDB_OSM/style.json ) in the maputnik editor it seems to work,

I use this for glyphs and sprites in my style

   "sprite":"{styleJsonFolder}/sprite",
   "glyphs":"{fontstack}/{range}.pbf",

[enkicoma]

@acalcutt thanks for the prompt response

yes, i tried this configuration:

  "sources": {
    "v3": {
      "type": "vector",
      "url": "mbtiles://{v3}"
    }
  },
  "sprite": "{styleJsonFolder}/sprite",
  "glyphs": "{fontstack}/{range}.pbf",

And the Tileserver-gl loads the map, but is missing some of the styles / missing city name / text.

The same style.json when used with maputnik online editor it gives me the text / desired style.

When used with Tileserver-gl, the city data / text is missing, hence I wanted to host all the endpoints for: sprite, glyphs & data sources with Tileserver-gl.

maybe the problem are in my local fonts? because I am not sure what kind of fonts is at this host: https://api.maptiler.com/fonts/{fontstack}/{range}.pbf?key={key}

I used the OpenSource ones, tho look a bit outdated... Maybe thats why I am getting 2 diff outcomes...

https://github.com/openmaptiles/fonts/tree/gh-pages

[acalcutt]

ya, so any font you have in your style you need to have locally available in your fonts folder.

Are you using the docker version? Like in the demo file https://github.com/maptiler/tileserver-gl/releases/download/v1.3.0/test_data.zip you would set your fonts folder in your config.json and then put all the fonts your style uses in that fonts folder.

Most of the fonts I use came from https://github.com/openmaptiles/fonts/releases the releases has some premade fonts but the repo has conversion tools.

I also saw this recently, which seems really useful for creating the font files https://maplibre.org/font-maker/ , but i haven't yet tested it myself

[enkicoma]

@acalcutt yes, I am using Docker on OSX

docker run --rm -it -v $(pwd):/data -p 8080:8080 maptiler/tileserver-gl --mbtiles n***.mbtiles

Thanks for the releases, will give it a try!

[enkicoma]

@acalcutt nope :( the v2 didn't work

This is a bit weird because other cities/country names I can see (they are visible) But the city which I styled with maputnik (to apear the text at specific zoom level, size, color, etc) - is not showing...

[acalcutt]

Is it Chinese characters missing like in https://github.com/maptiler/tileserver-gl/issues/621 ?

[enkicoma]

@acalcutt very kind - nope, en and ro. And as I mentioned - I can see the names of the nearest cities that I didn't touch with maputnik... something maputnik online editor has, that I don't have locally with Tileserver-GL

[acalcutt]

Do you know which font those missing labels are using? is it ro labels missing or en ones too?

What's the style code for those labels look like?

[enkicoma]

@acalcutt yes, the online editor says: Metropolis Regular and Nato Sans Regular as you can see I have them both on my local

maybe it's the problem at source data layer?

[acalcutt]

Did you make sure to download the newer noto-sans.zip from those release at https://github.com/openmaptiles/fonts/releases

[enkicoma]

@acalcutt yes, I did download v2 and nato-sans

[acalcutt]

Just checking. I think the v2.0.zip has an older version, so just making sure you grabbed the newer version of noto-sans that is by itself

[enkicoma]

@acalcutt yes, I did

[acalcutt]

from what you are doing it sounds like it should work. it could still be some type of font issue though.

My only other idea would be maybe the metropolis fonts need updating? Maybe try taking the fonts you need from https://github.com/openmaptiles/fonts/tree/master/metropolis and run them through the online converter https://maplibre.org/font-maker/ ....at least then you would know you have up to date copies.

Can you post your style.json?

[acalcutt]

At the command prompt where you ran the docker command, do you see any font errors? usually if a font is missing it says something at the command line level.

[enkicoma]

@acalcutt I tried to convert but when I download the .zip is not doing anything when I try to unzip.. no folder or .pbf...

Some weird stuff, even can't read the error...

can't upload json (quite a long json..) ... here

do you have discord or email?

and no error on docker run command

Starting tileserver-gl v4.4.0
Using specified config file from config.json
Starting server
Listening at http://[::]:8080/
Startup complete
GET / 304 - - 61.978 ms
GET /index.css 304 - - 2.823 ms
GET /images/placeholder.png 304 - - 3.693 ms
GET /images/logo.png 304 - - 3.164 ms
GET /images/maptiler-logo.svg 304 - - 4.532 ms
GET /images/header-map-1280px.png 304 - - 1.826 ms
GET /fonts/OpenSans-Bold.ttf 304 - - 1.926 ms
GET /fonts/OpenSans-Regular.ttf 304 - - 2.217 ms
GET /styles/positron-gl-style/8/148/90.png 200 40272 - 575.505 ms
GET /styles/positron-gl-style/?vector 304 - - 19.816 ms
GET /maplibre-gl.css 304 - - 1.100 ms
GET /L.TileLayer.NoGap.js 304 - - 15.417 ms
GET /maplibre-gl-inspect.css 304 - - 13.926 ms
GET /leaflet.css 304 - - 18.031 ms
GET /leaflet.js 304 - - 7.317 ms
GET /leaflet-hash.js 304 - - 0.827 ms
GET /maplibre-gl.js 304 - - 0.450 ms
GET /maplibre-gl-inspect.min.js 304 - - 0.823 ms
GET /mapbox-gl-rtl-text.js 304 - - 0.567 ms
GET /styles/positron-gl-style/style.json 304 - - 3.998 ms
GET /data/v3.json 304 - - 2.458 ms
GET /styles/positron-gl-style/sprite@2x.png 304 - - 15.796 ms
GET /mapbox-gl-rtl-text.js 304 - - 1.908 ms
GET /styles/positron-gl-style/sprite@2x.json 304 - - 17.610 ms
GET /mapbox-gl-rtl-text.js 304 - - 1.229 ms
GET /mapbox-gl-rtl-text.js 304 - - 2.496 ms
GET /mapbox-gl-rtl-text.js 304 - - 4.323 ms
GET /mapbox-gl-rtl-text.js 304 - - 0.866 ms
GET /mapbox-gl-rtl-text.js 304 - - 0.810 ms
GET /fonts/Metropolis%20Regular,Noto%20Sans%20Regular/512-767.pbf 200 95625 - 23.380 ms
GET /fonts/Metropolis%20Regular,Noto%20Sans%20Regular/1024-1279.pbf 200 125242 - 33.749 ms
GET /fonts/Metropolis%20Regular,Noto%20Sans%20Regular/0-255.pbf 200 75833 - 35.885 ms
GET /fonts/Metropolis%20Regular,Noto%20Sans%20Regular/256-511.pbf 200 127305 - 47.539 ms
GET /fonts/Metropolis%20Light,Noto%20Sans%20Regular/0-255.pbf 200 74955 - 55.330 ms
GET /fonts/Metropolis%20Regular,Noto%20Sans%20Regular/8192-8447.pbf 200 34049 - 72.196 ms

[enkicoma]

@acalcutt zip works attached positron (4).json.zip

Look at "Ungheni" - it's missing the main city Label, should appear 2 + label for some park / lake name is missing too

[acalcutt]

If I load your style here the second Ungheni shows https://tiles.wifidb.net/styles/enkicoma/?raster#12/47.2116/27.8016

If you look at inspect on your openmaptiles v3 layer, do you see that point in your data? http://localhost:8080/data/v3/#11/47.2116/27.8016

[enkicoma]

@acalcutt yes, when inspect - data is there

yeah, the second Ungheni shows in maputnik as well - as I stated, may be something on my local conffiguration, if data is there, means remains, Glyphs or Sprites?

[enkicoma]

@acalcutt I think I found the issue... the problem is on "filter": if I remove

        ["all", ["!=", "capital", 2], ["==", "class", "city"], [">", "rank", 3]]

from "id": "place_city",

It will show me the second "Ungheni". Any ideas why ?

[acalcutt]

From your inspect screenshot it seems like your openmaptiles v3 source does not have rank.

If I look at my planetiler generated file I do have rank

[enkicoma]

@acalcutt yeah just noticed, you're right - I saw that even some ids are not consistent...

It should be just name not name:latin as my style.json can't find this id...

I thought openmaptiles uses the latest OSM data. I generated .mbtiles with: https://github.com/systemed/tilemaker http://download.geofabrik.de/europe.html

so I am a bit confused

means https://api.maptiler.com/tiles/v3/tiles.json?key={key} uses data from multiple sources... that's why I would like the local CLI Tileserver-GL to host all required endpoints without relying on something that I can't see in someone cloud/api.

Many thanks for your time and contribution, dude 🙏

[acalcutt]

My guess would be that tilemaker is making a mbtiles file with an older version of the OpenMapTiles schema, or an incomplete schema

I would try to generate an up to date mbtiles file with either the official OpenMapTiles tools or planetiler. I am using planetiler to make my tiles like this https://github.com/acalcutt/wdb-map-gen . you could generate just europe in a similar way.

java -Xmx45g \
    -XX:OnOutOfMemoryError="kill -9 %p" \
    -jar planetiler-dist-0.5-SNAPSHOT-with-deps.jar \
    --download --area=europe --fetch-wikidata \
    --mbtiles=$EXPORT_DIR/output.mbtiles \
    --nodemap-type=sparsearray --nodemap-storage=mmap --optimize_db=true

[acalcutt]

I was thinking another possibility could be tilemaker might not incorporate wikidata. I know both openmaptiles tools and planetiler fetch extra data from wikidata. I wasn't able to find anything about tilemaker using wikidata

[enkicoma]

@acalcutt again, you are on the right spot - it may be wikidata as I tried Planetiler and I got the same results as tilemaker.

docker run -e JAVA_TOOL_OPTIONS="-Xmx1g" -v "$(pwd)/data":/data ghcr.io/onthegomap/planetiler:latest --download --area=moldova  

And the new .mbtiles didn't give too much...

vs what Maputnik has:

[acalcutt]

try adding "--fetch-wikidata" to your planetiler command

[acalcutt]

Are you sure you have the right mbtiles in use there? I tried the command you posted and the file I got had the extra data

[enkicoma]

@acalcutt Looks like I used the planetiler generated one: http://localhost:8080/data/v3.json

the issue was in my browser... I had to open one in incognito mode... (cookies)

once again, thank you 🙏

[acalcutt]

awesome, good to hear.

I do notice that second point we talked about doesn't exist in the moldova area export, but I'm guessing that is just over the border, or something like that.

[enkicoma]

@acalcutt yeah border stuff - before I had to download (Ukraine, Romania and Moldova) OSM and use the box osmium extract --bbox=26.49,45.21,30.56,48.57 to include some things across the border, just FYI

[acalcutt]

This can be closed right? I think we have determined the styles for maputnik work with tileserver-gl (with maybe a few tweaks of sources/fonts). The main issue wasn't with fonts, but with the data mbtiles that was missing attributes your style used.

[acalcutt]

Just one more random fact before this closes.

In my style I wanted to prefer english, but I found sometimes the language names in the data were inconstant. Some were labeled under name_en, but others only had name:latin or name:en or name:nonlatin.

So I ended up changing the text-field in my style to use multiple case statements. It basically works like a hierarcy, if the first attribute exists, it uses that one, but if it doesn't it moves to the next case statement and tries to use the next language.

"text-field": ["case",["has","name_en"],["get","name_en"],["case",["has","name:en"],["get","name:en"],["case",["has","name:latin"],["get","name:latin"],["get","name:nonlatin"]]]],

This can be useful when there are differences in the data.

[enkicoma]

@acalcutt I raised this request to improve the Tileserver GL endpoint for Glyph because I wanted to host things locally and use maputnik CLI. (locally) That would give End to End experience on how to style a map with the locally available data, glyphs and fonts.

Not sure, should we close it? what do you think

[acalcutt]

I guess I don't see what is missing. The glyph is already providing something at the {fontstack}/{range}.pbf url.

[enkicoma]

@acalcutt which glyphs? Tileserver-GL has a diff endpoint {fontstack}/{start}-{end}.pbf this endpoint won;t work with maputnik, means there is no compatibility to host things locally between TIleserver-GL and maputnik.

[acalcutt]

Are you talking about in the built in default style? we have already shown {fontstack}/{range}.pbf works above didn't we? it depends what you put in your style.

Like my style here loaded fine in maputnik... https://tiles.wifidb.net/styles/WDB_OSM/style.json

[enkicoma]

@acalcutt I am talking about more Tileserver-GL with Maputnik compatibility. Yes, we proved that the style.js can be hosted with Tileserver-GL.

Scenario: We have already a Tileserver-GL that runs perfectly, but we received a client request to do some improvements on the map view / data. This means we will require again amap editor in our case Maputnik.

To edit the client style.js, we can use Maputnik Online Editor, but we won't get the same style / output, because the online editor has different data and styles, which we don't have control over or too much awareness of the updates, etc.

The ideal scenario would be to run editor and tileserver in-house / locally because we can iterate the data schema and add or remove data, fonts, styles, etc.

To run things in-house we need all INPUT URLs (endpoints) for the editor Maputnik: sprite, glyphs and data - Tileserver GL can hande and host all of these URLs. (If we fix the glyphs endpoint) from {fontstack}/{start}-{end}.pbf to {fontstack}/{range}.pbf https://github.com/maptiler/tileserver-gl/blob/a7af45ee3f61032c82b339174ddc9c31c0be2f8c/src/serve_font.js#L41

[acalcutt]

Functionally on the backend, I think {start}-{end} and {range} would be handled by the same regex. the {range} would get replaced with [number]-[number], which is the same as {start}-{end}

If mapnuk is showing {fontstack}/{start}-{end}.pbf, it would tell me thats what you have set in your local style folder

EDIT: looking at it again I am probably wrong about the regex.

[enkicoma]

@acalcutt maputnik CLI won't accept {fontstack}/{start}-{end}.pbf endpoint. It will say range missing:

https://github.com/maptiler/tileserver-gl/blob/master/docs/endpoints.rst

[acalcutt]

Where do you see {fontstack}/{start}-{end}.pbf used?

[acalcutt]

If you want to make a PR to change the documentation at https://github.com/maptiler/tileserver-gl/blob/master/docs/endpoints.rst to /fonts/{fontstack}/{range}.pbf I think that would be fine.

I looked at all the built in styles and they all seem to use {fontstack}/{range}.pbf (ex). Functionally I still think tileserver will accept both formats since they end up the same.

[enkicoma]

@acalcutt Tileserver-GL may accept {fontstack}/{range}.pbf format, from style.json but that doesn't mean the Tileserver-GL url: http://localhost:8080/fonts/{fontstack}/{range}.pbf will work - it gives you "not found", means such endpoint doesn't exist as an ENDPOINT.

The current documentation is correct, to access the Glyphs you have to access: fonts/{fontstack}/{start}-{end}.pbf endpoint. Something like this: /fonts/Metropolis Light/0-255.pbf and the .pbf will start to download in the browser.

Hope you understood my point. There is no such endpoint fonts/{fontstack}/{range}.pbf in Tileserver GL.

[acalcutt]

As fas as I'm aware, the url http://localhost:8080/fonts/{fontstack}/{range}.pbf should not work directly. The {fontstack} gets replaced with the font used in the layer and the {range} gets replaced with a range of unicode values. So the request to the server, like you said, would be something like http://localhost:8080/fonts/Metropolis Light/0-255.pbf . Looking at the code we have I'm guessing this happens on the maplibre-gl-js/maplibre-native side of things, outside tileserver.

So really I'm still not sure what you are asking for. If you know how to get it done, maybe a PR with some code would help understand.

[enkicoma]

@acalcutt will try to spend some time during this weekend and update on this.

[wecand0]

Hello, I know that this tileserver-gl repo, but in https://github.com/maplibre/maplibre-native there is no answer how to use glyphs locally without server, I have managed to use mbtiles locally, like this

  "sources": {
    "openmaptiles": {
      "tiles": ["mbtiles:///home/user/planet.mbtiles"],
      "type": "vector"
    }
  }

Maps are working, all great, but I STUCKED with glyphs.. How to open it from local storage, I tried like this: "glyphs": "asset://data/glyphs/{fontstack}/{range}.pbf" I tried also with full path: "glyphs": "asset:///home/app/data/glyphs/{fontstack}/{range}.pbf" Files *.pbf are correct I checked it with tileserver-gl, all fine. But I do not need to user server. The sample of error: Failed to load glyph range 64256-64511 for font stack Open Sans Semibold,Noto Sans Bold: I would be very happy if someone helps how to resolve this issue...

[acalcutt]

To use local fonts, you have to specify the glyph url like https://github.com/acalcutt/wifidb-tileserver-gl/blob/88d06f1b1650219434c3378a2a1bdbca00291c72/tileserver-gl/styles/WDB_OSM/style.json#L87

then in your congig file you have to specify where the fond files stick like https://github.com/acalcutt/wifidb-tileserver-gl/blob/88d06f1b1650219434c3378a2a1bdbca00291c72/tileserver-gl/configs/config-3.1.1.json#L6 (note, you would probably want to use relative paths to /data/ if you are using the docker, like the example style on the main page)

For the fonts theselves you need to get fonts in the right format, like by using https://maplibre.org/font-maker/ to create them or getting them from something like https://github.com/openmaptiles/fonts