more complex demotiles for benchmarks and debuging

xabbu42 commented 3 years ago

For the benchmarks and debug server we need styles and data independend of mapbox. For the debug server #88 uses https://demotiles.maplibre.org/style.json to fix this. Imho this style is too simple for debugging and it certainly is for the benchmarks in bench/.

Judging from recent commits to mapbox-native-gl an alternative would be to let people get an api key from https://cloud.maptiler.com/maps/ and then use that. This would be about equivalent as it was before the fork. But I wonder if it would be possible to host tiles and a style which are more complex (i.e. the OpenStreetMap) but limited to a small area and few zoom levels without any api key similar to the demotiles? This should be enough for simple debugging and certainly is enough for benchmarks. we could even fallback to the existing demotiles outside of that area and the zoom levels to avoid confusing people. So is this possible?

wipfli commented 3 years ago

For the docs website we use tiles from MapTiler. The API key here if fully functional and has as far as I know no domain restrictions, i.e. it can be used for local testing. Here is an example:

<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8" />
<title>Add a default marker</title>
<meta name="viewport" content="initial-scale=1,maximum-scale=1,user-scalable=no" />
<script src="https://unpkg.com/maplibre-gl@1.15.0/dist/maplibre-gl.js"></script>
<link href="https://unpkg.com/maplibre-gl@1.15.0/dist/maplibre-gl.css" rel="stylesheet" />
<style>
    body { margin: 0; padding: 0; }
    #map { position: absolute; top: 0; bottom: 0; width: 100%; }
</style>
</head>
<body>
<div id="map"></div>

<script>
var map = new maplibregl.Map({
container: 'map',
style:
'https://api.maptiler.com/maps/streets/style.json?key=get_your_own_OpIi9ZULNHzrESv6T2vL',
center: [12.550343, 55.665957],
zoom: 8
});

var marker = new maplibregl.Marker()
.setLngLat([12.550343, 55.665957])
.addTo(map);
</script>

</body>
</html>

wipfli commented 3 years ago

Regarding tiles with restricted coverage there were already some discussions in the past about this. If I am not wrong https://github.com/cigone-openindoor was interested in something like this once in the slack chat. Maybe you can work together on some GitHub hosted tiles for testing?

xabbu42 commented 3 years ago

Just to clarify, we are allowed to just use https://api.maptiler.com/maps/streets/style.json?key=get_your_own_OpIi9ZULNHzrESv6T2vL as default in the bench/ and debug/ directories? this would of course be the simplest solution for now.

wipfli commented 3 years ago

Let's ask Petr: @klokan the docs website API key get_your_own_OpIi9ZULNHzrESv6T2vL can also be used for MapLibre GL JS testing and benchmarking purposes, right?

klokan commented 2 years ago

I don't have a problem with this usage for this use case.

In the long term, it would better to have benchmarks dependent only on locally hosted cached tiles (MBTiles from OpenMapTiles - for example Zurich data from https://data.maptiler.com/downloads/tileset/osm/europe/switzerland/zurich/ which we typically use for this).

github-actions[bot] commented 2 years ago

This issue is stale because it has been open 180 days with no activity. Remove stale label or comment or this will be closed in 30 days.

xabbu42 commented 2 years ago

I think this should be kept open for the long term solution of locally hosted tiles.

klokan commented 2 years ago

Short answer: Yes, you can.

Long answer:

For benchmarks and automated tests whenever possible use demotiles.maplibre.org (or it's local copy inside CI with hosted MBTiles from release on github and something like tileserver-gl-light via npm or docker). That's why we have made these demo tiles... This is also in readme.md in their repo.

If you are after benchmarks you may anyway want to have a local server i guess - to have zero latency and no dependencies on network performance at all.

You could also use locally hosted free Zurich OpenMapTiles from https://data.maptiler.com/downloads/dataset/osm/europe/switzerland/zurich/ or generated on your own. This is what we typically use internally for our tests - a local MBTiles. It can be part of releases on demotiles (it is in fact AWS S3) and then test could run on local machine - to avoid network delays and hiccups.

For public demonstration purpose, documentation, debug or tests we have made the above mentioned MapTiler Cloud key.

It is intended for the official open-source maplibre/* projects - ideally with code in GitHub or running on gh-pages of the subprojects (exposed on MapLibre.org websites automatically).

Anybody reading this: For any other project or purpose, your applications using MapLibre or your commercial projects get your own FREE key at cloud.maptiler.com please.

We grant not billed open-source friendly keys to other projects too. Contact us and explain your project.

On Thu, Jul 8, 2021, 8:34 PM Oliver Wipfli @.***> wrote:

Let's ask Petr: @klokan https://github.com/klokan the docs website API key get_your_own_OpIi9ZULNHzrESv6T2vL can also be used for MapLibre GL JS testing and benchmarking purposes, right?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/maplibre/maplibre-gl-js/issues/198#issuecomment-876656569, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAOPFBKSN7Z54ALBJK4ZVTTWXVSHANCNFSM5ABIBOXQ .

HarelM commented 2 years ago

@xabbu42 where are we on this? Is there a reason to keep this issue open? Are we planning something in that area?

xabbu42 commented 2 years ago

I agree with @klokan that its best to locally host freely awailable tiles for benchmarks and debug pages. I started a little prototype and will make a draft pull request shortly for comments.

xabbu42 commented 2 years ago

after the discussion in #1552 I'm no longer sure about a consensus here. The options in my mind are:

keep as is with demotiles, I don't think that is very useful as the demotiles are too limited to manually test maplibre-gl-js
use the given api key also for debug pages
build a selfhosting solution with an external tileserver
build a selfhosting solution using static tile files

nyurik commented 2 years ago

I would vote for the last option -- once Martin is capable of that, we can simply point it to a local file and use it for all the testing and demoing

xabbu42 commented 2 years ago

As I understood @HarelM and @wipfli that is not what they meant, but I may be wrong. Anyway using Martin with a local file falls under the third option as I understand it.

HarelM commented 2 years ago

The simpler it is the better we are of. That's my 2 cents. We have a large burden of maintaining third party dependencies in this library which we are obligated due to security concerns. If we can avoid adding dependencies and keeping the DX simple this should be our goal, IMHO.

acalcutt commented 2 years ago

I was looking a bit today at what it would take to have a more detailed OpenMapTiles based tiles and style in our current demotiles github. I made this test, though its not quite working yet due to the pbfs not downloading with compression, which would likely work if it were at the demotiles url.

https://github.com/WifiDB/demotiles/tree/gh-pages/tiles-omt

First I used planetiler to generate 3 separate mbtiles files to compare the size. I made these files massachusetts.mbtiles 189MB switzerland.mbtiles 331MB austria.mbtiles 597MB

I then extracted each of these to flat pbf files using mb-util 278M ./massachusetts (30698 files) 361M ./switzerland (24282 files) 658M ./austria (49628 files)

I then made the fork of the demotiles repo and uploaded the austria files to the 'tiles-omt' directory. This was possibly the worst choice since it was biggest and had the most files, but I wanted to be able to match the demotiles terrain tiles. the pbf files are all small enough to upload to github, but it takes time to work with this high amount of tiles.

After getting these tiles uploaded, I realized I had no style. I added OSM Bright, being sure to keep the license included in the directory. I changed the urls for the glyphs and fonts. I used the existing font folder, but I uploaded sprite files I created based on open sources.

I finally got this into a state that I think it would work, but because of the way I am using githib urls, the Content-Encoding is wrong for the pbf files and they get an unsupported error (doh). I think this would work if it was at the real demotiles url, since it does with the other pbf based demotiles there.

So what did I learn from this test 1.) it would probably be possible to create a more detailed demotiles source on the github based tile server we have, though I can't test this through github hosted urls I am currently using.

2.) flat files a bit of a pain to work with. even the smallest area I did took 24,282 files. I know in the past when we did the terrain, the merge the github PR ui did not like working with a large number of files like this.

3.) We might need a clearer picture of what we need to be able to test. do we need a full 14 zoom levels? I know the switzerland one suggested doesn't have that many zoom levels, but i never found it good enough to test road level things with.

acalcutt commented 2 years ago

I've been messing around more with making a OpenMapTiles demotile and style that can sit in github and I finally got it working. I'm wondering what you all think about something like this for a better testing tile without changing much other than uploading some new demotiles and styles.

Right now I am testing in https://github.com/WifiDB/demotiles/tree/gh-pages/

I created two sample pages for testing https://wifidb.github.io/demotiles/tiles-omt/ (new omt demo tile with a osm-bright-gl style) https://wifidb.github.io/demotiles/terrain-tiles/ (same with terrain and hillshade added)

I made a planetiler export of the N047E011 region, extracted the files to pdb with mb-util, then uncompressed the pbf files and pit them in the 'tiles-omt' directory

./mb-util --image_format=pbf N047E011.mbtiles N047E011
gzip -d -r -S .pbf *
find . -type f -exec mv '{}' '{}'.pbf \;

I also loaded the mbtiles file in tileserver-gl and let it generate a proper tilejson which I also used.

For the style I used osm-bright and added in 3d buildings and house numbers. I also needed to include some fonts which this style used.

HarelM commented 2 years ago

Can you share the clone time of this project before and after this change? If it's not too long (up to 10 minutes) I think this is a good improvement. We can always add more specific tiles for specific use cases in this repo like we did in the past.

acalcutt commented 2 years ago

I'd say from a quick test, checkout pre change was about 30 seconds, checkout after change was 2 minutes.

HarelM commented 2 years ago

Sounds reasonable to me. Can you share a map with the styles and tiled so we can see how it looks?

acalcutt commented 2 years ago

there are two samples at these urls

https://wifidb.github.io/demotiles/tiles-omt/ (new omt demo tile with a osm-bright-gl style) https://wifidb.github.io/demotiles/terrain-tiles/ (same with terrain and hillshade added)

The only difference between this and the PR I submitted is the urls are pointed to my github pages url

xabbu42 commented 1 year ago

As the maptiler key stopped working, I looked into using the new demotiles for benchmarks and I don't think they are adequate yet.

We need the demotiles to cover all the tiles listed in https://github.com/maplibre/maplibre-gl-js/blob/main/test/bench/data/style-benchmark-locations.json. we can achieve this by changing/removing the locations in that file and by covering more areas and to larger zoomlevels.

From what I gathered from the comments in that geojson, the intend of the current list is to cover the following:

large city with many roads and buildings, using different scripts (the file currently uses Houston, New York, Paris, and Tokyo)
landuse and landuse with roads (Houston and Paris)
landcovers with low zoom (points in germany)
hillshading and contour (point in switzerland)
complex water polygon (point in finland)
large zoom in cities with many labels (zoom level 16 and also 17 for the overzoomed case, this are currently not used, but should be?)

I'd like to have the whole world up to some low zoom level in the demotiles anyway, what zoom level would be feasible for that? this may cover some of the used locations already.

we should add a second region with cjk labels, for example centered on Tokyo so that we can reuse these locations. maybe we should add a larger city than Innsbruck or even replace Innsbruck? we could use one of the cities already used to reuse some locations of the file.

what other regions do we want, how many can we afford?

I assume we should just adapt the zoomlevels of the high zoom locations to 14 and 15 respectively. Is this correct?

wipfli commented 1 year ago

I think for automated benchmarking we can just have a handful of tiles stored somewhere. There are already some .pbf files in the MapLibre GL JS repo for testing if I am not wrong.