Large mbtiles

[mantesat]

I'm using a 14GB mbtile file with tileserver and it seems it takes a lot of time to load the initial server page. With smaller files (around 1-2 GB) things are almost instant.

[klokan]

You have very probably incomplete MBTiles file - where some required fields are missing in the metadata table so the tileserver needs to detect it. Fix you file. If you use MapTiler (http://www.maptiler.com/) you would not have this problem.

See #34

[klokan]

BTW if I am wrong in any point @mantesat, please let me know and we reopen this ticket. ;-)

[mantesat]

Well, if I increase the timeout setting on the http server, the page finally loads, and the map works as expected. This mbtile file also works with tilemill. Are you sure this is not related with the file size?

[mantesat]

Also downloaded MapTiler free and tried to open the file. It opens normally. So this must be an issue of the initial tileserver webpage creation.

[klokan]

Well, I just expect that internal metadata table inside of your MBTiles files is incomplete - as described in #32 - where you also find command and open-source utilities how to fix it.

Unrelated: Viewing your existing MBTiles file in MapTiler Free does not imply anything in this case. You would need to generate the MBTiles files inside MapTiler. I expect you file is not generated by TileMill either. It was probably created with a different utility, right?

On Fri, Aug 7, 2015 at 1:55 PM, mantesat notifications@github.com wrote:

Also downloaded MapTiler free and tried to open the file. It opens normally. So this must be an issue of the initial tileserver webpage creation.

— Reply to this email directly or view it on GitHub https://github.com/klokantech/tileserver-php/issues/51#issuecomment-128682514 .

Petr Pridal, Ph.D. CEO

Klokan Technologies GmbH Hofnerstrasse 98, 6314 Unterageri, Switzerland Tel: +41 (0)41 511 26 12 Email: info@klokantech.com Web: http://www.klokantech.com/

[mantesat]

I believe it was generated with TileMill (didn't do it myself, a colleque gave me the file). Will try to check the metadata table as you suggested, and I'll let you know.

[mantesat]

Well, tried all sorts of tests, compared my files with TileMill files or mbutils generated files... metadata contain all necessary fields. I can't ignore that the larger the mbtiles file, the more the webpage takes to load (tried with 100MB, 1GB, 5GB, 15GB files). Note that by increasing the timeout parameter for my http server the page eventually loads as it should, and everything works as expected. It's just that the larger the files, the longer the delay needed for rendering the initial tileserver webpage.

[klokan]

What is result of command:

$ sqlite3 yourfile.mbtiles 'SELECT * FROM metadata'

for your file? I expect one of minzoom, maxzoom, format, bounds is missing.

This is an open-source project - you can examine the complete source code, identify the problem, fix it and submit a patch (pull request) so the community can benefit from your work and effort.

If you want us (@KlokanTech team) dig deeper into the problem we can also review the problem in detail and fix it as part of a contract.

[mantesat]

This is my output on the sqlite3 query:

$ sqlite3 ENCs.mbtiles 'select * from metadata';
bounds|18.281,34.312,30.932,42.033
center|23.8815,38.0545,8
minzoom|8
maxzoom|18
name|ENC Charts
description|
attribution|
template|
version|1.0.0
format|png

I believe no fields are missing. I also profiled the php script and I found that the delay is caused by a query: 'SELECT grid FROM grids LIMIT 1' which is located in tileserver.php:548. I tried to run the following command script for my file:

$ date;sqlite3 ENCs.mbtiles 'select grid from grids limit 1';date;

The result was:

Tue, Aug 11, 2015 11:17:09 AM
Tue, Aug 11, 2015 11:20:22 AM

So this means that this query causes the entire delay. Any ideas about this?

I'm not that proficient in PHP/sqlite3, if I was (and I had the available time), I could give it a try to dig more into it.

[klokan]

Interesting.

Could you please try the speed of: SELECT name FROM sqlite_master WHERE name='grids';

(tip from http://stackoverflow.com/questions/1601151/how-do-i-check-in-sqlite-whether-a-table-exists)

[mantesat]

Tried it, doesn't return anything. Opened the file using a visual sqlite viewer and saw that 'grids' is actually a view, not a table. Does this mean that the server script probably creates the view when the select query is executed?

Looking a little more, I saw that this is only used for UTFGrids. I suppose that if this functionality is disabled, things should be normal.

[klokan]

Could you please try this file: https://raw.githubusercontent.com/klokantech/tileserver-php/large-mbtiles/tileserver.php ?

It should be fast.

[mantesat]

Yes, it's very quick now, loads the webpage with my 15GB file instantly! Good job, thanks for looking into this. Glad I could help.