search

CartoDB / mobile-tile-packager

Packages data from CARTO map/dataset as mbtiles with vector tiles
6 stars 5 forks source link

Problems for working in a production environment #4

Open josemazo opened 5 years ago

josemazo commented 5 years ago

Hi! We are working with the mobile CARTO SDKs and for getting offline cartography we are using this tool, but we've encountered a few problems:

Don't hesitate in contact us again.

jaakla commented 5 years ago

Regarding point 3 above - can you explain what you mean exactly?

The tool has 2 methods, which one do you use? The tippecanoe method (via SQL API) has theoretical way to change layer name, using additional Tippecanoe parameter. Another method with Maps API method takes tiles from CARTO Maps API, from the server (using tilelive.copy tool), and the packager tool does not look into vector tile binaries.

josemazo commented 5 years ago

Sorry, I'll explain myself better.

We are using the second method, taking the tiles from the CARTO Maps API because we are using multiple layers. So, while using the mobile SDKs, we need to identify the layers inside the MBTiles. Our MBTiles are generated dynamically for each location of our customer (a bunch of department stores), so we can't identify the different layers inside each MBTiles. Would be possible to add the name of the layer in the .json file that is generated along the MBTiles, so we can associate the names with their IDs?

josemazo commented 5 years ago

About the metadata, the tool generates:

bounds: -180,-85.0511,180,85.0511
minzoom: 0
maxzoom: infinty

If we don't set manually the bounds and the zoom levels that we used for the file generation, the SDK tries to get the tiles for the whole world and hangs. So, it looks likes the metadata needs to be fixed, and I think even the SDK.

alasarr commented 5 years ago

When a tile has no geometry the tool responds with a 204 and the process fail. We think that the 204 tiles should be empty, or even ignored in the tiles table inside the MBTiles file.

Priority 1

Some clients can't set their maps as public, and we don't mind set our API keys in the tool for accessing private maps or datasets.

Priority 2

The layers are using their CARTO IDs, an UUID, for identifying them in the MBTiles file. We need their IDs for styling the layers in the mobile SDKs. Because the MBTiles generation is going to be dynamic, could we use their names as IDs? Because if not, we can't know to which table apply the styles

We can forget about this

Some metadata is missing the metadata table, like the json row. Also, the zoom and the bbox rows have wrong information. But it looks like the SDKs can work with this problem.

We can forget about this

jaakla commented 5 years ago

Some clients can't set their maps as public, and we don't mind set our API keys in the tool for accessing private maps or datasets

Please explain what is the exact situation and solution you expect here?

josemazo commented 5 years ago

For not making public our visualizations, even as a link, we prefer to provide a CARTO API key for reading private maps and obtaning the MBTiles file.

jaakla commented 5 years ago

@josemazo I did some testing and it seems that you cannot get data from CARTO Maps API via API if map is completely private. Map init just gives Unauthorized template instantiation as error, even with API_KEY in the URL. You need to have 'shared with a link' as minimum; and once you have this, it works even without API_KEY, so this should work without changes needed in the packager.

Alternative would be to scrape tile URLs used in Builder: right-click + open in new tab seems to work in Chrome browser. The replace png with mvt as format/extension in the URL, and zoom/x/y numbers with placeholder {z}/{x}/{-y} . Then use any tile scraper, for example https://github.com/oscarfonts/tilelive-tms (there are possible others) to get the tiles. The tile URL for private map should expire soon, so I don't think it works for automated way.

josemazo commented 5 years ago

Oops, bad news. Let us check if we can work with public with link maps. Thank you for check it!

jaakla commented 5 years ago

About 204 error (we should really have separate issues here): mapbox tilelive which is used internally to download map tiles is expecting 404 response 'no data' case, probably mapbox uses this. CARTO maps API uses 204 'no content' for same case instead, which makes more sense logically, but tilelive considers this as very bad error and stops downloading. So there are a couple of solutions, none of them is perfect: a) create fork and fix for tilelive, and use this. Tilelive itself needs just 1 code line change to fix, but using modified dependency version needs some nodejs level tweaking. b) create simple http proxy which converts 204 responses to 404, run this locally. It may need minimal code change for URL rewrite. c) change CARTO Maps API to respond with 404. Not very realistic as it may affect other users.

mtehver commented 5 years ago

About the metadata, the tool generates:

bounds: -180,-85.0511,180,85.0511
minzoom: 0
maxzoom: infinty

If we don't set manually the bounds and the zoom levels that we used for the file generation, the SDK tries to get the tiles for the whole world and hangs. So, it looks likes the metadata needs to be fixed, and I think even the SDK.

@josemazo A comment about SDK behaviour - SDK currently (4.1.4) uses 'bounds' metadata parameter but ignores 'minzoom' and 'maxzoom'. When you construct MBTilesTileDatasource, then SDK tries to detect minzoom and maxzoom by simply scanning the tiles. If you have very large file without additional 'zoom_level' index, this may take a while. Is this the behaviour you are seeing? I will add a issue to the SDK repo about SDK taking 'minzoom' and 'maxzoom' into account and when present, not scanning the tiles table. As a workaround, there is a fix though: MBTilesTileDataSource has additional constructor with explicit minzoom/maxzoom arguments. If you use explicit form, tiles are not scanned.

mtehver commented 5 years ago

@josemazo About the 204 issue. I pushed a preliminary hotfix for this, hope this is solved now.

josemazo commented 5 years ago

Hi @mtehver,

I'll answer you inline.

If you have very large file without additional 'zoom_level' index, this may take a while. Is this the behaviour you are seeing?

Well, the files weren't any big, About 100KB. And the application was completely frozen.

About the 204 issue. I pushed a preliminary hotfix for this, hope this is solved now.

I just used the new code and I got the same error. This is the response from the job:

{
    "id": "5",
    "username": "XYZ",
    "template": "tpl_XYZ",
    "minzoom": 15,
    "maxzoom": 18,
    "bounds": "XX,YY,XX,YY",
    "state": "failed",
    "created_at": "2018-12-17T12:24:49.830Z",
    "updated_at": "2018-12-17T12:24:51.205Z",
    "started_at": "2018-12-17T12:24:49.831Z",
    "error": "Error: Upstream error: https://jcamacho.carto.com/api/v1/map/jcamacho@9e20305d@44d41da5f3014e5c38e8c6e6c51914b2:1543237111772/17/63353/50837.mvt returned 204\n    at err404 (/usr/src/app/service.js:36:17)\n    at onGetMbtiles (/usr/src/app/service.js:56:16)\n    at /usr/src/app/node_modules/@mapbox/tilelive/lib/tilelive.js:355:29\n    at Scanline.done (/usr/src/app/node_modules/@mapbox/tilelive/lib/tilelive.js:383:17)\n    at Scanline.emit (events.js:182:13)\n    at /usr/src/app/node_modules/@mapbox/tilelive/lib/stream-scanline.js:105:24\n    at /usr/src/app/node_modules/@mapbox/tilelive/lib/stream-util.js:271:22\n    at /usr/src/app/node_modules/tilelive-http/index.js:168:16\n    at Request._callback (/usr/src/app/node_modules/tilelive-http/index.js:84:20)\n    at Request.self.callback (/usr/src/app/node_modules/request/request.js:187:22)",
    "failed_at": "2018-12-17T12:24:51.205Z"
}
mtehver commented 5 years ago

@josemazo Could you do a 'clean install' of the packager and try again? It seems that the local tilelive-http is not properly updated when you do 'git pull/npm install'.

Regarding the freezing - could you send us a sample .mbtiles file and a code snippet that freezed your app?

josemazo commented 5 years ago

Hi @mtehver, let me work on that. I'll send you the .mbtiles file as soon as possible.

josemazo commented 5 years ago

Hi again @mtehver. I'm having problems installing the dependencies there is any extra step for installing tilelive-http?

mtehver commented 5 years ago

@josemazo I made local copy of the tilelive-http and placed it under the libs, it includes the relevant 204 status code fix. packages.json references it locally and you do not need to install it separately. Doing a clean checkout of the project seems to work for me. But I am not really a nodejs developer and there could be some versionining or other issues I have overlooked.

josemazo commented 5 years ago

Hi @mtehver, so, after a huge fight with node and docker, we got the packager to work! Thanks a lot, no more 204s!