Module breaks with error: Invalid tilesource protocol: tmsource

[mohsentaleb]

Although tilelive-tmsource is installed globally (as a proof, Tessera server works fine with command: tessera tmstyle://./test.tm2 in which test.tm2 is internally linked to a tm2source).

logs this error:

/AppData/Roaming/npm/node_modules/tl/lib/commands/render.js:95 throw err; ^ Error: Invalid tilesource protocol: tmsource:

[mojodna]

I don't think this will actually work, but it's worth a try. Can you explicitly require tilelive-tmsource?

tl render -r tilelive-tmsource ...

What's more likely is that modules.js is swallowing whatever error is being produced (though if it's working with tessera...). Can you try adding logging to modules.js: https://github.com/mojodna/tl/blob/master/lib/modules.js#L17 ?

(Revisiting that, I see a couple things that deserve improvement: specifically it doesn't check for null values returned from require().)

[mohsentaleb]

Already did tl render -r tilelive-tmsource ... but no luck.. Just put a console.log(e) where you mentioned in modules.js#17 and the output was:

{ [Error: Cannot find module 'cdbtiles'] code: 'MODULE_NOT_FOUND' }
{ [Error: Cannot find module 'mongotiles'] code: 'MODULE_NOT_FOUND' }
{ [Error: Cannot find module 'tilelive-blend'] code: 'MODULE_NOT_FOUND' }
{ [Error: Cannot find module 'tilelive-bridge'] code: 'MODULE_NOT_FOUND' }
{ [Error: Cannot find module 'tilelive-carto'] code: 'MODULE_NOT_FOUND' }
{ [Error: Cannot find module 'tilelive-csv'] code: 'MODULE_NOT_FOUND' }
{ [Error: Cannot find module 'tilelive-csvin'] code: 'MODULE_NOT_FOUND' }
{ [Error: Cannot find module 'tilelive-http'] code: 'MODULE_NOT_FOUND' }
{ [Error: Cannot find module 'tilelive-mapnik'] code: 'MODULE_NOT_FOUND' }
{ [Error: Cannot find module 'tilelive-merge'] code: 'MODULE_NOT_FOUND' }
{ [Error: Cannot find module 'tilelive-null'] code: 'MODULE_NOT_FOUND' }
{ [Error: Cannot find module 'tilelive-overlay'] code: 'MODULE_NOT_FOUND' }
{ [Error: Cannot find module 'tilelive-s3'] code: 'MODULE_NOT_FOUND' }
{ [Error: Cannot find module 'tilelive-solid'] code: 'MODULE_NOT_FOUND' }
[Error: The specified procedure could not be found.
~\AppData\Roaming\npm\node_modules\tilelive-tmsource\node_modules\
mapnik\lib\binding\node-v11-win32-ia32\mapnik.node]
{ [Error: Cannot find module 'tilelive-utfgrid'] code: 'MODULE_NOT_FOUND' }
[Error: The specified procedure could not be found.
~\AppData\Roaming\npm\node_modules\tilelive-vector\node_modules\ma
pnik\lib\binding\node-v11-win32-ia32\mapnik.node]
{ [Error: Cannot find module 'tilelive-xray'] code: 'MODULE_NOT_FOUND' }
[Error: The specified procedure could not be found.
~\AppData\Roaming\npm\node_modules\tilelive-tmsource\node_modules\
mapnik\lib\binding\node-v11-win32-ia32\mapnik.node]

[mojodna]

This is great, thank you.

Mapnik (and apparently other binary modules) behave strangely when there are multiple copies. One solution might be to install mapnik at the top level and then uninstalling and then reinstalling any of the tilelive modules.

The more fundamental problem may be the adoption of a dependency on a different version of mapnik to support the render command. I was going to suggest trying a version of tl from before the render command, was added, but that's counter-productive.

HOWEVER, I just thought of a work-around. Since tessera works, I'd suggest serving up your style with that and then rendering it using a tilejson+http: URL pointed to the tessera-hosted version.

[mohsentaleb]

Just tried uninstalling mapnik and then installing it at top level, also the third solution. Still no luck, throws the same errors as above.

I just need a simple way to export a TM2 project into an MBtiles file. the third solution makes the conversion stack a bit complex.

Does this module work on your machine BTW?

[mojodna]

Good question. I don't think I've tried it since adding rendering and I've been slammed the last couple days. I'll be able to give it a try tomorrow.

If you're just going to MBTiles:

tl copy tmstyle://... mbtiles://./archive.mbtiles

(Install tl@0.2.1 to prevent conflicts from the render command.)

[mojodna]

(This was originally written to render ("copy"; "render" in context here generates a single image) TM2 projects to MBTiles, so I need to make sure that works with the current version.)

[mohsentaleb]

@mojodna Installed @0.2.1. throws error while exporting. FATAL ERROR: JS Allocation failed - process out of memory

Also tried adding --max-old-space-size=2000 option to node but no luck.

[mojodna]

Ugh. Any chance you can share the project and the bounds / zoom you're trying to export?

[mohsentaleb]

@mojodna I wish, but the database is pretty huge, and contains sensitive data. The project is bound to a single city with not-too-vast bounds [51.091041, 35.509701, 51.626207, 35.8765]

The weird thing is, no matter which range of zoom levels I'm exporting it crashes when it exports 54 images to mbtiles with error FATAL ERROR: JS Allocation failed - process out of memory.

I Googled a bit about this error message and many of answers were regarding optimizing the code. Any ideas on that?

Update: monitored node.exe process to see its memory consumption and I think that's the problem. It goes up to 750MB and that's when your module crashes. Have you considered garbage collection to work properly in your code?

[mojodna]

Interesting (and no worries, I totally understand). I've seen that before, but not in this context. I suspect that it could be Mapnik (in which case I've got a utility I need to publish that I've been using to profile styles), but it's more likely mbtiles (specifically the streaming wrapper around it here: https://github.com/mojodna/tilelive-streaming/blob/master/lib/mbtiles.js).

I'm slowly getting a project together that I can use to try this myself. More progress once I'm back home tomorrow. Thanks for bearing with me and helping!

[mojodna]

I made various updates to tilelive modules today that should "prevent binary pwnage" and just successfully generated an mbtiles containing 340 tiles with no problems. (In other words, reinstall any tilelive modules to pick up the newer versions.)

Can you paste the output (which is just tile coordinates and byte sizes) from your crash? I'm wondering if could be related to the size of individual tiles and/or if requesting the tiles at the end of the list (or the ones that are next up) directly from tessera or mapbox studio will produce similar crashes...

[mohsentaleb]

Thanks! Could you also list the tilelive modules that you made updates to?

[springmeyer]

@mohsentaleb i see you are using a 32 bit install of node on windows. I would recommend using 64 bit since the memory limit for a 32 bit process is ~ 700 mb and not really enough for heavy tiling jobs.

[mohsentaleb]

@springmeyer I could, but when I tried to install 64bit versions on my 64bit machine a couple of days ago, I got stuck with bunch of dependency errors with windows binaries (e.g. mapnik). Unless you tell me it's been fixed.

I've been using Tilemill on 32bit windows since its early releases and been exporting large mbtiles files with no problem. I guess there must be something wrong with either new mapnik binaries or tilelive modules.

[mohsentaleb]

@mojodna @springmeyer I managed to install it on a 64bit windows. Memory consumption was around 1.5GB when it crashed. output: http://pastebin.com/uRAPrZSm

npm list -g --depth=0:

├── express@4.11.0
├── mapnik@3.1.4
├── mbtiles@0.7.7
├── tessera@0.5.1
├── tilelive@5.5.2
├── tilelive-tmsource@0.3.0
├── tilelive-tmstyle@0.4.0
├── tilelive-vector@3.2.1
└── tl@0.3.1

It exported ~130 tiles and I'm trying to export a 34,000+ tiles project!

[springmeyer]

@mohsentaleb I'm realizing you are running your own nodejs version installed from nodejs on windows (rather than say the Mapbox Studio desktop package). In this case if you want to use the latest node-mapnik 3.x series then you also need to (for the time being, until nodejs supports visual studio 2014/2015) run a custom built node.exe that is compatible with visual studio 2014. it is here: https://github.com/mapbox/node-cpp11. and you need to reinstall everything with npm by passing the --toolset=v140 flag. more details at https://github.com/mapnik/node-mapnik/wiki/WindowsBinaries.

insane huh? sorry I did not realize this before. studio handles all this internally but if you want to run outside studio then this is critical. otherwise you'll have mixed up binaries from different visual studio versions and the outcome is that memory will not be released correctly leading to the crash you are seeing.

[mohsentaleb]

@springmeyer No luck again. I just copied Mapbox's version of node.exe to node's default path and ran the app, crashed at 1.5GB memory usage.

UPDATE: Also tried the version compatible with VS2014 and installing all modules with --toolset=v140 ! nope, not working.

[wilhelmberg]

@mohsentaleb would you mind repeating the steps (Debug Mapbox Studio) we discussed in (Windows: Crash on Upload Source/Download Mbtiles) to see, if we can get any additional information?

[mohsentaleb]

@BergWerkGIS Sure, how can I do it in case of this command-line tool?

[wilhelmberg]

@mohsentaleb Download a Mapbox node.exeto a directory of your choice, 64bit preferred. https://github.com/mapbox/node-cpp11/blob/master/README.md#node-for-windows--visual-studio-2014-ctp-4--64-bit

In WinDbg: Executable: <PATH\TO\MAPBOX>\node.exe Arguments: "<PATH\TO\TL-CLONE-DIR>\bin\tl.js" <arg1> <arg2>

[mohsentaleb]

@BergWerkGIS Done. log: http://pastebin.com/qhHTnvQX dump file: http://mohsentaleb.com/tl-crash.rar

[wilhelmberg]

@mohsentaleb thanks for the dump file. @springmeyer FYI

---

tl;dr

@mohsentaleb Is it feasible for you to break the export into several steps? e.g.: split your area into four quadrants and then combine the mbtiles?

Reason: By using a smaller bounding box I could work arround that problem.

I suppose you have some dense data. e.g. footprints of houses. If you are rendering at zoom 8, these wouldn't even be visible, so they should not even be passed to the renderer. This reduces memory usage a lot!

Since you are using PostGIS you could use z() function (referenced in Windows: Crash on Upload Source/Download Mbtiles) to do this.

Another thing to try, is to reduce the number of attributes to the absolute minimum that is needed for your styling. They are passed around and consume memory, too.

---

I was able to replicate, these were my steps:

• Had node v0.10.33 64bit installed on the system

git clone https://github.com/mojodna/tl.git
cd tl
npm install
npm install mbtiles tilelive-tmstyle tilelive-mapbox tilejson tilelive-tmsource

• downloaded directly into tl folder
  • https://mapbox.s3.amazonaws.com/node-cpp11/v0.10.33/x64/node.exe
  • https://mapbox.s3.amazonaws.com/node-cpp11/v0.10.33/x64/node.pdb
• copied the fonts I used in my style from https://github.com/mapbox/mapbox-studio-default-fonts to a fonts directory within the style directory
• referenced the fonts directory within the style Map { font-directory: url("fonts/"); }
• closed Mapbox Studio
• removed the letter c from source: "tmsource://c/Users/... in project.yml in my style with a text editor
• executed:

node.exe bin\tl.js copy tmstyle://../../../basemap.at/basemap.tm2 mbtiles://./out.mbtiles

which instantly showed Error: bad allocation and finally failed after seven tiles with

events.js:72
        throw er; // Unhandled 'error' event
              ^
FATAL ERROR: Malloced operator new Allocation failed - process out of memory

Another run showed this output:

• using a smaller bouding box didn't expose that behavior

[mohsentaleb]

@BergWerkGIS Thanks for putting the time on this. It may sound weird, but I'm able to export an mbtiles file in TileMill v0.10.1with the exact same bounding box with the same exact layers, and exact same stylesheets. Does that help narrowing down the problem roots?

[wilhelmberg]

@mohsentaleb Great that it is working for you with TileMill. Does this solve your problem, or do you need further help?

TileMill and MapBox Studio are completely different under the hood and thus cannot be compared.

Thanks again for your input, it is important that I can reproduce the problem to get rid of it.

[mohsentaleb]

@BergWerkGIS No, I meant I had been using TileMill in the past with no problem. Migrated to Mapbox about 10 days ago and all these issues are coming out. I can't use TileMill to export retina tiles (@2x size)

[mohsentaleb]

@BergWerkGIS @springmeyer I mean there is some way that Mapbox currently renders the whole world, right? We're just talking about a small city here.

@springmeyer My steps on using this command line tools and the setup are almost the same as yours. Also I've spent a lot of time simplifying my SQL queries and I use a bunch of z() functions to reduce memory usage.

[wilhelmberg]

@mohsentaleb

Have you tried Scale Factor in TileMill? TileMill Docs: High-Resolution Tiles

The whole world can be rendered, because the underlying vector tiles have been highly optimized (ST_Simplify, geom && !bbox!, z(), labelgrid()) to just include the data (geometry- and attribute-wise), that is absolutely necessary to render them at each zoom level.

[mohsentaleb]

@BergWerkGIS Scale Factor will increase the thickness of lines, scale of icons, and size of text in a default 256x246 tile which is not exactly what I need. I just need 512x512 replica of default tiles, which I managed to export them with tl but it crashes and wont let me export all the tiles.

P.S. I made a report on how many rows are returned from each of my SQLqueries and found out that the most large data-set is 100,000 rows. which I think is nothing compared to a world-wide-query.

[wilhelmberg]

@mohsentaleb I suppose you want to target retina displays, right?

TileMill Docs: High-Resolution Tiles: "TileMill includes a number of features that make it easy to create high-resolution versions of map designs, for example to display on Apple Retina displays."

The thickness of lines and the scale of icons have to be altered for Retina tiles. That's the way it works. Otherwise your style will not look good on the client, if you produce 512px tiles and squish them into an area that's normally covered by 256px tiles.

Of course the whole world is not queried at once. Each query only covers the area of one tile. If there is too much data in this one query, you run out of memory.

Have you tried geom && !bbox!? There is a known problem on Windows and adding this to your query might really help.

[mohsentaleb]

@BergWerkGIS As far as I know, there's two methods for providing retina tiles for hi-res devices. 1.exporting 256x256 tiles with scale factor of 2 in Tilemill and adding option detectRetina in leaflet 0.7.x which does a trick by displaying tiles of zoomX in zoomX-1 so that texts are more readable.

2. exporting 512x512 tiles with @ 2x suffix which leaflet 0.8.x supports and displays them in their own zoomX placements. (just like providing @ 2x icons for retina displays)

Method 1 is not the best solution out there and it's just a hack.

look at these two tiles rendered for retina and non-retina displays: @ 2x: https://a.tiles.mapbox.com/v4/base.mapbox-streets+bg-e8e0d8_landuse_water_buildings_streets/5/20/12@2x.png?access_token=pk.eyJ1IjoibWFwYm94IiwiYSI6IlhHVkZmaW8ifQ.hAMX5hSW-QnTeRCMAy9A8Q

normal: https://a.tiles.mapbox.com/v4/base.mapbox-streets+bg-e8e0d8_landuse_water_buildings_streets/5/20/12.png?access_token=pk.eyJ1IjoibWFwYm94IiwiYSI6IlhHVkZmaW8ifQ.hAMX5hSW-QnTeRCMAy9A8Q

---

And no, I haven't used geom && !bbox!. What's all about them?

[mojodna]

Semi-off-topic: you can use 512x512 tiles with Leaflet 0.7 with minimal configuration; it does the right thing and displays 512x512px as 256x256pt:

https://github.com/mojodna/tessera/blob/master/public/index.html#L57-L74

geom && !bbox! will further filter the generated SQL query according to the bounding rectangle of the area being rendered (!bbox! is replaced with a geometry at query-time) so that only intersecting features will be returned.

[mojodna]

This may also be helpful in optimizing your style (despite its extreme lack of documentation and immaturity thus far):

https://github.com/mojodna/mapnik-query-analyzer

When run with node index.js <stylesheet.xml>, it will assemble queries that more or less match what Mapnik will generate (it doesn't filter the SELECT according to attributes referred to in the style yet and doesn't respect a couple of less-used Mapnik options). It will then run them and provide information about the amount of time each query took to execute (optimize the query / indexes to improve this), the number of rows returned (keep it small), as well as the number of vertices (simplify geometries to reduce this) and the average number of vertices per feature.

I haven't tried it with tm2sources, but it should work since it doesn't use styles for anything.

To change the tile coordinates that are used, change https://github.com/mojodna/mapnik-query-analyzer/blob/master/index.js#L31-L35

[wilhelmberg]

@mojodna sorry for hi-jacking this thread about TileMill and Mabox Studio

@mohsentaleb I just checked: you are right, TileMill only exports 256px tiles. Retina support was already worked on in 2012. So I thought that had been merged long ago, but it didn't happen. Sorry, no 512px tiles from TileMill.

[mojodna]

@BergWerkGIS no, please! I'm loving the detail and am following along fascinated.

[mojodna]

@mohsentaleb this is the local patch I'm using for retina tiles in TileMill. It works for the editor, but I'm not sure about exporting.

diff --git c/servers/Tile.bones w/servers/Tile.bones
index 957f5b6..460d2af 100644
--- c/servers/Tile.bones
+++ w/servers/Tile.bones
@@ -101,8 +101,9 @@ server.prototype.load = function(req, res, next) {
         pathname: path.join(settings.files, 'project', id, id + '.xml'),
         query: {
             updated:req.query.updated,
-            scale: +req.query.scale || 1.0,
-            metatile: req.query.metatile|0 || 2
+            scale: (+req.query.scale || 1.0) * 2,
+            metatile: req.query.metatile|0 || 2,
+            tileSize: 512
         },
         // Need not be set for a cache hit. Once the cache is
         // warmed the project need not be loaded/localized again.

[mojodna]

@mohsentaleb you may also be able to use tl to render retina tiles from a mapnik: source:

tl copy <opts> "mapnik://./stylesheet.xml?scale=2&tileSize=512" mbtiles://./archive.mbtiles

[mohsentaleb]

@mojodna

1. Thanks for introducing mapnik-analyzer. I'll have a look at it.
2. About the tl command with 512x512 output, I've figured it out before and I even exported some tiles. This tool is perfect for me in case mapnik wont give us headaches like this.
3. Patching Tilemill for exporting 512 tiles doesn't work.

[mohsentaleb]

@mojodna @springmeyer @BergWerkGIS I just ran a test on a pretty good machine (Centos 7 - 64bit - Xeon 2.4G, with 80GB of RAM). Replaced node with the one introduced in https://github.com/mapbox/node-cpp11/blob/master/README.md#downloads and again it gave me similar results compared to a 32bit windows machine with 4GB of RAM, crashed on ~1.5GB memory consumption.

I just wanted to test the workflow on a better hardware/environment to make sure it's not related to those factors.

Some facts and numbers regarding my project:

• My project contains 34,000 tiles. (According to what Tilemill exports for the given bbox)
• The most complex query I run returns 100,000 rows, which takes 0.20sec to complete.
• I've optimized my queries to SELECT only fields that are really used.
• I've used z() function to limit queries in needed zoom levels.

[wilhelmberg]

tl;dr

@mohsentaleb

If your queries are optimized well and don't return too much data for each tile, especially the first one (did you include geom && !bbox!?), you should be set with this workaround until we've tackled this at lower levels.

Please try the following:

• Add global.gc(); before console.log
• Execute node like this:

node.exe --expose-gc --always-compact bin\tl.js copy tmstyle://path/to/your.tm2 mbtiles://path/to/your.mbtiles

---

details

I spent the day debugging node and it's memory allocation. It seems that the garbage collector (GC) is too lazy with releasing memory. I've found out, that internally (for my test project) our native C++ node addons are using between 80 and 160MB only, but node/v8 is still holding on to already released memory. When trying to allocate even more memory is goes beyong its limits and crashes.

With --expose-gc and global.gc(); we force it, to release memory, when we need it.

Without the addtional parameters it goes straight to 1GB with the first two tiles and I was just lucky it didn't crash. This steep increase in memory usage is also due to the fact that I, on purpose, included too much data on low zoom levels (The first two tiles almost need 500MB to be rendered):

With --expose-gc --always-compact

[mohsentaleb]

@BergWerkGIS Thanks. I did what you told me here. It's very weird that the behavior in my case didn't change much.

The memory usage increases in a constant rate. The steep increase in the middle is not something that occurs every time. You can imagine it almost follows the pattern shown with red arrow, in my case of course.

[wilhelmberg]

@mohsentaleb strange.

Just to make sure, you

• used Mapbox node.exe 0.10.33 64bit
• added global.gc(); to <tl-dir>\lib\commands\copy.js
• started your tl copy with node.exe --expose-gc --always-compact bin\tl.js copy <your params>

There are two things that might be the cause of the spike in the middle:

• another program is allocating memory, e.g. PostGIS. You can also see that the base line of the memory usage after your copy is higher than before
• the export reaches a zoom level where more data has to be processed: the z() function delivers data that wasn't included before

Overall I think the memory usage looks good, if only the GC would do its work.

The only thing I can think about left for you to try: Split the copy process into several steps. Like divide your area in four quadrants and export zoom level by zoom level, or at least zoom level ranges, e.g. 8, 9-11, 12-18. That way node.exe exists in between the steps and memory is completely released.

[mohsentaleb]

@BergWerkGIS Yes, I did the exact same things you listed.

As I mentioned before, that spike in the middle was a one-time thing. It didn't repeat in the next iterations.

And about splitting the area I don't think that would work, because every time a crash happens, it's not because the area is big or something. It crashes whenever a total number of tiles is exported and memory is not released, Say about 200 tiles. My project consists of ~34,000 tiles. How much splitting shall I do? you do the math.

[wilhelmberg]

it's not because the area is big or something

It's a pity that you cannot provide a test case. There are a lot of reasons why something can go wrong. It could be that some your geometries are not valid (Have you tried ST_IsValid?). It could be that your geometries are too complex (ST_Simplify).

It could be that there are too many geometries within the area of one tile at low zoom levels. I still think your queries are returning too much data per tile.

You said your most complex query returns 100,000 rows. At z10 Tehran fits into 1 tile. When rendering z10 that would mean loading 100,000 geometries into memory, processing and rendering them. This is just not possible.

    +----------+   +----------+   +----------+   +----------+
    |          |   |  x       |   |     x    |   |  xx  xxxx|
    |     x    |   |      x   |   |       x  |   | xxxx   xx|
    |          |   |          |   | xx       |   | xx  x  xx|
    +----------+   +----------+   +----------+   +----------+
     z17 - good     z15 - good      z13 - ehh      z10 - FAIL

How much splitting shall I do? you do the math

I don't know, it depends on your data. Have you tried my suggestion above? It would also help to narrow down problem: e.g. if zoom 11-20 copy without a problem and zoom 10 crashes, then you know, that your queries for z10 return too much data.

Furthermore, you seem pretty tech savvy. It would just be a batch script to call tl copy in a loop.

[mohsentaleb]

Thank you @BergWerkGIS. I really appreciate that you've borne with me until now. I'll do a bunch of new tests on validating and simplifying geometries, and also splitting queries tomorrow and I'll update here.

[wilhelmberg]

@mojodna

Is there an easy way to switch tl copy from async to sync, in order to have a more reproduce-able flow of action? If so, could you provide a patch or a branch for my debugging?

[mojodna]

@BergWerkGIS making concurrency configurable, as in #1, so it could be set to 1 sounds like it would help, right?

I'll take a whack at implementing that when I get back tonight; in the meantime, manually changing the highWaterMark: 32 values for Readable, Writable, and Collector in https://github.com/mojodna/tilelive-streaming/blob/master/index.js to 1 should work.

[wilhelmberg]

@mojodna Thanks. That did it. Tiles are returned in the same order now.

[wilhelmberg]

@mohsentaleb there are some new resources available, that show you how to use reduce the data coming from PostGIS: https://www.mapbox.com/blog/postgis-sql-studio/

Another thing, that might help: is your data already in WebMercator (EPSG:3857) or do your queries at least return the data in WebMercator (ST_Transform)? That would lift the weight from Studio doing the re-projection on the fly.

[mohsentaleb]

@BergWerkGIS Thanks, that was a good read. I'll try to use them in my queries and see what happens. My data is in Google Mercator (SRID: 900913)

[wilhelmberg]

900913 is also ok.