Slow generation of tiles using "node index.js export"

[dkasipovic]

I have a server with dual xeon cpu (8 cores with HT, 32 cores all in total), and 32 gig of ram, running Ubuntu 18.04 server. After downloading and running osm-for-my-country for Serbia using node index.js init "Europe, Serbia" and starting kosmtik and export, I only get output of about 1 tile per second.

Attached are the logs of top -b

Export has been running for few days now (done levels up to 15)

free -mh shows:

          total        used        free      shared  buff/cache   available

Mem: 31G 3.0G 15G 659M 12G 26G Swap: 8.0G 0B 8.0G

Attached is the 5 minute output of "top -b": top.log

Should this not be much faster on machine like this?

[gagan-bansal]

Postgres seems taking too much of processing almost all time 99% CPU. Is there any other service also using postgres?

[dkasipovic]

No, postgres is the one that came in your install.sh. Well, the truth is that I had to modify the install.sh to use the latest version of Postgres and gis, because version that install.sh tried to install was not available for 18.04

Postgres (I suppose kosmtik, actually) runs 4 parallel threads at any given moment, and each of those threads is a SELECT query, and they each keep their core at above 90% at all times.

[gagan-bansal]

I have created a droplet (8 GB RAM | 4 vCPUs | 160 GB | 5 TB) from digitalocean and installed osm-for-my-country. Yes I too had to do changes for Ubuntu 18.04 (I'll commit those changes)

Instead of postgresql-9.5-postgis-2.2 Installed postgresql-10-postgis-2.4 And osm2pgsql not taking default database as gis so changed that.

Initiated the project for Serbia and started the export command. I am getting the throughput of ~14 tiles per sec.

Enclosing the top log, CPU (all 4 cores) usage is more than 90% and postgres processes' (major 2 only) cpu usage is always less than 50%. I think node-mapnik is managing very well the postgres processes internally.

Here is output of free mh

              total        used        free      shared  buff/cache   available
Mem:           7.8G        677M        1.7G        139M        5.4G        6.7G
Swap:            0B          0B          0B

top.log

Can you please consult once with your system engineer, if he can throw some more light on this.

[gagan-bansal]

Try once the windows standard solution. Restart the postgres service

sudo systemctl stop postgresql

sudo systemctl start postgresql

[dkasipovic]

Yes, those are the changes I had to do too. Can you commit them and I'll try and install it on a separate VPS and see what I get?

[gagan-bansal]

Committed the changes.

[dkasipovic]

So I created new VPS, 8 cores and 32 gig of ram, and cloned and ran "bash install.sh"

After that, I ran "node index.js init -r 'Europe, Serbia'"

That first failed on pbf2osm because osmconvert was not installed, so I had to manually install osmctools and symlink osmconvert and osmfilter in the folder

After that it failed because osm2pgsql was not installed, so I installed it manually and symlinked it

After that it seemed that postgresql and postgis have not been installed at all, so I installed that manually too. Also, it seems that "postgresql-10-postgis-2.4-scripts" is required on 18.04 too so I installed that too (it was failing to add postgis extension in postgresql without it).

Then, I manually ran osm2pgsql and it imported everything and then I realized that because postgres was not installed before, I have to rerun init, so I did that :)

Then, I had to manually run openstreetmap-carto/get-shapefiles.sh because kosmtik sad that .shp files were missing. When I tried to run get-shapefiles.sh it was evident that zip/unzip was missing. After I got that solved, kosmtik started working.

Then, when I ran "node index.js export" I am getting output of about 15-18 tiles per second, which is huge.

Also, "demo" and "serve" are failing with error that "Buffer.alloc is not a function", which I just remembered was error on the old server too, so I had to update node to a version that supports Buffer.alloc.

Sidenote: Is it possible to regenerate final-tile-list to include levels 16 and 17 (it is now up to 15)?

[gagan-bansal]

I use python command to check the os name. On Ubuntu 18.04 there are chances that there is no python command. Its python3. Please create alias

alias python=python3

If you are working as sudo user please run following

update-alternatives --install /usr/bin/python python /usr/bin/python3 10

(source: https://stackoverflow.com/a/50331137/713573)

Or in install.sh change it to python3

After that please run again install.sh, I think somehow I should try to catch that in install script to use python or python3

[gagan-bansal]

Can you give me more log about

Also, "demo" and "serve" are failing with error that "Buffer.alloc is not a function", which I just remembered was error on the old server too, so I had to update node to a version that supports Buffer.alloc.

[gagan-bansal]

You can generate tile-list by changing export.myAreaMaxZoom to 17 from 15. And run init once again. Or if you want to dig in, I use osm-tile-list npm module to create tile list.

node node_modules/.bin/osm-tile-list --help

Store the resulted tile list file to data/my-area-tile-list.txt Then run export tiles.

[dkasipovic]

damir@osm:~/osm-for-my-country$ node index.js serve
pm2

Using config file: './config/default.json'

/home/damir/osm-for-my-country/cmd-serve.js:18
    if (err) throw err
             ^

Error: Command failed: pm2 start ./node_modules/.bin/http-server -i 0 --name tile-server --merge-logs --output /home/damir/osm-for-my-country/log/tile-server-out.log --error /home/damir/osm-for-my-country/log/tile-server-error.log -- /home/damir/osm-for-my-country/map-tiles -c  -p 4040
Buffer.alloc is not a function

    at ChildProcess.exithandler (child_process.js:213:12)
    at emitTwo (events.js:87:13)
    at ChildProcess.emit (events.js:172:7)
    at maybeClose (internal/child_process.js:827:16)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:211:5)
damir@osm:~/osm-for-my-country$ node index.js demo
pm2

Using config file: './config/default.json'

/home/damir/osm-for-my-country/cmd-demo.js:15
    if (err) throw err
             ^

Error: Command failed: pm2 start ./node_modules/.bin/http-server --name demo --merge-logs --output /home/damir/osm-for-my-country/log/demo-out.log --error /home/damir/osm-for-my-country/log/demo-error.log -- /home/damir/osm-for-my-country/demo -p 4141
Buffer.alloc is not a function

    at ChildProcess.exithandler (child_process.js:213:12)
    at emitTwo (events.js:87:13)
    at ChildProcess.emit (events.js:172:7)
    at maybeClose (internal/child_process.js:827:16)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:211:5)
damir@osm:~/osm-for-my-country$ node -v
v4.4.7

[dkasipovic]

And for tile generation, yes, I used osm-tile-list to generate levels 16 and 17 after the initial 15 were done.

Offline tile generation is keeping steady pace of about 15-18 tiles per second. I wonder if I could make it go 4 times as fast on that 32 core machine that I have :)

[gagan-bansal]

Try this to start tile server (serve)

./node_modules/.bin/http-server /home/damir/osm-for-my-country/map-tiles   -p 4040

[gagan-bansal]

Hope your issue is resolved.

Basically in serve and demp command I start http-server with pm2 (excellent framework to maintain services). You can learn easily (in less than 10 min) about pm2 and it would help in long run.