kosmtik / kosmtik

Make maps with OpenStreetMap and Mapnik
Other
724 stars 84 forks source link

kosmtik on pm2 #160

Closed gagan-bansal closed 8 years ago

gagan-bansal commented 8 years ago

I am trying to run kosmtik on pm2. Its starting properly but getting error in between. Enclosing the logs herewith. Its interesting if we run kosmtik on pm2.

node.js v4.4.7 starting kosmtik with pm2

cd ~/kosmtik
pm2 start index.js -i 0 --name="kosmtik" -- serve ~/openstreetmap-carto/project.yaml --host 0.0.0.0

pm2 logs output:

PM2: 2016-08-19 11:46:44: [PM2][WORKER] Started with refreshing interval: 30000
PM2: 2016-08-19 11:46:44: [[[[ PM2/God daemon launched ]]]]
PM2: 2016-08-19 11:46:44: BUS system [READY] on port /home/gaganb/.pm2/pub.sock
PM2: 2016-08-19 11:46:44: RPC interface [READY] on port /home/gaganb/.pm2/rpc.sock

[PM2] Streaming realtime logs for [all] processes 

PM2 Starting execution sequence in -cluster mode- for app name:kosmtik id:0
PM2 App name:kosmtik id:0 online
PM2 Starting execution sequence in -cluster mode- for app name:kosmtik id:1
PM2 App name:kosmtik id:1 online
PM2 Starting execution sequence in -cluster mode- for app name:kosmtik id:2
PM2 App name:kosmtik id:2 online
PM2 Starting execution sequence in -cluster mode- for app name:kosmtik id:3
PM2 App name:kosmtik id:3 online
kosmtik-0 [Core] Loading config from /home/gaganb/.config/kosmtik.yml
kosmtik-0 [Core] Loading plugin from ../plugins/base-exporters/index.js
kosmtik-0 [Core] Loading plugin from ../plugins/hash/index.js
kosmtik-0 [Core] Loading plugin from ../plugins/local-config/index.js
kosmtik-0 [Core] Loading plugin from ../plugins/datasource-loader/index.js
kosmtik-0 [Core] Loading plugin from kosmtik-tiles-export
kosmtik-0 [Core] Loading project from /home/gaganb/openstreetmap-carto/project.yaml
kosmtik-0 [Core] PreviewServer started, you can browse http://0.0.0.0:6789
kosmtik-0 [Local Config] Patched config from /home/gaganb/openstreetmap-carto/localconfig.json
kosmtik-1 [Core] Loading config from /home/gaganb/.config/kosmtik.yml
kosmtik-1 [Core] Loading plugin from ../plugins/base-exporters/index.js
kosmtik-1 [Core] Loading plugin from ../plugins/hash/index.js
kosmtik-1 [Core] Loading plugin from ../plugins/local-config/index.js
kosmtik-1 [Core] Loading plugin from ../plugins/datasource-loader/index.js
kosmtik-1 [Core] Loading plugin from kosmtik-tiles-export
kosmtik-2 [Core] Loading config from /home/gaganb/.config/kosmtik.yml
kosmtik-2 [Core] Loading plugin from ../plugins/base-exporters/index.js
kosmtik-2 [Core] Loading plugin from ../plugins/hash/index.js
kosmtik-2 [Core] Loading plugin from ../plugins/local-config/index.js
kosmtik-2 [Core] Loading plugin from ../plugins/datasource-loader/index.js
kosmtik-2 [Core] Loading plugin from kosmtik-tiles-export
kosmtik-1 [Core] Loading project from /home/gaganb/openstreetmap-carto/project.yaml
kosmtik-3 [Core] Loading config from /home/gaganb/.config/kosmtik.yml
kosmtik-3 [Core] Loading plugin from ../plugins/base-exporters/index.js
kosmtik-3 [Core] Loading plugin from ../plugins/hash/index.js
kosmtik-3 [Core] Loading plugin from ../plugins/local-config/index.js
kosmtik-3 [Core] Loading plugin from ../plugins/datasource-loader/index.js
kosmtik-3 [Core] Loading plugin from kosmtik-tiles-export
kosmtik-0 [Core] Generating Mapnik XML…
kosmtik-0 [Core] Using mapnik version 3.0.0
kosmtik-2 [Core] Loading project from /home/gaganb/openstreetmap-carto/project.yaml
kosmtik-1 [Core] PreviewServer started, you can browse http://0.0.0.0:6789
kosmtik-1 [Local Config] Patched config from /home/gaganb/openstreetmap-carto/localconfig.json
kosmtik-2 [Core] PreviewServer started, you can browse http://0.0.0.0:6789
kosmtik-3 [Core] Loading project from /home/gaganb/openstreetmap-carto/project.yaml
kosmtik-2 [Local Config] Patched config from /home/gaganb/openstreetmap-carto/localconfig.json
kosmtik-3 [Core] PreviewServer started, you can browse http://0.0.0.0:6789
kosmtik-3 [Local Config] Patched config from /home/gaganb/openstreetmap-carto/localconfig.json
kosmtik-1 [Core] Generating Mapnik XML…
kosmtik-1 [Core] Using mapnik version 3.0.0
kosmtik-2 [Core] Generating Mapnik XML…
kosmtik-2 [Core] Using mapnik version 3.0.0
kosmtik-3 [Core] Generating Mapnik XML…
kosmtik-3 [Core] Using mapnik version 3.0.0
kosmtik-2 [Core] Loading map…
kosmtik-2 [Core] Map ready
kosmtik-2 [Core] Loading map…
kosmtik-2 [Core] Map ready
kosmtik-0 [Core] Loading map…
kosmtik-0 [Core] Map ready
kosmtik-0 [Core] Loading map…
kosmtik-0 [Core] Map ready
kosmtik-1 [Core] Loading map…
kosmtik-1 [Core] Map ready
kosmtik-1 [Core] Loading map…
kosmtik-1 [Core] Map ready
kosmtik-3 [Core] Loading map…
kosmtik-3 [Core] Map ready
kosmtik-3 [Core] Loading map…
kosmtik-3 [Core] Map ready

and here is the list of pm2 processes

pm2 ls
┌──────────┬────┬─────────┬──────┬────────┬─────────┬────────┬──────────────┬──────────┐
│ App name │ id │ mode    │ pid  │ status │ restart │ uptime │ memory       │ watching │
├──────────┼────┼─────────┼──────┼────────┼─────────┼────────┼──────────────┼──────────┤
│ kosmtik  │ 0  │ cluster │ 1613 │ online │ 0       │ 6m     │ 97.570 MB    │ disabled │
│ kosmtik  │ 1  │ cluster │ 1622 │ online │ 0       │ 6m     │ 96.813 MB    │ disabled │
│ kosmtik  │ 2  │ cluster │ 1641 │ online │ 0       │ 6m     │ 123.199 MB   │ disabled │
│ kosmtik  │ 3  │ cluster │ 1662 │ online │ 0       │ 6m     │ 95.445 MB    │ disabled │
└──────────┴────┴─────────┴──────┴────────┴─────────┴────────┴──────────────┴──────────┘

and error log:


smtik-1 (out): Postgis Plugin: no connection to the server
kosmtik-1 (out): in executeAsyncQuery Full sql was: 'SELECT ST_SRID("way") AS srid FROM planet_osm_polygon WHERE "way" IS NOT NULL LIMIT 1;'
kosmtik-1 (out):   encountered during parsing of layer 'landcover-low-zoom' in Layer
kosmtik-1 (out): Postgis Plugin: no connection to the server
kosmtik-1 (out): in executeAsyncQuery Full sql was: 'SELECT ST_SRID("way") AS srid FROM planet_osm_polygon WHERE "way" IS NOT NULL LIMIT 1;'
kosmtik-1 (out):   encountered during parsing of layer 'landcover-low-zoom' in Layer
kosmtik-1 (out): image_reader: can't determine type from input data

kosmtik-2 (out): image_reader: can't determine type from input data
kosmtik-2 (out): image_reader: can't determine type from input data
kosmtik-2 (out): Postgis Plugin: no connection to the server
kosmtik-2 (out): in executeAsyncQuery Full sql was: 'SELECT ST_SRID("way") AS srid FROM planet_osm_polygon WHERE "way" IS NOT NULL LIMIT 1;'
kosmtik-2 (out):   encountered during parsing of layer 'landcover-low-zoom' in Layer
kosmtik-2 (out): Postgis Plugin: no connection to the server
kosmtik-2 (out): in executeAsyncQuery Full sql was: 'SELECT ST_SRID("way") AS srid FROM planet_osm_polygon WHERE "way" IS NOT NULL LIMIT 1;'
kosmtik-2 (out):   encountered during parsing of layer 'landcover-low-zoom' in Layer
kosmtik-2 (out): Postgis Plugin: no connection to the server
kosmtik-2 (out): in executeAsyncQuery Full sql was: 'SELECT ST_SRID("way") AS srid FROM planet_osm_polygon WHERE "way" IS NOT NULL LIMIT 1;'
kosmtik-2 (out):   encountered during parsing of layer 'landcover-low-zoom' in Layer
kosmtik-2 (out): image_reader: can't determine type from input data
kosmtik-2 (out): image_reader: can't determine type from input data
kosmtik-2 (out): image_reader: can't determine type from input data
kosmtik-2 (out): image_reader: can't determine type from input data
kosmtik-2 (out): image_reader: can't determine type from input data
kosmtik-2 (out): image_reader: can't determine type from input data
kosmtik-2 (out): image_reader: can't determine type from input data

kosmtik-3 (out): image_reader: can't determine type from input data
kosmtik-3 (out): image_reader: can't determine type from input data
kosmtik-3 (out): image_reader: can't determine type from input data
kosmtik-3 (out): image_reader: can't determine type from input data
kosmtik-3 (out): image_reader: can't determine type from input data
kosmtik-3 (out): image_reader: can't determine type from input data
kosmtik-3 (out): image_reader: can't determine type from input data
kosmtik-3 (out): image_reader: can't determine type from input data
kosmtik-3 (out): image_reader: can't determine type from input data
kosmtik-3 (out): image_reader: can't determine type from input data
kosmtik-3 (out): image_reader: can't determine type from input data
kosmtik-3 (out): image_reader: can't determine type from input data
kosmtik-3 (out): image_reader: can't determine type from input data
kosmtik-3 (out): image_reader: can't determine type from input data
kosmtik-3 (out): image_reader: can't determine type from input data
kosmtik-3 (out): image_reader: can't determine type from input data
kosmtik-3 (out): image_reader: can't determine type from input data
kosmtik-3 (out): image_reader: can't determine type from input data
kosmtik-3 (out): image_reader: can't determine type from input data
kosmtik-3 (out): image_reader: can't determine type from input data

kosmtik-0 (out): Postgis Plugin: no connection to the server
kosmtik-0 (out): in executeAsyncQuery Full sql was: 'SELECT ST_SRID("way") AS srid FROM planet_osm_polygon WHERE "way" IS NOT NULL LIMIT 1;'
kosmtik-0 (out):   encountered during parsing of layer 'landcover-low-zoom' in Layer
kosmtik-0 (out): Postgis Plugin: no connection to the server
kosmtik-0 (out): in executeAsyncQuery Full sql was: 'SELECT ST_SRID("way") AS srid FROM planet_osm_polygon WHERE "way" IS NOT NULL LIMIT 1;'
kosmtik-0 (out):   encountered during parsing of layer 'landcover-low-zoom' in Layer
kosmtik-0 (out): Postgis Plugin: no connection to the server
kosmtik-0 (out): in executeAsyncQuery Full sql was: 'SELECT ST_SRID("way") AS srid FROM planet_osm_polygon WHERE "way" IS NOT NULL LIMIT 1;'
kosmtik-0 (out):   encountered during parsing of layer 'landcover-low-zoom' in Layer
kosmtik-0 (out): image_reader: can't determine type from input data
kosmtik-0 (out): image_reader: can't determine type from input data
kosmtik-0 (out): image_reader: can't determine type from input data
kosmtik-0 (out): image_reader: can't determine type from input data
kosmtik-0 (out): image_reader: can't determine type from input data
kosmtik-0 (out): image_reader: can't determine type from input data
kosmtik-0 (out): image_reader: can't determine type from input data
kosmtik-0 (out): image_reader: can't determine type from input data
kosmtik-0 (out): image_reader: can't determine type from input data
kosmtik-0 (out): image_reader: can't determine type from input data

kosmtik-0 (err): [httpserver] /openstreetmap-carto/tile/11/1254/805.png 200
kosmtik-0 (err): [httpserver] /openstreetmap-carto/tile/11/1254/809.png 200
kosmtik-0 (err): [httpserver] /openstreetmap-carto/tile/11/1254/813.png 200
kosmtik-0 (err): [httpserver] /openstreetmap-carto/tile/11/1254/817.png 200
kosmtik-0 (err): [httpserver] /openstreetmap-carto/tile/11/1254/821.png 200
kosmtik-0 (err): [httpserver] /openstreetmap-carto/tile/11/1254/825.png 200
kosmtik-0 (err): [httpserver] /openstreetmap-carto/tile/11/1254/829.png 200
kosmtik-0 (err): [httpserver] /openstreetmap-carto/tile/11/1254/832.png 200
kosmtik-0 (err): [httpserver] /openstreetmap-carto/tile/11/1254/837.png 200
kosmtik-0 (err): [httpserver] /openstreetmap-carto/tile/11/1254/841.png 200
kosmtik-0 (err): [httpserver] /openstreetmap-carto/tile/11/1254/845.png 200
kosmtik-0 (err): [httpserver] /openstreetmap-carto/tile/11/1254/849.png 200
kosmtik-0 (err): [httpserver] /openstreetmap-carto/tile/11/1254/853.png 200
kosmtik-0 (err): [httpserver] /openstreetmap-carto/tile/11/1254/857.png 200
kosmtik-0 (err): [httpserver] /openstreetmap-carto/tile/11/1254/861.png 200
kosmtik-0 (err): [httpserver] /openstreetmap-carto/tile/11/1254/865.png 200
kosmtik-0 (err): [httpserver] /openstreetmap-carto/tile/11/1254/869.png 200
kosmtik-0 (err): [httpserver] /openstreetmap-carto/tile/11/1254/873.png 200
kosmtik-0 (err): [httpserver] /openstreetmap-carto/tile/11/1254/877.png 200
kosmtik-0 (err): [httpserver] /openstreetmap-carto/tile/11/1254/881.png 200

kosmtik-1 (err): [httpserver] /openstreetmap-carto/tile/11/1254/867.png 200
kosmtik-1 (err): [httpserver] /openstreetmap-carto/tile/11/1254/871.png 200
kosmtik-1 (err): [httpserver] /openstreetmap-carto/tile/11/1254/875.png 200
kosmtik-1 (err): [httpserver] /openstreetmap-carto/tile/11/1254/879.png 200
kosmtik-1 (err): [httpserver] /openstreetmap-carto/tile/11/1254/883.png 200
kosmtik-1 (err): [httpserver] /openstreetmap-carto/tile/11/1254/887.png 200
kosmtik-1 (err): [httpserver] /openstreetmap-carto/tile/11/1254/891.png 200
kosmtik-1 (err): [httpserver] /openstreetmap-carto/tile/11/1254/895.png 200
kosmtik-1 (err): [httpserver] /openstreetmap-carto/tile/11/1254/899.png 200
kosmtik-1 (err): [httpserver] /openstreetmap-carto/tile/11/1254/903.png 200
kosmtik-1 (err): [httpserver] /openstreetmap-carto/tile/11/1254/907.png 200
kosmtik-1 (err): [httpserver] /openstreetmap-carto/tile/11/1254/911.png 200
kosmtik-1 (err): [httpserver] /openstreetmap-carto/tile/11/1254/915.png 200
kosmtik-1 (err): [httpserver] /openstreetmap-carto/tile/11/1254/919.png 200
kosmtik-1 (err): [httpserver] /openstreetmap-carto/tile/11/1254/923.png 200
kosmtik-1 (err): [httpserver] /openstreetmap-carto/tile/11/1254/927.png 200
kosmtik-1 (err): [httpserver] /openstreetmap-carto/tile/11/1254/931.png 200
kosmtik-1 (err): [httpserver] /openstreetmap-carto/tile/11/1254/935.png 200
kosmtik-1 (err): [httpserver] /openstreetmap-carto/tile/11/1254/939.png 200
kosmtik-1 (err): [httpserver] /openstreetmap-carto/tile/11/1254/943.png 200

kosmtik-2 (err): [httpserver] /openstreetmap-carto/tile/11/1254/870.png 200
kosmtik-2 (err): [httpserver] /openstreetmap-carto/tile/11/1254/874.png 200
kosmtik-2 (err): [httpserver] /openstreetmap-carto/tile/11/1254/882.png 200
kosmtik-2 (err): [httpserver] /openstreetmap-carto/tile/11/1254/878.png 200
kosmtik-2 (err): [httpserver] /openstreetmap-carto/tile/11/1254/885.png 200
kosmtik-2 (err): [httpserver] /openstreetmap-carto/tile/11/1254/889.png 200
kosmtik-2 (err): [httpserver] /openstreetmap-carto/tile/11/1254/893.png 200
kosmtik-2 (err): [httpserver] /openstreetmap-carto/tile/11/1254/896.png 200
kosmtik-2 (err): [httpserver] /openstreetmap-carto/tile/11/1254/900.png 200
kosmtik-2 (err): [httpserver] /openstreetmap-carto/tile/11/1254/904.png 200
kosmtik-2 (err): [httpserver] /openstreetmap-carto/tile/11/1254/908.png 200
kosmtik-2 (err): [httpserver] /openstreetmap-carto/tile/11/1254/912.png 200
kosmtik-2 (err): [httpserver] /openstreetmap-carto/tile/11/1254/916.png 200
kosmtik-2 (err): [httpserver] /openstreetmap-carto/tile/11/1254/920.png 200
kosmtik-2 (err): [httpserver] /openstreetmap-carto/tile/11/1254/924.png 200
kosmtik-2 (err): [httpserver] /openstreetmap-carto/tile/11/1254/928.png 200
kosmtik-2 (err): [httpserver] /openstreetmap-carto/tile/11/1254/932.png 200
kosmtik-2 (err): [httpserver] /openstreetmap-carto/tile/11/1254/936.png 200
kosmtik-2 (err): [httpserver] /openstreetmap-carto/tile/11/1254/940.png 200
kosmtik-2 (err): [httpserver] /openstreetmap-carto/tile/11/1254/944.png 200

kosmtik-3 (err): [httpserver] /openstreetmap-carto/tile/11/1254/868.png 200
kosmtik-3 (err): [httpserver] /openstreetmap-carto/tile/11/1254/872.png 200
kosmtik-3 (err): [httpserver] /openstreetmap-carto/tile/11/1254/884.png 200
kosmtik-3 (err): [httpserver] /openstreetmap-carto/tile/11/1254/876.png 200
kosmtik-3 (err): [httpserver] /openstreetmap-carto/tile/11/1254/880.png 200
kosmtik-3 (err): [httpserver] /openstreetmap-carto/tile/11/1254/888.png 200
kosmtik-3 (err): [httpserver] /openstreetmap-carto/tile/11/1254/892.png 200
kosmtik-3 (err): [httpserver] /openstreetmap-carto/tile/11/1254/897.png 200
kosmtik-3 (err): [httpserver] /openstreetmap-carto/tile/11/1254/901.png 200
kosmtik-3 (err): [httpserver] /openstreetmap-carto/tile/11/1254/905.png 200
kosmtik-3 (err): [httpserver] /openstreetmap-carto/tile/11/1254/909.png 200
kosmtik-3 (err): [httpserver] /openstreetmap-carto/tile/11/1254/913.png 200
kosmtik-3 (err): [httpserver] /openstreetmap-carto/tile/11/1254/917.png 200
kosmtik-3 (err): [httpserver] /openstreetmap-carto/tile/11/1254/921.png 200
kosmtik-3 (err): [httpserver] /openstreetmap-carto/tile/11/1254/925.png 200
kosmtik-3 (err): [httpserver] /openstreetmap-carto/tile/11/1254/929.png 200
kosmtik-3 (err): [httpserver] /openstreetmap-carto/tile/11/1254/933.png 200
kosmtik-3 (err): [httpserver] /openstreetmap-carto/tile/11/1254/937.png 200
kosmtik-3 (err): [httpserver] /openstreetmap-carto/tile/11/1254/941.png 200
kosmtik-3 (err): [httpserver] /openstreetmap-carto/tile/11/1254/945.png 200

[PM2] Streaming realtime logs for [all] processes
yohanboniface commented 8 years ago

Postgis Plugin: no connection to the server

Are you sure your psql credentials are OK?

image_reader: can't determine type from input data

Any missing shp or something?

gagan-bansal commented 8 years ago

Without pm2 its working fine. I requested many tiles continuously on kosmtik with pm2 from the script on server itself and I get such error. Now also on pm2 its fine, in between I am getting such error.

gagan-bansal commented 8 years ago

I am exporting tiles for given tile list, here is the code

var fs = require('fs')
var path = require('path')
var mkdirp = require('mkdirp')
var request = require('request')
var async = require('async')
var argv = require('yargs')
  .usage('Usage: $0 -u [str] -t [str] -o [str]')
  .option('u', {
    alias: 'baseURL',
    demand: true,
    default: 'http://127.0.0.1:6789/openstreetmap-carto/tile',
    describe: 'base url serving map tile',
    type: 'string'
  })
  .option('t', {
    alias: 'tileList',
    demand: true,
    describe: 'a file for tiles list',
    type: 'string'
  })
  .option('o', {
    alias: 'outPath',
    demand: true,
    desc: 'output tile directory',
    default: './',
    type: 'string'
  })
  .help()
  .argv
if(!isDir(argv.o)) return console.log('Not valid outPath: ' + argv.o)
fs.readFile(argv.t, 'utf8', function(err, data) {
  if (err) throw err
  var tiles = data.split('\n')
  tiles.pop()
  async.eachOfLimit(tiles,20, function(tile, index, callback) {
    request(
      argv.u + '/' + tile,
      {encoding:'binary'},
      function(err, resp, body) {
        if (err) return console.error('Error for: ' + tile + ': ' + err)
        var filePath = argv.o + '/' + tile
        var dirPath = path.dirname(filePath)
        if (!isDir(dirPath)) mkdirp.sync(dirPath)
        fs.writeFile(filePath, body, 'binary', function(err) {
          if (err) throw err;
          console.log(filePath)
          callback()
        })
      }
    )
  }, function(err) {
    if (err) {
      console.log('oh here is the error')
      console.error(err)
    } else {
      console.log('all done')
    }
  })
})

function isDir(dir) {
  try {
    return fs.statSync(dir).isDirectory()
  } catch (err) {
    return false
  }
}
yohanboniface commented 8 years ago

I am exporting tiles for given tile list, here is the code

Have you tried https://github.com/kosmtik/kosmtik-tiles-export ?

gagan-bansal commented 8 years ago

Yes I tried it, there was small issue for labels on the edge (partial labels) may be some configuration required for meta buffer. Another thing I thought to do the tile export as asynchronous and running kosmtik on multi core machine.

yohanboniface commented 8 years ago

Another thing I thought to do the tile export as asynchronous and running kosmtik on multi core machine.

The trunk version (not yet in npm, but I may release tomorrow) should already be multi core. You can control the number of workers with the --workers option.

gagan-bansal commented 8 years ago

Oh nice it will solve a lot. Then I think pm2 is not required to me the only purpose of pm2 was to run on multi core.

yohanboniface commented 8 years ago

Released.

gagan-bansal commented 8 years ago

Thanks Its working well. Creating separate issue for edge label.

yohanboniface commented 8 years ago

Yes, thanks.