search

pelias / whosonfirst

Importer for Who's on First gazetteer
MIT License
27 stars 43 forks source link

Unable to start pip after download with sqlite=true #454

Closed otbutz closed 5 years ago

otbutz commented 5 years ago

Executing the following snippet with sqlite and importPostalcodes enabled:

cd /opt/pelias/whosonfirst
npm run download
cd /opt/pelias/pip-service
npm start

The download finishes properly:

> pelias-whosonfirst@0.0.0-development download /opt/pelias/whosonfirst
> ./bin/download

Downloading whosonfirst-data-latest.db.bz2
Downloading whosonfirst-data-admin-ad-latest.db.bz2
Downloading whosonfirst-data-admin-ae-latest.db.bz2
Downloading whosonfirst-data-admin-af-latest.db.bz2
...
done downloading whosonfirst-data-admin-zm-latest.db.bz2
Downloading whosonfirst-data-admin-zw-latest.db.bz2
done downloading whosonfirst-data-admin-zw-latest.db.bz2
done downloading whosonfirst-data-admin-za-latest.db.bz2
done downloading whosonfirst-data-admin-us-latest.db.bz2
done downloading whosonfirst-data-latest.db.bz2
All done!

But the sqlite DBs seem to be locked:

/opt/pelias/pip-service/node_modules/pelias-whosonfirst/src/components/sqliteStream.js:13
      .exec('CREATE INDEX IF NOT EXISTS spr_obsolete ON spr (is_deprecated, is_superseded)')
       ^

/opt/pelias/pip-service/node_modules/pelias-whosonfirst/src/components/sqliteStream.js:13
      .exec('CREATE INDEX IF NOT EXISTS spr_obsolete ON spr (is_deprecated, is_superseded)')
       ^
SqliteError: database is locked
    at new SQLiteStream (/opt/pelias/pip-service/node_modules/pelias-whosonfirst/src/components/sqliteStream.js:13:8)
    at readSqliteRecords (/opt/pelias/pip-service/node_modules/pelias-wof-admin-lookup/src/pip/readStream.js:21:10)
    at readData (/opt/pelias/pip-service/node_modules/pelias-wof-admin-lookup/src/pip/readStream.js:42:5)
    at Object.<anonymous> (/opt/pelias/pip-service/node_modules/pelias-wof-admin-lookup/src/pip/worker.js:34:1)
    at Module._compile (internal/modules/cjs/loader.js:776:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:787:10)
    at Module.load (internal/modules/cjs/loader.js:653:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
    at Function.Module._load (internal/modules/cjs/loader.js:585:3)
    at Function.Module.runMain (internal/modules/cjs/loader.js:829:12)
SqliteError: database is locked
    at new SQLiteStream (/opt/pelias/pip-service/node_modules/pelias-whosonfirst/src/components/sqliteStream.js:13:8)
    at readSqliteRecords (/opt/pelias/pip-service/node_modules/pelias-wof-admin-lookup/src/pip/readStream.js:21:10)
    at readData (/opt/pelias/pip-service/node_modules/pelias-wof-admin-lookup/src/pip/readStream.js:42:5)
    at Object.<anonymous> (/opt/pelias/pip-service/node_modules/pelias-wof-admin-lookup/src/pip/worker.js:34:1)
    at Module._compile (internal/modules/cjs/loader.js:776:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:787:10)
    at Module.load (internal/modules/cjs/loader.js:653:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
    at Function.Module._load (internal/modules/cjs/loader.js:585:3)
    at Function.Module.runMain (internal/modules/cjs/loader.js:829:12)

/opt/pelias/pip-service/node_modules/pelias-whosonfirst/src/components/sqliteStream.js:13
      .exec('CREATE INDEX IF NOT EXISTS spr_obsolete ON spr (is_deprecated, is_superseded)')
       ^
SqliteError: database is locked
    at new SQLiteStream (/opt/pelias/pip-service/node_modules/pelias-whosonfirst/src/components/sqliteStream.js:13:8)
    at readSqliteRecords (/opt/pelias/pip-service/node_modules/pelias-wof-admin-lookup/src/pip/readStream.js:21:10)
    at readData (/opt/pelias/pip-service/node_modules/pelias-wof-admin-lookup/src/pip/readStream.js:42:5)
    at Object.<anonymous> (/opt/pelias/pip-service/node_modules/pelias-wof-admin-lookup/src/pip/worker.js:34:1)
    at Module._compile (internal/modules/cjs/loader.js:776:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:787:10)
    at Module.load (internal/modules/cjs/loader.js:653:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
    at Function.Module._load (internal/modules/cjs/loader.js:585:3)
    at Function.Module.runMain (internal/modules/cjs/loader.js:829:12)

/opt/pelias/pip-service/node_modules/pelias-whosonfirst/src/components/sqliteStream.js:13
      .exec('CREATE INDEX IF NOT EXISTS spr_obsolete ON spr (is_deprecated, is_superseded)')
       ^
SqliteError: database is locked
    at new SQLiteStream (/opt/pelias/pip-service/node_modules/pelias-whosonfirst/src/components/sqliteStream.js:13:8)
    at readSqliteRecords (/opt/pelias/pip-service/node_modules/pelias-wof-admin-lookup/src/pip/readStream.js:21:10)
    at readData (/opt/pelias/pip-service/node_modules/pelias-wof-admin-lookup/src/pip/readStream.js:42:5)
    at Object.<anonymous> (/opt/pelias/pip-service/node_modules/pelias-wof-admin-lookup/src/pip/worker.js:34:1)
    at Module._compile (internal/modules/cjs/loader.js:776:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:787:10)
    at Module.load (internal/modules/cjs/loader.js:653:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
    at Function.Module._load (internal/modules/cjs/loader.js:585:3)
    at Function.Module.runMain (internal/modules/cjs/loader.js:829:12)

/opt/pelias/pip-service/node_modules/pelias-whosonfirst/src/components/sqliteStream.js:13
      .exec('CREATE INDEX IF NOT EXISTS spr_obsolete ON spr (is_deprecated, is_superseded)')
       ^

/opt/pelias/pip-service/node_modules/pelias-whosonfirst/src/components/sqliteStream.js:13
      .exec('CREATE INDEX IF NOT EXISTS spr_obsolete ON spr (is_deprecated, is_superseded)')
       ^
SqliteError: database is locked
    at new SQLiteStream (/opt/pelias/pip-service/node_modules/pelias-whosonfirst/src/components/sqliteStream.js:13:8)
    at readSqliteRecords (/opt/pelias/pip-service/node_modules/pelias-wof-admin-lookup/src/pip/readStream.js:21:10)
    at readData (/opt/pelias/pip-service/node_modules/pelias-wof-admin-lookup/src/pip/readStream.js:42:5)
    at Object.<anonymous> (/opt/pelias/pip-service/node_modules/pelias-wof-admin-lookup/src/pip/worker.js:34:1)
    at Module._compile (internal/modules/cjs/loader.js:776:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:787:10)
    at Module.load (internal/modules/cjs/loader.js:653:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
    at Function.Module._load (internal/modules/cjs/loader.js:585:3)
    at Function.Module.runMain (internal/modules/cjs/loader.js:829:12)
SqliteError: database is locked
    at new SQLiteStream (/opt/pelias/pip-service/node_modules/pelias-whosonfirst/src/components/sqliteStream.js:13:8)
    at readSqliteRecords (/opt/pelias/pip-service/node_modules/pelias-wof-admin-lookup/src/pip/readStream.js:21:10)
    at readData (/opt/pelias/pip-service/node_modules/pelias-wof-admin-lookup/src/pip/readStream.js:42:5)
    at Object.<anonymous> (/opt/pelias/pip-service/node_modules/pelias-wof-admin-lookup/src/pip/worker.js:34:1)
    at Module._compile (internal/modules/cjs/loader.js:776:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:787:10)
    at Module.load (internal/modules/cjs/loader.js:653:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
    at Function.Module._load (internal/modules/cjs/loader.js:585:3)
    at Function.Module.runMain (internal/modules/cjs/loader.js:829:12)

/opt/pelias/pip-service/node_modules/pelias-whosonfirst/src/components/sqliteStream.js:13
      .exec('CREATE INDEX IF NOT EXISTS spr_obsolete ON spr (is_deprecated, is_superseded)')
       ^

/opt/pelias/pip-service/node_modules/pelias-whosonfirst/src/components/sqliteStream.js:13
      .exec('CREATE INDEX IF NOT EXISTS spr_obsolete ON spr (is_deprecated, is_superseded)')
       ^
SqliteError: database is locked
    at new SQLiteStream (/opt/pelias/pip-service/node_modules/pelias-whosonfirst/src/components/sqliteStream.js:13:8)
    at readSqliteRecords (/opt/pelias/pip-service/node_modules/pelias-wof-admin-lookup/src/pip/readStream.js:21:10)
    at readData (/opt/pelias/pip-service/node_modules/pelias-wof-admin-lookup/src/pip/readStream.js:42:5)
    at Object.<anonymous> (/opt/pelias/pip-service/node_modules/pelias-wof-admin-lookup/src/pip/worker.js:34:1)
    at Module._compile (internal/modules/cjs/loader.js:776:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:787:10)
    at Module.load (internal/modules/cjs/loader.js:653:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
    at Function.Module._load (internal/modules/cjs/loader.js:585:3)
    at Function.Module.runMain (internal/modules/cjs/loader.js:829:12)
SqliteError: database is locked
    at new SQLiteStream (/opt/pelias/pip-service/node_modules/pelias-whosonfirst/src/components/sqliteStream.js:13:8)
    at readSqliteRecords (/opt/pelias/pip-service/node_modules/pelias-wof-admin-lookup/src/pip/readStream.js:21:10)
    at readData (/opt/pelias/pip-service/node_modules/pelias-wof-admin-lookup/src/pip/readStream.js:42:5)
    at Object.<anonymous> (/opt/pelias/pip-service/node_modules/pelias-wof-admin-lookup/src/pip/worker.js:34:1)
    at Module._compile (internal/modules/cjs/loader.js:776:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:787:10)
    at Module.load (internal/modules/cjs/loader.js:653:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
    at Function.Module._load (internal/modules/cjs/loader.js:585:3)
    at Function.Module.runMain (internal/modules/cjs/loader.js:829:12)

/opt/pelias/pip-service/node_modules/pelias-whosonfirst/src/components/sqliteStream.js:13
      .exec('CREATE INDEX IF NOT EXISTS spr_obsolete ON spr (is_deprecated, is_superseded)')
       ^
SqliteError: database is locked
    at new SQLiteStream (/opt/pelias/pip-service/node_modules/pelias-whosonfirst/src/components/sqliteStream.js:13:8)
    at readSqliteRecords (/opt/pelias/pip-service/node_modules/pelias-wof-admin-lookup/src/pip/readStream.js:21:10)
    at readData (/opt/pelias/pip-service/node_modules/pelias-wof-admin-lookup/src/pip/readStream.js:42:5)
    at Object.<anonymous> (/opt/pelias/pip-service/node_modules/pelias-wof-admin-lookup/src/pip/worker.js:34:1)
    at Module._compile (internal/modules/cjs/loader.js:776:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:787:10)
    at Module.load (internal/modules/cjs/loader.js:653:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
    at Function.Module._load (internal/modules/cjs/loader.js:585:3)
    at Function.Module.runMain (internal/modules/cjs/loader.js:829:12)
2019-06-06T15:51:45.622Z - error: [wof-pip-service:master] neighbourhood worker exited unexpectedly with code 1, signal null

/opt/pelias/pip-service/node_modules/pelias-whosonfirst/src/components/sqliteStream.js:13
      .exec('CREATE INDEX IF NOT EXISTS spr_obsolete ON spr (is_deprecated, is_superseded)')
       ^
SqliteError: database is locked
    at new SQLiteStream (/opt/pelias/pip-service/node_modules/pelias-whosonfirst/src/components/sqliteStream.js:13:8)
    at readSqliteRecords (/opt/pelias/pip-service/node_modules/pelias-wof-admin-lookup/src/pip/readStream.js:21:10)
    at readData (/opt/pelias/pip-service/node_modules/pelias-wof-admin-lookup/src/pip/readStream.js:42:5)
    at Object.<anonymous> (/opt/pelias/pip-service/node_modules/pelias-wof-admin-lookup/src/pip/worker.js:34:1)
    at Module._compile (internal/modules/cjs/loader.js:776:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:787:10)
    at Module.load (internal/modules/cjs/loader.js:653:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
    at Function.Module._load (internal/modules/cjs/loader.js:585:3)
    at Function.Module.runMain (internal/modules/cjs/loader.js:829:12)

/opt/pelias/pip-service/node_modules/pelias-whosonfirst/src/components/sqliteStream.js:13
      .exec('CREATE INDEX IF NOT EXISTS spr_obsolete ON spr (is_deprecated, is_superseded)')
       ^

/opt/pelias/pip-service/node_modules/pelias-whosonfirst/src/components/sqliteStream.js:13
      .exec('CREATE INDEX IF NOT EXISTS spr_obsolete ON spr (is_deprecated, is_superseded)')
       ^

/opt/pelias/pip-service/node_modules/pelias-whosonfirst/src/components/sqliteStream.js:13
      .exec('CREATE INDEX IF NOT EXISTS spr_obsolete ON spr (is_deprecated, is_superseded)')
       ^
2019-06-06T15:51:45.627Z - error: [wof-pip-service:master] empire worker exited unexpectedly with code 1, signal null
SqliteError: database is locked
    at new SQLiteStream (/opt/pelias/pip-service/node_modules/pelias-whosonfirst/src/components/sqliteStream.js:13:8)
    at readSqliteRecords (/opt/pelias/pip-service/node_modules/pelias-wof-admin-lookup/src/pip/readStream.js:21:10)
    at readData (/opt/pelias/pip-service/node_modules/pelias-wof-admin-lookup/src/pip/readStream.js:42:5)
    at Object.<anonymous> (/opt/pelias/pip-service/node_modules/pelias-wof-admin-lookup/src/pip/worker.js:34:1)
    at Module._compile (internal/modules/cjs/loader.js:776:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:787:10)
    at Module.load (internal/modules/cjs/loader.js:653:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
    at Function.Module._load (internal/modules/cjs/loader.js:585:3)
    at Function.Module.runMain (internal/modules/cjs/loader.js:829:12)
SqliteError: database is locked
    at new SQLiteStream (/opt/pelias/pip-service/node_modules/pelias-whosonfirst/src/components/sqliteStream.js:13:8)
    at readSqliteRecords (/opt/pelias/pip-service/node_modules/pelias-wof-admin-lookup/src/pip/readStream.js:21:10)
    at readData (/opt/pelias/pip-service/node_modules/pelias-wof-admin-lookup/src/pip/readStream.js:42:5)
    at Object.<anonymous> (/opt/pelias/pip-service/node_modules/pelias-wof-admin-lookup/src/pip/worker.js:34:1)
    at Module._compile (internal/modules/cjs/loader.js:776:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:787:10)
    at Module.load (internal/modules/cjs/loader.js:653:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
    at Function.Module._load (internal/modules/cjs/loader.js:585:3)
    at Function.Module.runMain (internal/modules/cjs/loader.js:829:12)
SqliteError: database is locked
    at new SQLiteStream (/opt/pelias/pip-service/node_modules/pelias-whosonfirst/src/components/sqliteStream.js:13:8)
    at readSqliteRecords (/opt/pelias/pip-service/node_modules/pelias-wof-admin-lookup/src/pip/readStream.js:21:10)
    at readData (/opt/pelias/pip-service/node_modules/pelias-wof-admin-lookup/src/pip/readStream.js:42:5)
    at Object.<anonymous> (/opt/pelias/pip-service/node_modules/pelias-wof-admin-lookup/src/pip/worker.js:34:1)
    at Module._compile (internal/modules/cjs/loader.js:776:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:787:10)
    at Module.load (internal/modules/cjs/loader.js:653:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
    at Function.Module._load (internal/modules/cjs/loader.js:585:3)
    at Function.Module.runMain (internal/modules/cjs/loader.js:829:12)

/opt/pelias/pip-service/node_modules/pelias-wof-admin-lookup/src/pip/index.js:166
        throw `${layer} worker shutdown unexpectedly`;
        ^
neighbourhood worker shutdown unexpectedly
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! pelias-pip-service@0.0.0-development start: `./bin/start`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the pelias-pip-service@0.0.0-development start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

I tried it multiple times even with a ps -aux between download and start but it seems that all processes start by download have properly finished.

Are there any lock files i'm missing?

Edit: noticed that i posted parts the log of my first run in which i used the pip-service download instead of the whosonfirst one.

otbutz commented 5 years ago

Update: it seems to only crash on the first start. If i retry to start the pip-service it works.

Maybe it's related to #453

Edit: it is.... https://github.com/pelias/whosonfirst/pull/431

orangejulius commented 5 years ago

Hi @otbutz, We just merged https://github.com/pelias/whosonfirst/pull/458, which should make any issues with the database lock error go away. We're now creating the database index in a way that handles failure gracefully, so even though not all the distfiles from dist.whosonfirst.org have that index, it shouldn't cause any issues in the meantime.

Let us know if there are still any problems.

otbutz commented 5 years ago

Thanks, will give it a spin ;)

otbutz commented 5 years ago

It works flawlessly now :+1: