search

pelias / pip-service

Pelias point-in-polygon-service
https://pelias.io
MIT License
16 stars 18 forks source link

SQL database is locked #110

Closed jdomag closed 5 years ago

jdomag commented 5 years ago

I'm using pelias k8s setup and pip container after finishing sqlite files throws below error. I didn't see this happening when i used EmptyDir in pip-deployment, it started when i switched to k8s PVC. 

done downloading whosonfirst-data-admin-za-latest.db.bz2                                                                                                                                                                                      
done downloading whosonfirst-data-admin-us-latest.db.bz2                                                                                                                                                                                      
done downloading whosonfirst-data-latest.db.bz2                                                                                                                                                                                               
All done!                                                                                                                                                                                                                                     
{"level":"info","message":"starting with layers neighbourhood,borough,locality,localadmin,county,macrocounty,macroregion,region,dependency,country,empire,continent,marinearea,ocean","label":"wof-pip-service:master","timestamp":"2019-06-19
T10:59:39.869Z"}                                                                                                                                                                                                                              
pip-service is now running on port 3102                                                                                                                                                                                                       

/code/pelias/pip-service/node_modules/pelias-whosonfirst/src/components/sqliteStream.js:13                                                                                                                                                    
      .exec('CREATE INDEX IF NOT EXISTS spr_obsolete ON spr (is_deprecated, is_superseded)')                                                                                                                                                  
       ^                                                                                                                                                                                                                                      
SqliteError: database is locked                                                                                                                                                                                                               
    at new SQLiteStream (/code/pelias/pip-service/node_modules/pelias-whosonfirst/src/components/sqliteStream.js:13:8)                                                                                                                        
    at readSqliteRecords (/code/pelias/pip-service/node_modules/pelias-wof-admin-lookup/src/pip/readStream.js:21:10)                                                                                                                          
    at readData (/code/pelias/pip-service/node_modules/pelias-wof-admin-lookup/src/pip/readStream.js:42:5)                                                                                                                                    
    at Object.<anonymous> (/code/pelias/pip-service/node_modules/pelias-wof-admin-lookup/src/pip/worker.js:34:1)                                                                                                                              
    at Module._compile (internal/modules/cjs/loader.js:689:30)                                                                                                                                                                                
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:700:10)                                                                                                                                                                  
    at Module.load (internal/modules/cjs/loader.js:599:32)                                                                                                                                                                                    
    at tryModuleLoad (internal/modules/cjs/loader.js:538:12)                                                                                                                                                                                  
    at Function.Module._load (internal/modules/cjs/loader.js:530:3)                                                                                                                                                                           
    at Function.Module.runMain (internal/modules/cjs/loader.js:742:12)                                                                                                                                                                        
{"level":"error","message":"macrocounty worker exited unexpectedly with code 1, signal null","label":"wof-pip-service:master","timestamp":"2019-06-19T10:59:46.756Z"}                                                                         

/code/pelias/pip-service/node_modules/pelias-whosonfirst/src/components/sqliteStream.js:13                                                                                                                                                    
      .exec('CREATE INDEX IF NOT EXISTS spr_obsolete ON spr (is_deprecated, is_superseded)')
       ^
SqliteError: database is locked
    at new SQLiteStream (/code/pelias/pip-service/node_modules/pelias-whosonfirst/src/components/sqliteStream.js:13:8)
    at readSqliteRecords (/code/pelias/pip-service/node_modules/pelias-wof-admin-lookup/src/pip/readStream.js:21:10)
    at readData (/code/pelias/pip-service/node_modules/pelias-wof-admin-lookup/src/pip/readStream.js:42:5)
    at Object.<anonymous> (/code/pelias/pip-service/node_modules/pelias-wof-admin-lookup/src/pip/worker.js:34:1)
    at Module._compile (internal/modules/cjs/loader.js:689:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:700:10)
    at Module.load (internal/modules/cjs/loader.js:599:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:538:12)
    at Function.Module._load (internal/modules/cjs/loader.js:530:3)
    at Function.Module.runMain (internal/modules/cjs/loader.js:742:12)

/code/pelias/pip-service/node_modules/pelias-whosonfirst/src/components/sqliteStream.js:13
      .exec('CREATE INDEX IF NOT EXISTS spr_obsolete ON spr (is_deprecated, is_superseded)')
       ^
missinglink commented 5 years ago

Duplicate of https://github.com/pelias/whosonfirst/pull/431 and related issues.

jdomag commented 5 years ago

How do you then run PIP if sqlite approach is not working (it worked fine within whosonfirst importer) and standard curl | tar doesn't handle larger files like localities?

orangejulius commented 5 years ago

@jdomag, what we have done for geocode.earth is store our own copy of the sqlite database with the required additional index created (for example by running the WOF importer on it once), and using the imports.whosonfirst.datahost option to point to it in our production setup.