pelias / docker

Run the Pelias geocoder in docker containers, including example projects.
MIT License
315 stars 218 forks source link

address.db #271

Closed myrondeliverone closed 2 years ago

myrondeliverone commented 2 years ago

I ran the build script. I am working with the texas project. The script doesnt create the address.db. Because that file is missing I get can't find file error. Here is my .env file: COMPOSE_PROJECT_NAME=pelias DATA_DIR=/Applications/deliverone OPENADDRESSES_PARALLELISM=2

Here is my pelias.json file: { "logger": { "level": "info", "timestamp": false }, "esclient": { "apiVersion": "7.5", "hosts": [ { "host": "elasticsearch" } ] }, "elasticsearch": { "settings": { "index": { "refresh_interval": "10s", "number_of_replicas": "0", "number_of_shards": "1" } } }, "acceptance-tests": { "endpoints": { "docker": "http://api:4000/v1/" } }, "api": { "services": { "placeholder": { "url": "http://placeholder:4100" }, "pip": { "url": "http://pip:4200" }, "interpolation": { "url": "http://interpolation:4300" }, "libpostal": { "url": "http://libpostal:4400" } } }, "imports": { "adminLookup": { "enabled": true }, "blacklist": { "files": [ "/data/blacklist/osm.txt" ] }, "geonames": { "datapath": "/data/geonames", "countryCode": "US" }, "openstreetmap": { "download": [ { "sourceURL": "http://download.geofabrik.de/north-america/us/texas-latest.osm.pbf" } ], "leveldbpath": "/tmp", "datapath": "/data/openstreetmap", "import": [{ "filename": "texas-latest.osm.pbf" }] }, "openaddresses": { "datapath": "/data/openaddresses", "files": [ "us/tx/anderson.csv", "us/tx/andrews.csv", "us/tx/angelina.csv", "us/tx/aransas.csv", "us/tx/archer.csv", "us/tx/armstrong.csv", "us/tx/atascosa.csv", "us/tx/austin.csv", "us/tx/bailey.csv", "us/tx/bandera.csv", "us/tx/bastrop.csv", "us/tx/baylor.csv", "us/tx/bee.csv", "us/tx/bell.csv", "us/tx/bexar.csv", "us/tx/blanco.csv", "us/tx/borden.csv", "us/tx/bosque.csv", "us/tx/bowie.csv", "us/tx/brazoria.csv", "us/tx/brazos.csv", "us/tx/brewster.csv", "us/tx/briscoe.csv", "us/tx/brooks.csv", "us/tx/brown.csv", "us/tx/burleson.csv", "us/tx/burnet.csv", "us/tx/caldwell.csv", "us/tx/calhoun.csv", .......eetc lots of other files ] }, "polyline": { "datapath": "/data/polylines", "files": [ "extract.0sv" ] }, "whosonfirst": { "datapath": "/data/whosonfirst", "countryCode": "US", "importPlace": [ "85688753" ] }, "transit": { "datapath": "/data/transit", "feeds": [ ] }, "interpolation": { "download": { "tiger": { "datapath": "/data/tiger", "states": [ { "state_code": 48 } ] } } } } }

missinglink commented 2 years ago

I suspect this might be due to either out-of-date docker images or skipped build steps.

The step which produces address.db and street.db is: pelias prepare interpolation. Please run pelias compose pull to pull the latest docker images and try running pelias prepare interpolation again.

This command depends on the pelias download * steps and the pelias prepare polylines step, skipping these will result in failure.

If you're still not able to find the address.db afterwards then check pelias compose logs for more info.

note: If you paste large code blocks to Github again, please use backticks to make the code readable.

myrondeliverone commented 2 years ago

I went ahead and deleted everything and started over again. Same problem. I am in the Texas project. Even with everything new and running the commands you posted it still won't produce an address.db. I looked through the log but only found the errors where is can't find the address.db file. Any suggestions?

@map texas % pelias system env  
DATA_DIR is already set to '/Applications/deliverone/vetday' - this may cause the DATA_DIR specified in the .env to be ignored
[warn] skip setting environment var "DATA_DIR=/Applications/deliverone/vetday", already set "DATA_DIR=/Applications/deliverone/vetday"
CMD_READLINK=greadlink
TERM_PROGRAM=Apple_Terminal
SHELL=/bin/zsh
TERM=xterm-256color
OPENADDRESSES_PARALLELISM=4
TMPDIR=/var/folders/4z/7bs6740925n6pc71ggp8f8yc0000gn/T/
TERM_PROGRAM_VERSION=443
DATA_DIR=/Applications/deliverone/vetday
TERM_SESSION_ID=5DE43C35-001E-49C6-B58E-8FC5671356FC
USER=myronschram
SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.246Bevo9Ms/Listeners
PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
LaunchInstanceID=C069BEB6-A900-4692-B6AB-BC684286AC0C
__CFBundleIdentifier=com.apple.Terminal
PWD=/Applications/deliverone/vetday/docker/projects/texas
LANG=en_US.UTF-8
XPC_FLAGS=0x0
DOCKER_USER=501:20
XPC_SERVICE_NAME=0
COMPOSE_PROJECT_NAME=pelias
SHLVL=2
HOME=/Users/myronschram
LOGNAME=myronschram
SECURITYSESSIONID=186a5
_=/usr/bin/env

@map texas % pelias prepare interpolation
DATA_DIR is already set to '/Applications/deliverone/vetday' - this may cause the DATA_DIR specified in the .env to be ignored
Creating pelias_interpolation_run ... done
- importing polylines
ERROR: 1

DATA_DIR is already set to '/Applications/deliverone/vetday' - this may cause the DATA_DIR specified in the .env to be ignored
Attaching to pelias_interpolation
interpolation_1  | 
interpolation_1  | /code/pelias/interpolation/node_modules/better-sqlite3/lib/database.js:51
interpolation_1  |      [util.cppdb]: { value: new CPPDatabase(filename, filenameGiven, anonymous, readonly, fileMustExist, timeout, verbose || null, buffer || null) },
interpolation_1  |                             ^
interpolation_1  | SqliteError: unable to open database file
interpolation_1  |     at new Database (/code/pelias/interpolation/node_modules/better-sqlite3/lib/database.js:51:26)
interpolation_1  |     at setup (/code/pelias/interpolation/api/search.js:12:14)
interpolation_1  |     at Object.<anonymous> (/code/pelias/interpolation/cmd/server.js:32:11)
interpolation_1  |     at Module._compile (internal/modules/cjs/loader.js:999:30)
interpolation_1  |     at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
interpolation_1  |     at Module.load (internal/modules/cjs/loader.js:863:32)
interpolation_1  |     at Function.Module._load (internal/modules/cjs/loader.js:708:14)
interpolation_1  |     at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:60:12)
interpolation_1  |     at internal/main/run_main_module.js:17:47
interpolation_1  | 
interpolation_1  | /code/pelias/interpolation/node_modules/better-sqlite3/lib/database.js:51
interpolation_1  |      [util.cppdb]: { value: new CPPDatabase(filename, filenameGiven, anonymous, readonly, fileMustExist, timeout, verbose || null, buffer || null) },
interpolation_1  |                             ^
interpolation_1  | SqliteError: unable to open database file
interpolation_1  |     at new Database (/code/pelias/interpolation/node_modules/better-sqlite3/lib/database.js:51:26)
interpolation_1  |     at setup (/code/pelias/interpolation/api/search.js:12:14)
interpolation_1  |     at Object.<anonymous> (/code/pelias/interpolation/cmd/server.js:32:11)
interpolation_1  |     at Module._compile (internal/modules/cjs/loader.js:999:30)
interpolation_1  |     at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
interpolation_1  |     at Module.load (internal/modules/cjs/loader.js:863:32)
interpolation_1  |     at Function.Module._load (internal/modules/cjs/loader.js:708:14)
interpolation_1  |     at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:60:12)
interpolation_1  |     at internal/main/run_main_module.js:17:47
interpolation_1  | 
interpolation_1  | /code/pelias/interpolation/node_modules/better-sqlite3/lib/database.js:51
interpolation_1  |      [util.cppdb]: { value: new CPPDatabase(filename, filenameGiven, anonymous, readonly, fileMustExist, timeout, verbose || null, buffer || null) },
interpolation_1  |                             ^
interpolation_1  | SqliteError: unable to open database file
interpolation_1  |     at new Database (/code/pelias/interpolation/node_modules/better-sqlite3/lib/database.js:51:26)
interpolation_1  |     at setup (/code/pelias/interpolation/api/search.js:12:14)
interpolation_1  |     at Object.<anonymous> (/code/pelias/interpolation/cmd/server.js:32:11)
interpolation_1  |     at Module._compile (internal/modules/cjs/loader.js:999:30)
interpolation_1  |     at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
interpolation_1  |     at Module.load (internal/modules/cjs/loader.js:863:32)
interpolation_1  |     at Function.Module._load (internal/modules/cjs/loader.js:708:14)
interpolation_1  |     at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:60:12)
interpolation_1  |     at internal/main/run_main_module.js:17:47
orangejulius commented 2 years ago

Hi @myrondeliverone, Thanks for the report.

It looks like this was actually an issue with our Docker image for the interpolation service. We recently did some reshuffling of our build toolchain and some dependencies weren't being built correctly.

I was able to reproduce this and opened https://github.com/pelias/interpolation/pull/282 which really should fix everything, so I'm going to go ahead and let that PR close this issue. Don't hesitate to comment here if a pelias compose pull doesn't fix the issue for you.

For the record, the problem can be diagnosed fairly quickly but only if you know the somewhat secret place to look: /$DATA_DIR/interpolation/polylines.err. It took me a while to know to look here but once I did, the problem was fairly evident:

$ cat polyline.err
invalid polyline row 1

events.js:291
      throw er; // Unhandled 'error' event
      ^

Error: Could not locate the bindings file. Tried:
 → /code/pelias/interpolation/node_modules/node-postal/build/expand.node
 → /code/pelias/interpolation/node_modules/node-postal/build/Debug/expand.node
 → /code/pelias/interpolation/node_modules/node-postal/build/Release/expand.node
 → /code/pelias/interpolation/node_modules/node-postal/out/Debug/expand.node
 → /code/pelias/interpolation/node_modules/node-postal/Debug/expand.node
 → /code/pelias/interpolation/node_modules/node-postal/out/Release/expand.node
 → /code/pelias/interpolation/node_modules/node-postal/Release/expand.node
 → /code/pelias/interpolation/node_modules/node-postal/build/default/expand.node
 → /code/pelias/interpolation/node_modules/node-postal/compiled/12.22.2/linux/x64/expand.node
 → /code/pelias/interpolation/node_modules/node-postal/addon-build/release/install-root/expand.node
 → /code/pelias/interpolation/node_modules/node-postal/addon-build/debug/install-root/expand.node
 → /code/pelias/interpolation/node_modules/node-postal/addon-build/default/install-root/expand.node
 → /code/pelias/interpolation/node_modules/node-postal/lib/binding/node-v72-linux-x64/expand.node
    at bindings (/code/pelias/interpolation/node_modules/bindings/bindings.js:126:9)
    at Object.<anonymous> (/code/pelias/interpolation/node_modules/node-postal/index.js:3:32)
    at Module._compile (internal/modules/cjs/loader.js:999:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
    at Module.load (internal/modules/cjs/loader.js:863:32)
    at Function.Module._load (internal/modules/cjs/loader.js:708:14)
    at Module.require (internal/modules/cjs/loader.js:887:19)
    at require (internal/modules/cjs/helpers.js:74:18)
    at get_libpostal (/code/pelias/interpolation/lib/analyze.js:24:28)
    at Object.street (/code/pelias/interpolation/lib/analyze.js:35:18)
Emitted 'error' event on Transform instance at:
    at errorOrDestroy (internal/streams/destroy.js:108:12)
    at Transform.onerror (_stream_readable.js:754:7)
    at Transform.emit (events.js:314:20)
    at errorOrDestroy (/code/pelias/interpolation/node_modules/split2/node_modules/readable-stream/lib/internal/streams/destroy.js:98:101)
    at onwriteError (/code/pelias/interpolation/node_modules/split2/node_modules/readable-stream/lib/_stream_writable.js:424:5)
    at onwrite (/code/pelias/interpolation/node_modules/split2/node_modules/readable-stream/lib/_stream_writable.js:450:11)
    at WritableState.onwrite (/code/pelias/interpolation/node_modules/split2/node_modules/readable-stream/lib/_stream_writable.js:160:5)
    at Transform.afterTransform (/code/pelias/interpolation/node_modules/split2/node_modules/readable-stream/lib/_stream_transform.js:89:3)
    at Transform.transform [as _transform] (/code/pelias/interpolation/node_modules/split2/index.js:46:14)
    at Transform._read (/code/pelias/interpolation/node_modules/split2/node_modules/readable-stream/lib/_stream_transform.js:177:10) {
  tries: [
    '/code/pelias/interpolation/node_modules/node-postal/build/expand.node',
    '/code/pelias/interpolation/node_modules/node-postal/build/Debug/expand.node',
    '/code/pelias/interpolation/node_modules/node-postal/build/Release/expand.node',
    '/code/pelias/interpolation/node_modules/node-postal/out/Debug/expand.node',
    '/code/pelias/interpolation/node_modules/node-postal/Debug/expand.node',
    '/code/pelias/interpolation/node_modules/node-postal/out/Release/expand.node',
    '/code/pelias/interpolation/node_modules/node-postal/Release/expand.node',
    '/code/pelias/interpolation/node_modules/node-postal/build/default/expand.node',
    '/code/pelias/interpolation/node_modules/node-postal/compiled/12.22.2/linux/x64/expand.node',
    '/code/pelias/interpolation/node_modules/node-postal/addon-build/release/install-root/expand.node',
    '/code/pelias/interpolation/node_modules/node-postal/addon-build/debug/install-root/expand.node',
    '/code/pelias/interpolation/node_modules/node-postal/addon-build/default/install-root/expand.node',
    '/code/pelias/interpolation/node_modules/node-postal/lib/binding/node-v72-linux-x64/expand.node'
  ]
}

@missinglink we might want to consider documenting that file better or having the output there also sent to the Docker logs.