Data Load / WOF issues

[fpurcell]

I'm having problems running build.sh with recent (August 2017) commits to the dockerfiles project. The problems seem to stem from WOF data not being available for OA and OSM loaders.

A bit of history: back on August 1st, I was able to successfully run build.sh, and see a fully loaded Pelias system running (via the example pelias.json, thus data for just the Portland area). I was using dockerfiles commit 1e6857403fe6c565aeddc456296c7d5667487016 from master.

On August 17th, after doing a git update (now on commit 77ba1beb2e4d26e6a6b9774a11fa05259bb27db1), I tried to re-run build.sh again, which failed. On this same server, I nuked all of the docker images and volumes, and tried re-running build.sh. Pretty much the same failures, with the OA and OSM loaders hung because (seemingly) there was no WOF data available in $DATA_DIR/whosonfirst (meta directory and meta/whosonfirst_bundle_index.txt is there, but nothing else).

Yesterday I provisioned a fresh server, and ran the docker files again from master (commit 77ba1beb2e4d26e6a6b9774a11fa05259bb27db1). I again am seeing similar issues, and WOF data missing.

I'm attaching two console logs. First fail-1.txt is the initial run of build.sh on the fresh server. The second file fail-2.txt is after nuking the Pelias docker stuff from step one, and trying to run build.sh a second time.

BTW, here's what my $DATA_DIR looks like:

var/data/elasticsearch/: elasticsearch

var/data/interpolation/:

var/data/openaddresses/: README.txt us

var/data/openstreetmap/: portland_oregon.osm.pbf

var/data/placeholder/: graph.json store.sqlite3 wof.extract

var/data/tiger/: downloads shapefiles

var/data/whosonfirst/: meta

AND docker ps shows the following running after nuking and re-running build.sh:

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 71c969760a99 pelias/openstreetmap "npm start" 19 hours ago Up 19 hours dockerfiles_openstreetmap_run_1 de4c0170d5bd pelias/openaddresses "npm start" 19 hours ago Up 19 hours dockerfiles_openaddresses_run_1 68ddcf0aafd7 pelias/elasticsearch "/docker-entrypoint.s" 19 hours ago Up 19 hours 0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp pelias_elasticsearch

[dianashk]

I think at least some of your errors are a result of too little memory allocated to Docker on the host machine. It's odd that the whosonfirst data download is trying to grab bundles, since the pelias.json file should be limiting that to only the Portland area.

Could you share your pelias.json file here please so I can see what's going on. If you've added your API key please remove it before sharing. :)

[fpurcell]

Hey Diana,

AFAIK, the pelias.json in use is the same from https://github.com/pelias/dockerfiles/blob/master/pelias.json. I'm running ./build.sh directly out of the freshly cloned dockerfiles directory. I'm not aware of any other pelias.json file in use on the system (although, I could be wrong about that).

I'm going to search (and remove) any other pelias.json files, and also look to allocate more memory to docker. I'll post an update later today.

Thanks, Frank

[fpurcell]

Hmmm. Could the problem be a lack of an API key for WOF? I'm re-running build.sh now, so will find out. Btw, the only pelias.json file(s) are in the dockerfiles project...including the one I've just edited with a valid WOF key. Also, John Z. believes that Docker has access to as much of the ~14G of RAM allocated to the virtual server, so we're monitoring whether that's an issue. ./build.sh still chugging along ... just getting thru the ubuntu install junk...update as soon as the build finishes.

[fpurcell]

With a valid API key in the WOF config, a re-run of ./build.sh worked (partial success, as polylines failed ... thinking it might have been some artifacts of previous runs, so re-running).

BTW, I'd like to request a documentation change (if only to trigger my feeble brain in the future), with the addition of 'get your free MapZen key, and add it to pelias.json where ever it says "your-api-key"' to the https://github.com/pelias/dockerfiles#prerequisites as a prerequisite.

[missinglink]

Possibly related to https://github.com/pelias/whosonfirst/pull/264, I think I introduced a bug in a recent PR to that repo which has broken the whosonfirst imports in to elasticsearch.

I'll bring up a team discussion about it today and see if when can get this fixed ASAP

[dianashk]

I've merged the fix for the WOF import to master so when you get a chance please try to build the images again and rerun the setup from scratch. I'll try to publish the latest images to dockerhub soon.

[fpurcell]

Thanks Peter & Diana. Things are working fairly well on this end now, so I will close this issue.

(That said, the generation of polylines was problematic in my last attempt, and thus interpolation isn't building -- I'm re-running the build one more time to make sure, and will research a bit more before filing a new issue).

[dianashk]

We just fixed a bug that was introduced while we were away at foss4g last week. Please pull down latest images from dockerhub and rerun just the polylines and interpolation steps. On Fri, Aug 25, 2017 at 3:17 PM Frank Purcell notifications@github.com wrote:

Thanks Peter & Diana. Things are working fairly well on this end now, so I will close this issue.

(That said, the generation of polylines was problematic in my last attempt, and thus interpolation isn't building -- I'm re-running the build one more time to make sure, and will research a bit more before filing a new issue).

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/pelias/dockerfiles/issues/20#issuecomment-325012961, or mute the thread https://github.com/notifications/unsubscribe-auth/AFMbb_WQOgikG2aOodUEcXFJOxNMw_6Vks5sbx3ggaJpZM4PAe35 .