search

cfpb / grasshopper

CFPB's streaming batch geocoder
Creative Commons Zero v1.0 Universal
37 stars 13 forks source link

Adds test-harness Compose setup #205

Closed hkeeler closed 8 years ago

hkeeler commented 8 years ago

Here's a Docker Compose setup for building and running test-harness and all its dependencies. This setup works...mostly. However, there have been several inconsistent errors while loading data, primarily related to network and file access. I've done a full rebuild and re-load 3x, and I've encountered similar issues, but not always during the same point in the loading process.

It is unclear what's causing this. I'm putting this PR up now so others can try it on their machines and see if they encounter the same issues.

In order to run this, you will need more storage than the default Docker Machine Virtualbox image comes with. At the bare minimum you need to increase the disk space, but I've also bumped up RAM and CPUs. It looks like this:

docker-machine create --driver virtualbox \
--virtualbox-memory 4096 \
--virtualbox-cpu-count 4 \
--virtualbox-disk-size 102400 \
docker-dev

Once you've created the new image, you're ready to launch. However, before you do, make sure you have the latest version of all dependent projects, and that they're checked out into the same directory as your grasshopper project.

To launch the stack. This will take a while as it has to build the hmda-geo Docker images, which include a pretty big data load.

docker-compose -f docker-compose-test-harness.yaml up

Load the data:

# Address points
docker-compose -f docker-compose-test-harness.yaml run loader ./index.js -f data.json

# TIGER lines
docker-compose -f docker-compose-test-harness.yaml run loader ./tiger.js

Give it a whirl, and let me know how it goes.

wpears commented 8 years ago

Waiting on https://github.com/cfpb/grasshopper-loader/pull/164 to clear up some of the intermittent errors before being able to evaluate this properly

awolfe76 commented 8 years ago

https://github.com/cfpb/grasshopper-loader/pull/164 is merged!

wpears commented 8 years ago

I'd say this is merge-able. Any remaining loading troubles fall outside the purview of this PR (and the previous loader fixes have ameliorated most of the issues), so :+1: