refactor shell scripts, expose bash API and cli tool

[missinglink]

this PR is a big refactor of the bash code in this repo, it grew over time and became a little unsightly.

there are also some files which can be removed in this repo, such as example.sh but I won't conflate this PR with that work.

for now the cli is named pelias.sh, although I'd be happy to rename it, we could even call it pelias (without the extension) if we like:

$ ./pelias.sh 

Usage: pelias.sh [command] [action] [options]

  compose        exec                     run an arbitrary docker-compose command
  docker         pull                     update all docker images
  download       wof                      (re)download whosonfirst data
  download       oa                       (re)download openaddresses data
  download       osm                      (re)download openstreetmap data
  download       tiger                    (re)download TIGER data
  download       transit                  (re)download transit data
  download       all                      (re)download all data
  elasticsearch  drop                     delete elasticsearch index & all data
  elasticsearch  create                   create elasticsearch index with pelias mapping
  elasticsearch  start                    start elasticsearch server
  elasticsearch  stop                     stop elasticsearch server
  elasticsearch  status                   HTTP status code of the elasticsearch service
  elasticsearch  wait                     wait for elasticsearch to start up
  import         wof                      (re)import whosonfirst data
  import         oa                       (re)import openaddresses data
  import         osm                      (re)import openstreetmap data
  import         polylines                (re)import polylines data
  import         transit                  (re)import transit data
  import         all                      (re)import all data
  prepare        polylines                export road network from openstreetmap into polylines format
  prepare        interpolation            build interpolation sqlite databases
  prepare        placeholder              build placeholder sqlite databases
  prepare        all                      build all services which have a prepare step

I considered adding up down and logs commands but I didn't want to recreate docker-compose, which offers that functionality.

[orangejulius]

Hmm. I am getting errors when running ./prep_data.sh now. Any ideas? (output below is with set -x)

Status: Image is up to date for pelias/pip-service:latest                                                
++ elastic_status                                                                            
++ curl --output /dev/null --silent --write-out '%{http_code}' http://localhost:9200
+ test 200 -ne 200                                                                                   
+ elastic_schema_drop -f                           
+ compose_run schema node scripts/drop_index -f
+ docker-compose run -T --rm schema node scripts/drop_index -f
Elasticsearch ERROR: 2018-04-23T22:54:46Z
  Error: Request error, retrying                                             
  DELETE http://elasticsearch:9200/pelias => socket hang up
      at Log.error (/code/pelias/schema/node_modules/elasticsearch/src/lib/log.js:225:56)
      at checkRespForFailure (/code/pelias/schema/node_modules/elasticsearch/src/lib/transport.js:258:18)
      at HttpConnector.<anonymous> (/code/pelias/schema/node_modules/elasticsearch/src/lib/connectors/http.js:157:7)
      at ClientRequest.bound (/code/pelias/schema/node_modules/lodash/dist/lodash.js:729:21)
      at emitOne (events.js:96:13)
      at ClientRequest.emit (events.js:188:7)                                                               
      at Socket.socketCloseListener (_http_client.js:286:9)                 
      at emitOne (events.js:101:20)                                              
      at Socket.emit (events.js:188:7)
      at TCP._handle.close [as _onclose] (net.js:509:12)

Elasticsearch WARNING: 2018-04-23T22:54:49Z                                    
  Unable to revive connection: http://elasticsearch:9200/

Elasticsearch WARNING: 2018-04-23T22:54:49Z
  No living connections                                  

[delete mapping]         pelias { Error: No Living connections
    at sendReqWithConnection (/code/pelias/schema/node_modules/elasticsearch/src/lib/transport.js:225:15)
    at next (/code/pelias/schema/node_modules/elasticsearch/src/lib/connection_pool.js:213:7)
...

[missinglink]

@orangejulius weirdly it works for me, I noticed at the top you are targeting http://localhost:9200 and down below http://elasticsearch:9200, did you change anything from how I had it set in the refactor branch?

FYI you can change the elastic host by setting ELASTIC_HOST to something other than localhost:9200 in either your env or in .env

[orangejulius]

@missinglink did you test it with ELASTIC_HOST _un_set? The variable default is different in two places: localhost in elastic.sh, and elasticsearch in pelias.json. However, the importers also seem to have issues with elasticearch (same error pasted above), so there's more going on.

[missinglink]

That's odd, I didn't explicitly set ELASTIC_HOST, I liked the idea of having the elasticsearch part separated from the docker-compose setup in the event that someone wanted to run the ES portion elsewhere.

I can probably move it to run the curl command inside a container if that helps?

It works for me on both my desktop and laptop (running different OS and everything):

$ ./pelias.sh elastic status
200

So far as I remember, the code is the same as it was before the refactor, do you get the same issue on the master branch?

[orangejulius]

This has been replaced with https://github.com/pelias/dockerfiles/