No permission to create some data/* directories when using example build script

mattwelke commented 5 years ago

I tried using the example build script in the Git repo readme:

#!/bin/bash
set -x

# create directories
mkdir /code /data

# clone repo
cd /code
git clone https://github.com/pelias/docker.git
cd docker

# install pelias script
ln -s "$(pwd)/pelias" /usr/local/bin/pelias

# cwd
cd projects/portland-metro

# configure environment
sed -i '/DATA_DIR/d' .env
echo 'DATA_DIR=/data' >> .env

# run build
pelias compose pull
pelias elastic start
pelias elastic wait
pelias elastic create
pelias download all
pelias prepare all
pelias import all
pelias compose up

# optionally run tests
pelias test run

I put this into a script called build.sh in the root home directory and used chmod +x build.sh to make it executable.

But even though I was logged into root and created the directory /data, it was only able to create a few of the data/* directories (one of them was elasticsearch) before the script ran into errors indiciating no permission to create the others. This caused the script to fail right before it started downloading the address data:

root@docker-s-2vcpu-4gb-nyc1-01:~# ./build.sh 
+ mkdir /code /data
mkdir: cannot create directory ‘/data’: File exists
+ cd /code
+ git clone https://github.com/pelias/docker.git
Cloning into 'docker'...
remote: Enumerating objects: 24, done.
remote: Counting objects: 100% (24/24), done.
remote: Compressing objects: 100% (20/20), done.
remote: Total 595 (delta 11), reused 11 (delta 4), pack-reused 571
Receiving objects: 100% (595/595), 213.91 KiB | 10.70 MiB/s, done.
Resolving deltas: 100% (351/351), done.
+ cd docker
++ pwd
+ ln -s /code/docker/pelias /usr/local/bin/pelias
+ cd projects/portland-metro
+ sed -i /DATA_DIR/d .env
+ echo DATA_DIR=/data
+ pelias compose pull
Pulling libpostal     ... done
Pulling schema        ... done
Pulling api           ... done
Pulling placeholder   ... done
Pulling whosonfirst   ... done
Pulling openstreetmap ... done
Pulling openaddresses ... done
Pulling transit       ... done
Pulling polylines     ... done
Pulling interpolation ... done
Pulling pip           ... done
Pulling elasticsearch ... done
Pulling fuzzy-tester  ... done
+ pelias elastic start
Creating network "pelias_default" with driver "bridge"
Creating pelias_elasticsearch ... done
+ pelias elastic wait
waiting for elasticsearch service to come up
....
+ pelias elastic create

--------------
 create index 
--------------

[put mapping]    pelias { acknowledged: true } 

+ pelias download all
wget http://biketownpdx.socialbicycles.com/opendata/station_information.json to /data/transit/BIKETOWN-hubs.json
Could not use "nc", falling back to slower node.js method for sync requests.
{ Error: ENOENT: no such file or directory, open '/data/transit/BIKETOWN-hubs.json'
    at Object.fs.openSync (fs.js:646:18)
    at Object.fs.writeFileSync (fs.js:1299:33)
    at downloadFile (/code/pelias/transit/lib/prep_data.js:83:10)
    at /code/pelias/transit/lib/prep_data.js:35:17
    at Array.forEach (<anonymous>)
    at Object.<anonymous> (/code/pelias/transit/lib/prep_data.js:25:21)
    at Module._compile (module.js:652:30)
    at Object.Module._extensions..js (module.js:663:10)
    at Module.load (module.js:565:32)
    at tryModuleLoad (module.js:505:12)
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: '/data/transit/BIKETOWN-hubs.json' }
Had an issue obtaining new file /data/transit/BIKETOWN-hubs.json
wget http://developer.trimet.org/schedule/gtfs.zip to /data/transit/TRIMET-stops.txt
info: [openaddresses-download] Attempting to download selected data files: us/or/portland_metro.csv,us/or/city_of_salem.csv,us/or/marion_and_polk.csv,us/or/marion.csv,us/or/hood_river.csv,us/wa/city_of_richland.csv,us/wa/clark.csv
error: [openaddresses-download] error making directory /data/openaddresses message=EACCES: permission denied, mkdir '/data/openaddresses', stack=Error: EACCES: permission denied, mkdir '/data/openaddresses', errno=-13, code=EACCES, syscall=mkdir, path=/data/openaddresses
error: [download] Failed to download data message=EACCES: permission denied, mkdir '/data/openaddresses', stack=Error: EACCES: permission denied, mkdir '/data/openaddresses', errno=-13, code=EACCES, syscall=mkdir, path=/data/openaddresses
/code/pelias/interpolation/node_modules/fs-extra/lib/mkdirs/mkdirs-sync.js:45
        throw err0
        ^

Error: EACCES: permission denied, mkdir '/data/tiger'
    at Object.fs.mkdirSync (fs.js:885:18)
    at mkdirsSync (/code/pelias/interpolation/node_modules/fs-extra/lib/mkdirs/mkdirs-sync.js:31:9)
    at Object.mkdirsSync (/code/pelias/interpolation/node_modules/fs-extra/lib/mkdirs/mkdirs-sync.js:36:14)
    at downloadFilteredFiles (/code/pelias/interpolation/script/js/update_tiger.js:99:6)
    at /code/pelias/interpolation/node_modules/async/dist/async.js:3880:24
    at replenish (/code/pelias/interpolation/node_modules/async/dist/async.js:1011:17)
    at iterateeCallback (/code/pelias/interpolation/node_modules/async/dist/async.js:995:17)
    at /code/pelias/interpolation/node_modules/async/dist/async.js:969:16
    at /code/pelias/interpolation/node_modules/async/dist/async.js:3885:13
    at context.ftp.list (/code/pelias/interpolation/script/js/update_tiger.js:91:5)
/code/pelias/whosonfirst/node_modules/fs-extra/lib/mkdirs/mkdirs-sync.js:45
        throw err0
        ^

Error: EACCES: permission denied, mkdir '/data/whosonfirst'
    at Object.fs.mkdirSync (fs.js:885:18)
    at mkdirsSync (/code/pelias/whosonfirst/node_modules/fs-extra/lib/mkdirs/mkdirs-sync.js:31:9)
    at Object.mkdirsSync (/code/pelias/whosonfirst/node_modules/fs-extra/lib/mkdirs/mkdirs-sync.js:36:14)
    at download (/code/pelias/whosonfirst/utils/sqlite_download.js:15:6)
    at Object.<anonymous> (/code/pelias/whosonfirst/utils/download_data.js:15:3)
    at Module._compile (module.js:652:30)
    at Object.Module._extensions..js (module.js:663:10)
    at Module.load (module.js:565:32)
    at tryModuleLoad (module.js:505:12)
    at Function.Module._load (module.js:497:3)
info: [openstreetmap-download] Downloading sources: https://s3.amazonaws.com/metro-extracts.nextzen.org/portland_oregon.osm.pbf
error: [openstreetmap-download] error making directory /data/openstreetmap message=EACCES: permission denied, mkdir '/data/openstreetmap', stack=Error: EACCES: permission denied, mkdir '/data/openstreetmap', errno=-13, code=EACCES, syscall=mkdir, path=/data/openstreetmap
error: [openstreetmap-download] Failed to download data message=EACCES: permission denied, mkdir '/data/openstreetmap', stack=Error: EACCES: permission denied, mkdir '/data/openstreetmap', errno=-13, code=EACCES, syscall=mkdir, path=/data/openstreetmap

At this point, it's running, but appears to error out when a request comes in:

~ > curl '159.89.127.11:4000/v1/search?text=65%20Front%20St.%20W.%20Toronto%20Ontario%20Canada' | jq '.'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   486  100   486    0     0   4909      0 --:--:-- --:--:-- --:--:--  4909
{
  "geocoding": {
    "version": "0.2",
    "attribution": "http://159.89.127.11:4000/attribution",
    "query": {
      "text": "65 Front St. W. Toronto Ontario Canada",
      "size": 10,
      "private": false,
      "focus.point.lat": 45.52,
      "focus.point.lon": -122.67,
      "lang": {
        "name": "English",
        "iso6391": "en",
        "iso6393": "eng",
        "defaulted": true
      },
      "querySize": 20
    },
    "errors": [
      "connect ECONNREFUSED 172.18.0.4:4400"
    ],
    "engine": {
      "name": "Pelias",
      "author": "Mapzen",
      "version": "1.0"
    },
    "timestamp": 1540156052358
  },
  "type": "FeatureCollection",
  "features": []
}

orangejulius commented 5 years ago

Hey @welkie, Thanks for the report. I believe this issue is caused by running the above commands (and most importantly, making the /code and /data directories) as root.

All Pelias containers run as a non-root user. What happens if you re-run the script as a non-root user? You might have to pick different directories than /code and /data.

mattwelke commented 5 years ago

@orangejulius Thanks for the tip. Will try as non root user and report back.

mattwelke commented 5 years ago

I tried it as a sudo non-root user but I had the same problem. I'm going to spend some more time reading the documentation before raising further issues though because I have to admit I just grabbed that script and ran it, not understanding what the Pelias setup does in detail yet.

sophia-xll commented 5 years ago

@welkie Hello welkie. How did you solve this problem? I have the same questions too.

mattwelke commented 5 years ago

@sophia-xll Ended up feeling overwhelmed trying to set this up and gave up for now unfortunately. Sorry. :(

imran-5 commented 4 years ago

solved this error, set the permission of dir manually as docker user.

sudo chown -R pelias:pelias /data/openaddresses

orangejulius commented 3 years ago

Hi folks, Just doing some issue pruning, I think we've taken care of this one.

If anyone else winds up here with permission issues here are some general hints:

don't run any pelias ... commands as root or using sudo
Make sure whatever directory you've set to $DATA_DIR is owned by the same user you are running the pelias ... commands as (and also not owned by root)
If all else fails use rm -rf to (carefully) remove the $DATA_DIR and re-create it as the same (non-root) user you are running the pelias ... commands as

pelias / docker

No permission to create some data/* directories when using example build script #31