pelias / whosonfirst

Importer for Who's on First gazetteer
MIT License
28 stars 42 forks source link

I cannot use importPlace option, schema.js outdated in production branch #276

Closed darkyelox closed 7 years ago

darkyelox commented 7 years ago

I have a problem trying to download only my country data because when i install the production version using the script:

for repository in schema api whosonfirst geonames openaddresses openstreetmap polylines; do
    git clone git@github.com:pelias/${repository}.git    # clone from Github
    pushd $repository > /dev/null                        # switch into importer directory
    git checkout production                              # or remove this line to stay with master
    npm install                                          # install npm dependencies
    popd > /dev/null                                     # return to code directory
done

With the latest productoin of whosonfirst the schema.js is:

module.exports = Joi.object().keys({
  imports: Joi.object().keys({
    whosonfirst: Joi.object().keys({
      datapath: Joi.string(),
      importVenues: Joi.boolean().default(false).truthy('yes').falsy('no').insensitive(true),
      importPostalcodes: Joi.boolean().default(false).truthy('yes').falsy('no').insensitive(true),
      missingFilesAreFatal: Joi.boolean().default(false).truthy('yes').falsy('no').insensitive(true)
    }).requiredKeys('datapath').unknown(false)
  }).requiredKeys('whosonfirst').unknown(true)
}).requiredKeys('imports').unknown(true);

So Joi throws an error when checking the data, then i download the latest master of whosonfirst and the config can be parsed but in the process i get this error:

2017-09-01T15:55:34.415Z - info: [download_data_filtered] Getting parent ids
2017-09-01T15:55:34.961Z - debug: [download_data_filtered] parent ids: 0=102191577, 1=85632519
2017-09-01T15:55:35.427Z - info: [download_data_filtered] Getting descendants
2017-09-01T15:55:35.854Z - debug: [download_data_filtered] appending 0 descendants to continent
2017-09-01T15:55:36.292Z - debug: [download_data_filtered] appending 0 descendants to country
2017-09-01T15:55:36.669Z - debug: [download_data_filtered] appending 0 descendants to dependency
2017-09-01T15:55:37.091Z - debug: [download_data_filtered] appending 2 descendants to disputed
2017-09-01T15:55:37.451Z - debug: [download_data_filtered] appending 0 descendants to macroregion
2017-09-01T15:55:37.937Z - debug: [download_data_filtered] appending 34 descendants to region
2017-09-01T15:55:38.320Z - debug: [download_data_filtered] appending 0 descendants to macrocounty
2017-09-01T15:55:39.356Z - debug: [download_data_filtered] appending 500 descendants to county
2017-09-01T15:55:40.339Z - debug: [download_data_filtered] appending 500 descendants to county
2017-09-01T15:55:40.982Z - debug: [download_data_filtered] appending 122 descendants to county
2017-09-01T15:55:41.438Z - debug: [download_data_filtered] appending 0 descendants to county
2017-09-01T15:55:41.871Z - debug: [download_data_filtered] appending 2 descendants to localadmin
2017-09-01T15:55:42.852Z - debug: [download_data_filtered] appending 500 descendants to locality
2017-09-01T15:55:43.733Z - debug: [download_data_filtered] appending 500 descendants to locality
2017-09-01T15:55:44.359Z - debug: [download_data_filtered] appending 93 descendants to locality
2017-09-01T15:55:44.807Z - debug: [download_data_filtered] appending 0 descendants to locality
2017-09-01T15:55:45.225Z - debug: [download_data_filtered] appending 0 descendants to borough
2017-09-01T15:55:45.650Z - debug: [download_data_filtered] appending 0 descendants to macrohood
2017-09-01T15:55:46.301Z - debug: [download_data_filtered] appending 111 descendants to neighbourhood
2017-09-01T15:55:46.403Z - info: [download_data_filtered] Generating metafiles
2017-09-01T15:55:46.407Z - debug: [download_data_filtered] previous file not found /home/diego/pelias/data/whosonfirst/meta/wof-continent-latest.csv null false
2017-09-01T15:55:46.893Z - info: [download_data_filtered] Finished generating metafile for continent Error: EACCES: permission denied, mkdir '/home/diego'
    at Error (native)
events.js:160
      throw er; // Unhandled 'error' event
      ^

Error: EACCES: permission denied, mkdir '/home/diego'
    at Error (native)

npm ERR! Linux 4.4.0-93-generic
npm ERR! argv "/home/ubuntu/.nvm/versions/node/v6.11.2/bin/node" "/home/ubuntu/.nvm/versions/node/v6.11.2/bin/npm" "run" "download"
npm ERR! node v6.11.2
npm ERR! npm  v3.10.10
npm ERR! code ELIFECYCLE
npm ERR! pelias-whosonfirst@0.0.0-semantic-release download: `node ./utils/download_data.js`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the pelias-whosonfirst@0.0.0-semantic-release download script 'node ./utils/download_data.js'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the pelias-whosonfirst package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node ./utils/download_data.js
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs pelias-whosonfirst
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls pelias-whosonfirst
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     /home/ubuntu/pelias/whosonfirst/npm-debug.log
darkyelox commented 7 years ago

The last part about the mkdir was my bad, the path for the download data for whosonfirst was pointing to a inexistent directory, i checkout the master branch and seems to do the download fine.

orangejulius commented 7 years ago

Hey @darkyelox, We did have some issues with the Who's on First data download, and the fixes are in the master branch, and will be in the production branches soon. It looks like you've figured this out so I'm closing this issue, but if there are any other problems or you have other questions, please let us know.