Potential failure downloading metadata

orangejulius commented 4 years ago

Last week (while trying to merge the fix in #386), Geonames CI builds were failing with the following error:

> pelias-geonames@0.0.0-development download_metadata /code/pelias/geonames
> mkdirp metadata && node bin/updateMetadata.js

internal/streams/legacy.js:59
      throw er; // Unhandled stream error in pipe.
      ^

CsvError: Invalid Record Length: columns length is 19, got 21 on line 1
    at Parser.__onRow (/code/pelias/geonames/node_modules/csv-parse/lib/index.js:701:13)
    at Parser.__parse (/code/pelias/geonames/node_modules/csv-parse/lib/index.js:568:40)
    at Parser._transform (/code/pelias/geonames/node_modules/csv-parse/lib/index.js:382:22)
    at Parser.Transform._read (_stream_transform.js:189:10)
    at Parser.Transform._write (_stream_transform.js:177:12)
    at doWrite (_stream_writable.js:435:12)
    at writeOrBuffer (_stream_writable.js:419:5)
    at Parser.Writable.write (_stream_writable.js:309:11)
    at Request.ondata (internal/streams/legacy.js:17:31)
    at Request.emit (events.js:223:5) {
  code: 'CSV_RECORD_DONT_MATCH_COLUMNS_LENGTH',
  column: null,
  empty_lines: 0,
  header: false,
  index: 21,
  invalid_field_length: 0,
  quoting: false,
  lines: 1,
  records: 0,
  record: [
    '# GeoNames.org Country Information',
    '',
    '',
    '',
    '',
    '',
    '',
    '',
    '',
    '',
    '',
    '',
    '',
    '',
    '',
    '',
    '',
    '',
    '',
    '',
    ''
  ]
}

It looks like something changed in the http://geonames.org/export/dump/countryInfo.txt file that we weren't able to parse. It's back to normal this week, but perhaps at some point we should take a look at hardening the parsing process against unexpected input.

toton6868 commented 4 years ago

Is it related to similar bug???

Error: Cannot find module '../metadata/countryInfo'
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:636:15)
    at Function.Module._load (internal/modules/cjs/loader.js:562:25)
    at Module.require (internal/modules/cjs/loader.js:692:17)
    at require (internal/modules/cjs/helpers.js:25:18)
    at Object.<anonymous> (/opt/osgs/geocoding/geonames/lib/validateISOCode.js:3:21)
    at Module._compile (internal/modules/cjs/loader.js:778:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
    at Module.load (internal/modules/cjs/loader.js:653:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
    at Function.Module._load (internal/modules/cjs/loader.js:585:3)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! pelias-geonames@0.0.0-development download: `./bin/download`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the pelias-geonames@0.0.0-development download script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2020-05-26T01_09_06_171Z-debug.log

missinglink commented 4 years ago

Is this possibly fixed now "by magic"?

node bin/updateMetadata.js
➜  geonames git:(master) ✗ echo $?
0

git log -1
commit 7e49a2329cb82ddb8ed2c62d04aff08f810e7a2b (HEAD -> master, origin/master, origin/HEAD)
Author: Peter Johnson <insomnia@rcpt.at>
Date:   Thu May 28 18:04:11 2020 +0200

    docs: add project header to readme (#391)

node --version
v12.16.3

missinglink commented 4 years ago

@toton6868 I think your issue is unrelated, did you try updating the metadata as per https://github.com/pelias/geonames#updating-metadata

pelias / geonames

Potential failure downloading metadata #387