Closed missinglink closed 2 years ago
agh woops, so the xsv
failure was my fault since I wan't explicitly telling it the file was TSV
instead of CSV
:
curl -s http://download.geonames.org/export/dump/countryInfo.txt | sed '/^#/d' | xsv cat -d '\t' rows
I suspect there's just a weird bug in csv-parse
opened an issue upstream https://github.com/adaltas/node-csv/issues/325 hopefully we can remove these commits if a solution can be found within that lib natively.
as mentioned in https://github.com/pelias/geonames/issues/404#issuecomment-1055506176 there seems to be a weird bug with how the geonames metatadata files are encoding comments, (I think!)
using this custom comment handler stream we're able to work around the issue, although I'm still not clear why the
comment
option from https://csv.js.org/parse/options/ (andsed '/^#/d'
) doesn't do the same thing 🤷I've also taken the opportunity to do some simple housekeeping tasks:
csv-parse
modulebom
option forcsv-parse
as we have done in other modulesmkdirp
module introduced in https://github.com/pelias/geonames/pull/185, since that time we've consolidated on Docker and Windows has made progress in its terminal utilities, I hope it's no longer required.engines
definition inpackage.json
from>=l2.0.0
to>=12.0.0
, @orangejulius is this just a typo?The 'actual work' here is:
comment: ''
in the csv options to disable stripping comments within that libsplit2
and the newthrough
streams to handle this ourselves.resolves https://github.com/pelias/geonames/issues/404