search

pelias / openaddresses

Pelias import pipeline for OpenAddresses.
MIT License
51 stars 43 forks source link

add referer header to cURL requests #483

Closed missinglink closed 3 years ago

missinglink commented 3 years ago

As discussed in https://github.com/pelias/openaddresses/issues/482 this PR adds a referer header to cURL requests sent to download data from results.openaddresses.io.

requires confirmation that it works as expected before merging. closes https://github.com/pelias/openaddresses/issues/482

missinglink commented 3 years ago

before

/code/p/docker/p/jamaica master *6 !3 ?2 ❯ pelias download oa                       17s
Creating pelias_openaddresses_run ... done
info: [openaddresses-download] Attempting to download selected data files: jm/countrywide.csv
info: [openaddresses-download] Downloading jm/countrywide.csv
warn: [openaddresses-download] failed to download https://results.openaddresses.io/latest/run/jm/countrywide.zip: Error: Command failed: unzip -o -qq -d /data/openaddresses /data/openaddresses/jm-countrywide202129-1-17celf8.93sh.zip
[/data/openaddresses/jm-countrywide202129-1-17celf8.93sh.zip]
  End-of-central-directory signature not found.  Either this file is not
  a zipfile, or it constitutes one disk of a multi-part archive.  In the
  latter case the central directory and zipfile comment will be found on
  the last disk(s) of this archive.
unzip:  cannot find zipfile directory in one of /data/openaddresses/jm-countrywide202129-1-17celf8.93sh.zip or
        /data/openaddresses/jm-countrywide202129-1-17celf8.93sh.zip.zip, and cannot find /data/openaddresses/jm-countrywide202129-1-17celf8.93sh.zip.ZIP, period.

info: [openaddresses-download] All done!

after

/code/p/docker/p/jamaica master *6 !3 ?2 ❯ pelias download oa                       15s
Pulling openaddresses (pelias/openaddresses:referer_header-2021-03-09-f21121221717132a20f58980aec5c8e05fdb7bd5)...
referer_header-2021-03-09-f21121221717132a20f58980aec5c8e05fdb7bd5: Pulling from pelias/openaddresses
...
Digest: sha256:2eb4625ac8bb3c80cd929f1073ced0cf8812eb4404064b3dcf43d3c203ef59c3
Status: Downloaded newer image for pelias/openaddresses:referer_header-2021-03-09-f21121221717132a20f58980aec5c8e05fdb7bd5
Creating pelias_openaddresses_run ... done
info: [openaddresses-download] Attempting to download selected data files: jm/countrywide.csv
info: [openaddresses-download] Downloading jm/countrywide.csv
info: [openaddresses-download] All done!
/code/p/docker/p/jamaica master *6 !3 ?2 ❯ l data/openaddresses/jm/countrywide.csv  34s
.rw-r--r-- 12M peter 28 Feb 17:47 --  data/openaddresses/jm/countrywide.csv