search

pelias / docker

Run the Pelias geocoder in docker containers, including example projects.
MIT License
330 stars 223 forks source link

Openaddresses download failure for north-america #257

Closed collinjc closed 3 years ago

collinjc commented 3 years ago

Describe the bug pelias download oa throws the following warning when running a full north-america build:

warn: [openaddresses-download] failed to download https://results.openaddresses.io/latest/run/ca/ab/calgary.zip: Error: Command failed: unzip -o -qq -d /data/openaddresses /data/openaddresses/ca-ab-calgary202124-1-1cakyph.lcwp.zip [/data/openaddresses/ca-ab-calgary202124-1-1cakyph.lcwp.zip] End-of-central-directory signature not found. Either this file is not a zipfile, or it constitutes one disk of a multi-part archive. In the latter case the central directory and zipfile comment will be found on the last disk(s) of this archive. unzip: cannot find zipfile directory in one of /data/openaddresses/ca-ab-calgary202124-1-1cakyph.lcwp.zip or /data/openaddresses/ca-ab-calgary202124-1-1cakyph.lcwp.zip.zip, and cannot find /data/openaddresses/ca-ab-calgary202124-1-1cakyph.lcwp.zip.ZIP, period.

Upon inspection of the downloaded file, the file contents are an HTML 403 error rather than a ZIP. This is also the case when trying to hit the URL manually in a browser. One such example is https://results.openaddresses.io/latest/run/ca/ab/calgary.zip, which currently redirects to https://data.openaddresses.io/runs/1032167/ca/ab/calgary.zip.

It should be noted, that manually downloading the files from https://results.openaddresses.io/ is successful.

I think there are two potential issues here. One is that, perhaps openaddresses.io is now relying on some sort of token or referrer in order to proceed, and that behavior needs to be recreated in pelias. The other issue is that HTTP responses should be properly handled so that errors such as these are more easily surfaced.

Steps to Reproduce Steps to reproduce the behavior:

  1. Install a standard docker instance of pelias
  2. Move to the north-america project
  3. Run pelias download oa (after any prerequisites, if desired)
  4. Error

Expected behavior pelias download oa completes successfully

Environment (please complete the following information):

collinjc commented 3 years ago

This issue does seem to be related to the referer. I have opened a new bug on the pelias/openaddresses repository, as I believe that is where the fix would need to be made.

missinglink commented 3 years ago

@collinjc I'm hoping this issue can now be closed. Please pull the latest docker image(s) and try again.

collinjc commented 3 years ago

Yes, I can confirm that the issue is now resolved. Thanks!