curl returns an error code for ausgov data

codeforIATI / iati-data-dump

📷 A daily snapshot of all IATI data on the IATI Registry

https://iati-data-dump.codeforiati.org

GNU General Public License v3.0

2 stars 1 forks source link

curl returns an error code for ausgov data #1

Closed andylolz closed 3 years ago

andylolz commented 4 years ago

wget appears to get stuck while trying to download data from ausgov. Changing timeouts doesn’t appear to help.

I’ve just tried switching to curl, which might help.

andylolz commented 4 years ago

This is probably a good opportunity to switch to use the dportal downloader!

andylolz commented 3 years ago

We’ve now switched to use the dportal downloader. We download using curl, and we’ve changed the User-Agent to something the ausgov server is happy with. Downloads work when running locally, but fail on heroku. I suspect the IP address has been blacklisted, though I’m not sure why this would be.

andylolz commented 3 years ago

Switching to github actions has resolved this.