search

IATI / D-Portal

http://d-portal.org/
Other
31 stars 23 forks source link

Missing activities - HelpAge International #607

Closed andreaszenasidi closed 3 years ago

andreaszenasidi commented 3 years ago

Help Age International is missing its activities in d-portal. They initially had some bad XML formatting errors in their dataset which was fixed and the data was republished on Feb 22nd, however, the data is still missing in d-portal.

Registry: https://iatiregistry.org/publisher/helpage_international Missing dataset: https://iatiregistry.org/dataset/helpage_international-activities D-portal: https://d-portal.org/ctrack.html#view=search&reporting_ref=GB-COH-1762840

xriss commented 3 years ago

We had a glitch last night where postgres had an automatic security updated applied during import and a bunch of files failed to import during that downtime.

I've told it not to do that again and things should all be OK after another import, so tomorrow at the latest. Will check up on this issue again then.

xriss commented 3 years ago

This does not look like it was related to last nights glitch.

The download link from https://iatiregistry.org/dataset/helpage_international-activities

which is https://www.helpage.org/download/59c231af510fb/

Is very slow and gave a 401 error in the logs from last night, which is why it is not available

http://d-portal.org/ctrack.html?#view=dash_sluglog&slug=helpage_international-activities

It will try and download it again tonight so lets see if that works.

xriss commented 3 years ago

we are getting blocked from downloading and the xml is replaced by a html page containing this text

To help us keep this website secure, please wait while we verify you're not a robot! It will only take a few seconds...

which looks like its coming from google recaptcha https://www.google.com/recaptcha/about/

So whatever is doing that needs to be disabled or we are not going to be able to get the data.

andreaszenasidi commented 3 years ago

@xriss Thanks for the update! I will reach out to the publisher about this.

notshi commented 3 years ago

Hi @andreaszenasidi any updates on this? We are still getting the same 401 error from their server.

andreaszenasidi commented 3 years ago

@notshi Sorry, no updates. The publisher said they are going to think about a solution. I will ask them for an update later this week and I will let you know.

andreaszenasidi commented 3 years ago

@notshi Seems like they made some changes because the data is available in d-portal now: https://d-portal.org/ctrack.html#view=search&reporting_ref=GB-COH-1762840

Thanks for looking into this!

notshi commented 3 years ago

That's good news, thanks for the update @andreaszenasidi

We can confirm that the number of activities matches with the registry and the server logs look healthy.