SSL error on retrieving Norad data

[markbrough]

http://datastore.iatistandard.org/api/1/error/dataset/norad-activity_file

While trying to retrieve file from: https://www.norad.no/contentassets/23a62a96ff8b43f1886fc0619b51214f/iati_report_20151215.xml

IATI Registry URL: http://iatiregistry.org/dataset/norad-activity_file

[andylolz]

So https://www.norad.no provides two certificates: a valid SNI one, and an invalid non-SNI one. You can see that in the test results here: https://www.ssllabs.com/ssltest/analyze.html?d=www.norad.no

requests is using the invalid non-SNI certificate, and erroring (because it’s invalid). To make requests use the SNI certificate, you need to either:

• upgrade python to >=2.7.9, or
• install a bunch of other stuff, or
• use requests.get(…, verify=False) to switch off certificate verification. But it’s doubtful that’s the best course of action.

[andylolz]

NB I was able to recreate the error on ubuntu 14.04 using python 2.7.6, and was able to resolve it via all three methods. (For the second one, I needed to apt-get install libffi-dev)

[dalepotter]

Thanks for the further investigation @andylolz

I can confirm that the Datastore is running is running an old version of python, which I have now upgraded. We'll need to wait until the datastore does it's periodic update (probably this evening) before we can see if this has fully fixed the problem with Norad's files.

We'll know the issue is resolved when this API call returns a similar total-count number to that shown on the Registry dataset: http://iatiregistry.org/dataset/norad-activity_file

[dalepotter]

Having just checked this again, both the Datastore API and the activity count in the IATI Registry metadata section for this dataset are currently both returning 3926 activities.

Given this, I will now close this issue!