IATI Datastore down

[markbrough]

The IATI Datastore is down: http://datastore.iatistandard.org/api/1/access/activity.xml

The URL tries to load for 5 minutes, and then responds with Error 500. Internal Server Error.

Also, are there alerts set up to identify when the Datastore is down? NB @andylolz set up a useful service here: https://status.codeforiati.org/

Which also has a Twitter bot: https://twitter.com/iatistatus

[matmaxgeds]

Hi.......we are still timing out in Somalia (a live production system) on the datastore today - any update on this?

[PetyaKangalova]

Thank you for flagging the issue.

Our developers have now looked into the issue and investigated that Apache required restart - giving a consistent " (32)Broken pipe: core_output_filter: writing data to the network" error in the logs, which is likely to have been caused by some change in the vps.net network which required a restart.

The Apache service has been restarted and site is now back.

[matmaxgeds]

Thanks @PetyaKangalova :) this seems like a clear hint that prioritizing in the workplan a set of tests that run hourly on all the live services that would alert the relevant people if something like this happens - as even though being phased out in this case, having a production system down for 5 days is not ideal, and it can't sensibly rely on users to do the alerts, and mostly it is a very easy fix

[matmaxgeds]

@PetyaKangalova please can you confirm the DSv1 is working this morning?

[andylolz]

Related: It would be good to fix this email address:

[PetyaKangalova]

@andylolz our developers have done a full server reboot. Let me know if you still experiencing any issues. I'll flag the email to them as well.

[andylolz]

Screenshot is from this morning (2020-12-17 10:56:55 UTC). IATI Status was still complaining this morning, too. So I’m suspicious there may still be issues. Seems to be working right now, though.

I'll flag the email to them as well

Great – thanks

[andylolz]

Let me know if you still experiencing any issues

@PetyaKangalova There are definitely still problems. It looked slightly healthier after the reboot, but IATI Status suggests it didn’t manage a successful update.

[PetyaKangalova]

@andylolz would you be able to provide more detail on what the issues reported in the IATI status are? I am on leave from tomorrow - fyi @WendyMAThomas

[matmaxgeds]

Hi - I tested today with a simple query (maybe the query is wrong?) - got the 500 server error again - I think wget has a 900 second timeout - see pic

[matmaxgeds]

Also a look here: https://status.codeforiati.org/ suggests that it has been down at some point for 13 or the last 15 days

[andylolz]

IATI Status runs the same tests as https://github.com/IATI/IATI-Website-Tests.

There are more details in this blogpost.

An example call is: http://datastore.iatistandard.org/api/1/access/activity.xml?limit=0&last-change__gt=2020-12-20

…to check for recent activities.

[matmaxgeds]

@WendyMAThomas is it possible to get an update on this - we are using this datastore live in Somalia and it has been down for 14 of the last 16 days - I am worried that it is going to be down over the whole holiday period.

[dalepotter]

Is there any published SLA for the myriad of IATI tools and services? Perhaps it would be good to introduce this to manage expectations and obligations on all sides...

[matmaxgeds]

@dalepotter - that would be incredibly helpful, also for convincing donors to fund systems that will rely on IATI infrastructure. I raised it in the workshops done by Dan earlier this year but not sure whether it was taken up.

[WendyMAThomas]

Hi all, Thanks for providing more info. @matmaxgeds We are looking into what is going on with your example; I’ll update here once we know a bit more. (Like others, we are in the holiday period and thus have only a skeleton crew before we break for Christmas at midday tomorrow. But I’ll come back with an update before then.) @dalepotter SLAs will be part of our non-functional requirements (as per Stocktake feedback from me as well as others!). We have drafted those now and they are with our Board focal points for feedback; they will be shared with the wider community later in January. Once they are agreed we will look across the IATI estate to determine where and what is needed for SLAs.

[matmaxgeds]

Thanks @WendyMAThomas - messages left here will get though to me during my hols too in case I can help test anything :) Good news on the SLAs too - IATI is maturing!

[andreaszenasidi]

@matmaxgeds Hi Matt, sorry to hear you are having issues with the datastore. Based on the screenshot you shared you are trying to GET this data: http://datastore.iatistandard.org/api/1/access/activity.csv?reporting-org=XI-IATI-WBTF&recipient-country=SO

I did some testing and it seems like the datastore and the API is working fine. I can see both of these datasets: XML format: http://datastore.iatistandard.org/api/1/access/activity.xml?reporting-org=XI-IATI-WBTF&recipient-country=SO CSV format: http://datastore.iatistandard.org/api/1/access/activity.csv?reporting-org=XI-IATI-WBTF&recipient-country=SO Using the Query Builder:

Please let me know if this is not the data you are looking for.

[matmaxgeds]

Thanks @andreaszenasidi that sounds like good news - when I click the link you sent then I get the xml/csv file back :)

Can you tell me what I was doing wrong with using GET? I am not enough of a programmer to know the difference between this and clicking a link in the query-builder, and I often use the same request so thought that would be quicker than always going via the QB - and I thought that I had done this successfully in the past, is it possible that something has changed that means it would not work now?

Edit: my wget approach is also working again now so it seems like that might not be the issue - did you restart the server or fix something else?

[andreaszenasidi]

@matmaxgeds Glad to hear you can access the data. I don't think you were doing anything wrong, you should be using GET to return data. And while I wanted to suggest to you to change your WGET commands, I can see your edit now that was not the root of the problem.

I am not aware of any changes from our side. @WendyMAThomas are you aware of any changes?

[WendyMAThomas]

@andreaszenasidi thanks for your help here! @matmaxgeds I am not aware of any changes on our side since last week's full server reboot. Keep us posted and obviously let us know if anything goes awry. (Just to note though we are closed for the holidays from midday today; Andi and Petya are back on the 30th.) Have a good Christmas!

[matmaxgeds]

Hi @andreaszenasidi @WendyMAThomas - timing out again for me today I am afraid (also for the links above) - is anyone around to fix it?

[andreaszenasidi]

@matmaxgeds Hi Matt, sorry for the delayed response, our office was closed until today.

I checked the API and today it seems to work fine again. I flagged the issue to the developers, as soon as I hear back from them I will inform you. Apologies for the inconvenience caused.

[matmaxgeds]

Thanks @andreaszenasidi - whatever you do has the magic touch but maybe worth working out what is happening as the last two times it was fixed it went down again

[andreaszenasidi]

Hi @matmaxgeds, Happy New Year to you! As promised, I am getting back to you about the reported Datastore downtime. Looks like since you last reported there were no issues with the old Datastore. We have alerts setup now in case it goes down again and the developers will be ready to reboot the servers if this occurs.

Going forward we encourage users to use the new Datastore. We are happy to support you if you choose to transition, please don't hesitate to email us at support@iatistandard.org; we would be happy to work with you directly on this.

[matmaxgeds]

@andreaszenasidi thanks a lot/we are very grateful - we tried to change over in Nov but our programmer found that the redirects were not 100% identical so it upset our parsers - I think something to do with changing where different things were placed in the heirarchies, I am expecting a report from him on what the exact issues are, plus the time cost so we can work out the way forward, hoping next week and will get straight back to you when I have it as I am hoping it is just a few easy tweaks.

[andylolz]

We have alerts setup now in case it goes down again and the developers will be ready to reboot the servers if this occurs.

This is great to hear – thanks.

[andreaszenasidi]

@andreaszenasidi thanks a lot/we are very grateful - we tried to change over in Nov but our programmer found that the redirects were not 100% identical so it upset our parsers - I think something to do with changing where different things were placed in the heirarchies, I am expecting a report from him on what the exact issues are, plus the time cost so we can work out the way forward, hoping next week and will get straight back to you when I have it as I am hoping it is just a few easy tweaks.

@matmaxgeds Hi Matt, hope you are well. I am reaching out to follow up on our previous conversation. How are you getting on with the transition to use the new Datastore? Let me know if you have any questions or concerns.

[matmaxgeds]

@andreaszenasidi thanks for the followup - we have transitioned to the new DS :) but in the end did this by adjusting our parser as things like the IATI version are in a different place in the structure in DSv1 output as compared to DSv2 and I guessed that is not something that DSv2 is going to change? I think this means that anyone using the redirects whose parser is not already built to handle this (i.e. is looking for the '@version' in the 'iati-activity' tag for each activity, not in 'iati-activities' tag for the whole file, will break?

Compare: http://datastore.iatistandard.org/api/1/access/activity.xml?recipient-country=SO

<result>
<ok>True</ok>
<iati-activities generated-datetime="2021-01-27T11:45:07.059040">
<query>
<total-count>7959</total-count>
<start>0</start>
<limit>50</limit>
</query>
<iati-activity iati-extra:version="2.03" default-currency="USD" hierarchy="1" last-updated-datetime="2020-12-08T10:27:56" linked-data-uri="http://www.unfpa.org/transparency-portal">
<iati-identifier>41119-SO-O1-RT</iati-identifier>
<reporting-org ref="41119" secondary-reporter="0" type="40">
<narrative>United Nations Population Fund</narrative>
</reporting-org>

with: http://datastoreredirect.iatistandard.org/api/1/access/activity.xml?recipient-country=SO - which redirects to: https://iatidatastore.iatistandard.org/search/activity?q=recipient_country_code:(SO)&wt=xslt&tr=activity-xml.xsl&rows=5000000

<?xml version="1.0" encoding="UTF-8"?><iati-activities version="2.03">
<iati-activity last-updated-datetime="2020-08-27T15:40:15.000001Z" xml:lang="en" default-currency="GBP" humanitarian="0" hierarchy="1" linked-data-uri="">
<iati-identifier>GB-GOV-13-FUND--GCRF-NE_P015786_1</iati-identifier>
      <reporting-org ref="GB-GOV-13" type="10"><narrative>UK - Department for Business, Energy and Industrial Strategy</narrative></reporting-org>

[andreaszenasidi]

@matmaxgeds we are happy to hear that you successfully transitioned to the new Datastore!

The DSv1 and DSv2 structures are indeed different, this is because the DSv1 was an iati-like format, while DSv2 structure matches the iati standard structure. In your example, the @version should be on 'iati-activities' not on each 'iati-activity'. This means that depending on how people parse these API responses they might need to make adjustments when they transition from DSv1 to DSv2.

[matmaxgeds]

@andreaszenasidi - I think this means the redirects will not offer backwards compatibility for existing users whose parsers probably expect to find the @version in the place they have been getting it from in DSv1 output? We had been doing a lot of tweaking (including creating the whole redirects setup) to prevent the move to DSv2 breaking existing users systems - it sounds like you are saying that moving the @version is a bridge to far?

[andreaszenasidi]

@matmaxgeds exactly, there is no backward compatibility, and this is the expected behaviour. This Discuss post explains a bit more about the structural changes users will be facing when they move from the DSv1 API to the new one.

[matmaxgeds]

@andreaszenasidi - I had thought that we were aiming for backwards compatibility so that the move to DSv2 did not break existing systems - good to clear that up then - lucky we had a programmer available!