search

codeforIATI / iati-data-bugtracker

🐛 A public log of issues with IATI data and metadata
https://bugtracker.codeforiati.org
3 stars 0 forks source link

OECD CRS data in IATI xml is not validated 2.02 #22

Open notshi opened 3 years ago

notshi commented 3 years ago

Explanation of the issue

OECD is now providing CRS data in XML using 2.02 version of the IATI standard.
However, that is not the case as the the codes published are not numeric.

For example, in the attached screenshot, it should be the following when using the standard: <activity-date type="1" iso-date="2018-04-17"/>

Numeric values were introduced since version 2.01 of the standard. The file in question is iati_AF_2019.xml from iati_AF.zip You can also see this information on the IATI validator.

URLs / screenshots

Screenshot-20210506161516-778x79

Does the publisher know?

Yes, I've emailed dac.contact@oecd.org and used the email template.

Additional context

I wanted to see what OECD files looked like in d-portal so used d-preview.
http://011443a5ff12f5371c1e364d560d746b.d-preview.codeforiati.org/ctrack.html#view=main

By the way, this is a slightly different IATI data issue in that this data is not in the Registry so was unsure if it should be reported here.

notshi commented 3 years ago

They responded!

Thank you for your message and thank you for pointing this issue with the IATI format version 2.02. We have fixed this and regenerated the XML files, which are available here: https://www.oecd.org/dac/financing-sustainable-development/development-finance-topics/crs-xml.htm

Looks like they have fixed the numerical values for participating-org and activity-date@type. However, there are still issues in the data and validator results still show up critical errors, so I've listed the elements for them in a follow up email.

  1. <reporting-org> is missing attribute and flag for recipient data. This seems to be addressed for donor data, at least with the sample I have looked at. So for example, where applicable: <reporting-org ref="OECD"> should be <reporting-org ref="OECD" type="40" secondary-reporter="1">

  2. <activity-status> is missing.

  3. <activity-date> is missing in some data. Either @type="1" or @type="2" must be present.

  4. <participating-org> text should be placed within a<narrative> element. So for example, where applicable: <participating-org role="1" type="10" ref="">IFAD</participating-org> should be <participating-org role="1" type="10" ref=""><narrative>IFAD</narrative></participating-org>

  5. @type is missing in some <participating-org> so for example: <participating-org role="3" ref="1312-1">Global Fund</participating-org> should be <participating-org type="10" role="3" ref="1312-1">Global Fund</participating-org>

  6. All <transaction-type code="3"/> elements are missing <transaction-date iso-date="2019-12-31"/> So for example, where applicable: <transaction><transaction-type code="3"/><value currency="USD" value-date="2019-12-31">275000.01</value></transaction> should be <transaction><transaction-type code="3"/><transaction-date iso-date="2019-12-31"/><value currency="USD" value-date="2019-12-31">275000.01</value></transaction>

  7. For each transaction it is strongly recommended that both the provider-org and receiver-org are published.

  8. The other-identifier/@owner-ref and other-identifier/@owner-name attributes were removed as of version 2.01 So for example, where applicable: <other-identifier owner-ref="CRS" owner-name="OECD DAC CRS">2019000002</other-identifier> should be <other-identifier ref="2019000002" type="A2"><owner-org ref="OECD DAC CRS"><narrative>2019000002</narrative></owner-org></other-identifier>

I've also linked each element to their guidance and standard page for each list item.

andylolz commented 3 years ago

This is great – thanks for following up on this, @notshi.

notshi commented 3 years ago

An update - they've sent an updated file with some of the fixes recommended as there are some information that are not included in the CRS database.

Fixed:

Information not included in CRS database unless reported:

Having looked through their data, I've responded with the following and included a fixed xml file as reference.

  1. IATI XML is very particular about the order of the elements in the xml, otherwise the file will not be validated. As such, I've included a fixed XML in the right order, for your reference - hopefully, this will be of use to generate the finalised file. This is highly recommended by the IATI Secretariat but in general, and as data consumers, this order is not strictly upheld or needed so it is entirely your decision whether to follow it.

  2. The attribute <type> should be in lowercase. <other-identifier ref="2019000001" Type = "A2"><owner-org ref="OECD DAC CRS"><narrative>2019000001</narrative></owner-org></other-identifier> should be <other-identifier ref="2019000001" type = "A2"><owner-org ref="OECD DAC CRS"><narrative>2019000001</narrative></owner-org></other-identifier>

  3. There is a hypen in the <channel-code> attribute <crs-add><channelcode>12000</channelcode></crs-add> should be <crs-add><channel-code>12000</channel-code></crs-add>

  4. Technically, this file can be validated as version 2.03, which is the latest version of the standard so it is possible to change the version of your files to 2.03 <iati-activities version = "2.02" generated-datetime ="2021-07-08T19:45:28"> should be <iati-activities generated-datetime="2021-07-08T19:45:28" version="2.03">

codeforIATIbot commented 3 years ago

Hello! There has been no activity on this issue in the last 30 days. I wonder if it has now been resolved?

If you’re reading this, would you mind checking to see if the issue is still applicable?

Thank you!

notshi commented 3 years ago

Have sent a nudge to OECD contact.

notshi commented 3 years ago

They've responded with a fixed file! All looks good so they should be applying the requested changes to all OECD XML data, hopefully in time.

Note: The data will not pass any schema validation as OECD data is different, as noted in https://github.com/codeforIATI/iati-data-bugtracker/issues/22#issuecomment-878340073.

notshi commented 3 years ago

You can view all the updated OECD data on http://d-preview.codeforiati.org/ctrack.html#view=main

At present, this should be the complete 2018 and final 2019 Creditor Reporting System (CRS) data in IATI XML format, by donor and by recipient.

However, upon closer inspection, there seems to be transactions from as far back as 1986 so have asked OECD to confirm this.

Their response:

Thank you for your message. I check the transaction-date item in the IATI standard web page https://iatistandard.org/en/iati-standard/203/activity-standard/iati-activities/iati-activity/transaction/transaction-date/ and this item refers to : ‘The date on which the transaction was made or (in the case of commitments) agreed’

So this date refers to the commitment date, when exists in CRS database.

This makes sense if the transaction-type is Outgoing Commitment but it looks like the data is not quite so.

Disbursement in 1986 http://d-preview.codeforiati.org/ctrack.html#view=act&aid=909-2-2018-1986000523

Outgoing Commitment and Disbursement in 2016 http://d-preview.codeforiati.org/ctrack.html#view=act&aid=918-3-2018-2016002831

At this point, I am unsure how to move forward so would appreciate any advice.

matmaxgeds commented 3 years ago

In case it helps - here is what the 2018 crs text file has on the 1986 disbursement hospital IMPROVEMENTS_OF_NICKERIE_HOSPITAL.ods which suggests to me that the commitment was in 1986 (hence also the CRS ID starting with 1986), but the disbursement was in 2018, which seems to be a minor adjustment on a loan which is in repayment between 1996 and 2022 - maybe they made an overpayment that was met with a small disbursement to refund etc?

On the second example, the 'value-date' of the transactions is 2018, but the date set for the transaction is set in the xml to the project start date in 2016.....seems like the OECD converter might have picked up the wrong date? I also couldn't easily grep this project in the 2018 crs text file.

notshi commented 3 years ago

Thanks, @matmaxgeds for taking a look at this.

It'll be some time before I have some free time to take a look at this further and digest properly your response but was wondering where you got that crs text file from?

In any case, your theory seems sound and reflects what we assumed as well; ie. that there was a commitment in the past and a disbursement took place in the year 2018. But since there are no activity dates attached, it's hard to figure that out.

Interestingly, if I do a text search for 'Nickerie Hospital', I get 2 activities, one of which is the first example mentioned. The other activity has a Loan Repayment without a date. http://d-preview.codeforiati.org/ctrack.html?text_search=Nickerie%20Hospital#view=act&aid=909-1-2019-2019000631

Thanks for raising the value-date and iso-date discrepancy. This could be something I can raise with OECD.

I'm mostly a bit wary about responding or giving suggestions to OECD at this stage as this is now touching on how OECD reports their data and am not at all familiar with their methodology without spending some time researching and going through the OECD standard.

matmaxgeds commented 3 years ago

No probs - is so rare I can actually help out!

RE text files, I go to this site: https://stats.oecd.org/Index.aspx?DataSetCode=crs1 and then under the 'Export' menu, click on 'related files' which links to zips of text files (with '|' as the column delimiter) going back to 1973....which until IATI came along, my my main route for doing anything aid data related more than getting a single stat of the crs website.

codeforIATIbot commented 2 years ago

Hello! There has been no activity on this issue in the last 30 days. I wonder if it has now been resolved?

If you’re reading this, would you mind checking to see if the issue is still applicable?

Thank you!

codeforIATIbot commented 2 years ago

Hello! There has been no activity on this issue in the last 30 days. I wonder if it has now been resolved?

If you’re reading this, would you mind checking to see if the issue is still applicable?

Thank you!

codeforIATIbot commented 2 years ago

Hello! There has been no activity on this issue in the last 30 days. I wonder if it has now been resolved?

If you’re reading this, would you mind checking to see if the issue is still applicable?

Thank you!

codeforIATIbot commented 2 years ago

Hello! There has been no activity on this issue in the last 30 days. I wonder if it has now been resolved?

If you’re reading this, would you mind checking to see if the issue is still applicable?

Thank you!

codeforIATIbot commented 2 years ago

Hello! There has been no activity on this issue in the last 30 days. I wonder if it has now been resolved?

If you’re reading this, would you mind checking to see if the issue is still applicable?

Thank you!

codeforIATIbot commented 2 years ago

Hello! There has been no activity on this issue in the last 30 days. I wonder if it has now been resolved?

If you’re reading this, would you mind checking to see if the issue is still applicable?

Thank you!

codeforIATIbot commented 2 years ago

Hello! There has been no activity on this issue in the last 30 days. I wonder if it has now been resolved?

If you’re reading this, would you mind checking to see if the issue is still applicable?

Thank you!

codeforIATIbot commented 2 years ago

Hello! There has been no activity on this issue in the last 30 days. I wonder if it has now been resolved?

If you’re reading this, would you mind checking to see if the issue is still applicable?

Thank you!

codeforIATIbot commented 2 years ago

Hello! There has been no activity on this issue in the last 30 days. I wonder if it has now been resolved?

If you’re reading this, would you mind checking to see if the issue is still applicable?

Thank you!

codeforIATIbot commented 2 years ago

Hello! There has been no activity on this issue in the last 30 days. I wonder if it has now been resolved?

If you’re reading this, would you mind checking to see if the issue is still applicable?

Thank you!

codeforIATIbot commented 2 years ago

Hello! There has been no activity on this issue in the last 30 days. I wonder if it has now been resolved?

If you’re reading this, would you mind checking to see if the issue is still applicable?

Thank you!

codeforIATIbot commented 2 years ago

Hello! There has been no activity on this issue in the last 30 days. I wonder if it has now been resolved?

If you’re reading this, would you mind checking to see if the issue is still applicable?

Thank you!

codeforIATIbot commented 1 year ago

Hello! There has been no activity on this issue in the last 30 days. I wonder if it has now been resolved?

If you’re reading this, would you mind checking to see if the issue is still applicable?

Thank you!

codeforIATIbot commented 1 year ago

Hello! There has been no activity on this issue in the last 30 days. I wonder if it has now been resolved?

If you’re reading this, would you mind checking to see if the issue is still applicable?

Thank you!

codeforIATIbot commented 1 year ago

Hello! There has been no activity on this issue in the last 30 days. I wonder if it has now been resolved?

If you’re reading this, would you mind checking to see if the issue is still applicable?

Thank you!

codeforIATIbot commented 1 year ago

Hello! There has been no activity on this issue in the last 30 days. I wonder if it has now been resolved?

If you’re reading this, would you mind checking to see if the issue is still applicable?

Thank you!

codeforIATIbot commented 1 year ago

Hello! There has been no activity on this issue in the last 30 days. I wonder if it has now been resolved?

If you’re reading this, would you mind checking to see if the issue is still applicable?

Thank you!

codeforIATIbot commented 1 year ago

Hello! There has been no activity on this issue in the last 30 days. I wonder if it has now been resolved?

If you’re reading this, would you mind checking to see if the issue is still applicable?

Thank you!

codeforIATIbot commented 1 year ago

Hello! There has been no activity on this issue in the last 30 days. I wonder if it has now been resolved?

If you’re reading this, would you mind checking to see if the issue is still applicable?

Thank you!

codeforIATIbot commented 1 year ago

Hello! There has been no activity on this issue in the last 30 days. I wonder if it has now been resolved?

If you’re reading this, would you mind checking to see if the issue is still applicable?

Thank you!

codeforIATIbot commented 1 year ago

Hello! There has been no activity on this issue in the last 30 days. I wonder if it has now been resolved?

If you’re reading this, would you mind checking to see if the issue is still applicable?

Thank you!

codeforIATIbot commented 1 year ago

Hello! There has been no activity on this issue in the last 30 days. I wonder if it has now been resolved?

If you’re reading this, would you mind checking to see if the issue is still applicable?

Thank you!

codeforIATIbot commented 1 year ago

Hello! There has been no activity on this issue in the last 30 days. I wonder if it has now been resolved?

If you’re reading this, would you mind checking to see if the issue is still applicable?

Thank you!

codeforIATIbot commented 1 year ago

Hello! There has been no activity on this issue in the last 30 days. I wonder if it has now been resolved?

If you’re reading this, would you mind checking to see if the issue is still applicable?

Thank you!

codeforIATIbot commented 1 year ago

Hello! There has been no activity on this issue in the last 30 days. I wonder if it has now been resolved?

If you’re reading this, would you mind checking to see if the issue is still applicable?

Thank you!

codeforIATIbot commented 11 months ago

Hello! There has been no activity on this issue in the last 30 days. I wonder if it has now been resolved?

If you’re reading this, would you mind checking to see if the issue is still applicable?

Thank you!

codeforIATIbot commented 10 months ago

Hello! There has been no activity on this issue in the last 30 days. I wonder if it has now been resolved?

If you’re reading this, would you mind checking to see if the issue is still applicable?

Thank you!

codeforIATIbot commented 9 months ago

Hello! There has been no activity on this issue in the last 30 days. I wonder if it has now been resolved?

If you’re reading this, would you mind checking to see if the issue is still applicable?

Thank you!

codeforIATIbot commented 8 months ago

Hello! There has been no activity on this issue in the last 30 days. I wonder if it has now been resolved?

If you’re reading this, would you mind checking to see if the issue is still applicable?

Thank you!

codeforIATIbot commented 7 months ago

Hello! There has been no activity on this issue in the last 30 days. I wonder if it has now been resolved?

If you’re reading this, would you mind checking to see if the issue is still applicable?

Thank you!

codeforIATIbot commented 6 months ago

Hello! There has been no activity on this issue in the last 30 days. I wonder if it has now been resolved?

If you’re reading this, would you mind checking to see if the issue is still applicable?

Thank you!

codeforIATIbot commented 5 months ago

Hello! There has been no activity on this issue in the last 30 days. I wonder if it has now been resolved?

If you’re reading this, would you mind checking to see if the issue is still applicable?

Thank you!

codeforIATIbot commented 4 months ago

Hello! There has been no activity on this issue in the last 30 days. I wonder if it has now been resolved?

If you’re reading this, would you mind checking to see if the issue is still applicable?

Thank you!

codeforIATIbot commented 3 months ago

Hello! There has been no activity on this issue in the last 30 days. I wonder if it has now been resolved?

If you’re reading this, would you mind checking to see if the issue is still applicable?

Thank you!

codeforIATIbot commented 2 months ago

Hello! There has been no activity on this issue in the last 30 days. I wonder if it has now been resolved?

If you’re reading this, would you mind checking to see if the issue is still applicable?

Thank you!

codeforIATIbot commented 1 month ago

Hello! There has been no activity on this issue in the last 30 days. I wonder if it has now been resolved?

If you’re reading this, would you mind checking to see if the issue is still applicable?

Thank you!

codeforIATIbot commented 1 week ago

Hello! There has been no activity on this issue in the last 30 days. I wonder if it has now been resolved?

If you’re reading this, would you mind checking to see if the issue is still applicable?

Thank you!