search

pwyf / 2014-technical-consultation

Consultation for the 2014 Aid Transparency Index test
MIT License
2 stars 0 forks source link

Current data test #1

Open markbrough opened 10 years ago

markbrough commented 10 years ago

Description The Index only considers current data. We define current data as:

2013 Index test

activity-date[@type='end-planned']/@iso-date or activity-date[@type='end-planned']/text() or activity-date[@type='end-actual']/@iso-date or activity-date[@type='end-actual']/text() or transaction-date/@iso-date (for any transaction) is less than 13 months ago?

Issues

  1. Donors with (sometimes very old) projects which have closed, but are still receiving repayments, would count as current data because there are ongoing transactions.
  2. Donors with (sometimes very old) projects which are ongoing, but started a long time ago, would count as current data because the project is still ongoing.
  3. The expected time period for publication of information on indicators differs. For example, post-completion documents such as evaluation documents may not be expected for some time after the actual end of the project.

Questions

  1. Should projects that are closed, but still receiving repayments or reimbursements, count as current data?
  2. How would "closed" be defined?
  3. Should projects that started a long time ago be excluded? How long ago is long enough to be excluded?
  4. If a project is continuing for another (e.g.) 5 years, should that project then still be counted as current?
  5. How should we define current data for indicators applicable only after a project has completed?

2014 Index test How could we adjust the test to address these issues?

ErinCoppin commented 10 years ago

1) I think that projects that are closed but still receiving repayments or reimbursements should count as current data. If the donor is publishing up to date information about transactions this should be rewarded in the score. We (transparency advocates) want to reward transparency, so we should encourage all kinds of publication. Also... 2) Defining 'closed' is hard, especially if there are still ongoing transactions and possibly post-completion documents forthcoming. 3) I don't think that it is necessary to exclude projects that started a long time ago. If there remains activity on a project (even in the form of transactions) we would like to reward transparency about this. We want to avoid scoring the publication of a 'back catalogue of data' that has no relevance to the aim of improving management data, but the start date of a project is not the defining feature of this. The defining feature would be lack of any activity relevant to the project (including transaction dates) within the 12 month period, I think.
4) Yes, a project that is continuing for another 5 years should be counted as current.
5) Indicators applicable after a project is completed - I may not be understanding the problem correctly. Is the problem that the lack of score on one particular indicator would affect the rest of the score for that activity? A few options:

a) Is the problem that say, an evaluation may not have been published so that indicator would have no score because of having no current data? That doesn't seem a problem to me - the rest of the indicators for that project would still score. b) If the problem is that an evaluation published 2 years after the project ended suddenly counts as current data for that indicator, that also seems fine to me. The whole of the project's data should then be considered current. Considering users, if a user sees an evaluation of a project they will want to access other data about that project so should be able to, and the donor should be rewarded if that information remains available.
c) A more thorny problem is if the evaluation is published on IATI more than a year after it was published elsewhere - I'm not sure this should then count as current data, but I'm not sure if there is a way of comparing these two pieces of information.

The aim is to improve data quality going foward and to incentive donors to capture information in their management systems which can then be automatically pushed into IATI. Ensuring that current/recent information is scored without too many caveats seems an important part of this. Back catalogue data is not as important as current data but we don't score data that is over a year old anyway.

johnadamsDFID commented 10 years ago

Here's a radical thought. Only measure projects that started after the formal agreement of the IATI standard in February 2011? That avoids over-weighting the historical back-catalogue, and focussing on the most current collection of data.

This applies more to document tests than to transaction tests. I would expect all transactions to be available, but I would not expect documents created before 2011 to be available. So if a project was initiated in 2009 I would not expect to see the 2009 business case.

akshaysinha commented 10 years ago

Should projects that are closed, but still receiving repayments or reimbursements, count as current data?------ I think such projects should still be published in the registry. We currently post all projects with any transactions in the registry, including ones that are closed but have adjustments in the reporting period. This is important for transparency and does add value in looking at a full set of projects data for a country level analysis. If these are excluded then you risk losing historicals.

How would "closed" be defined?------ Each organization or entity defines this differently in their financial systems and programmaticly. Most organization policies require that projects go through these stages: ongoing; operationally closed; and financially closed. Financial closure generally must take place within 12 months of operational closure. The current IATI codes allow for all these stages in addition to cancelled. We think this covers all project status. Therefore, if a project is operationally closed, it can still be marked in IATI as completion, while the financial closure is in process. Once that is completed, a post-completion status can be used while the appraisal or the evaluation stage is in progress. This allows for organizations to still continue posting documents or show transactions in each phase as required.

Should projects that started a long time ago be excluded? How long ago is long enough to be excluded?----- I think anything and everything posted- even closed 10 years ago makes sense for trend analysis, time value analysis etc., it might not be feasible and could create data dump. It does make sense however to post all projects that were in post-completion at least 2-3 years in past since they might still have some evaluation or donor document posted recently etc.

If a project is continuing for another (e.g.) 5 years, should that project then still be counted as current?------ Yes as long as there are transactions posted for that project or documents then it makes sense for being transparent. Donors for example want to see their funded project live and included. I do not see any advantage in hiding such projects. Also as just one scenario, it shows inception and pipeline work that might avoid repetition of same possible project for example by another agency/donor.

I think having accurate set of dates is extremely important in IATI datasets. We are seeing lot of missing or illogical data in these fields in the sets. For example, projected closed data = actual closed data for projects that are still active or invalid dates altogether. We have proposed some changes on these as well. Dates link closely to status so both are equally important to build a proper testing mechanism or what should be included in registry and what should be archived or deleted.

thanks, Akshay

markbrough commented 10 years ago

Thanks for all the comments.

For 2014, current data will be defined as:

(Activities that ended more than 12 months ago, but are still receiving loan or interest repayments, will not be counted as "current".)

A note on historic data

We agree with the comments made on the importance of historic data. However, the Index only measures current data, and including historic data in the Index creates a perverse incentive to remove this data as we recognise that documentation for very old activities may not always exist.

We also agree with the principle that organisations should be concentrating on publishing high-quality data for their current activities, rather than trying to retrofit old activities. Unfortunately, we think the approach suggested of measuring activities begun since January 2011 would not be workable because different organisations became involved in IATI at different points. Measuring each organisation for activities begun only after they signed IATI poses significant methodological challenges in terms of retaining a comparable and consistent measure.