Evaluations

[andylolz]

Description

Evaluation documents consider what the activity achieved, whether the intended objectives were met, what the major factors influencing the achievement or non-achievement of the objectives were and an assessment of the impact, effect and value of the activity.

This is only expected if the activity is in the completion or post-completion phase. This is not expected if the sector code is administrative costs (91010).

Proposed test

For each current activity,
i​f `activity-status/@code` i​s one of (3, 4)
a​nd `sector/@code` i​s n​ot 91010
then `document-link/category[@code="A07"]` should be present

Issues identified

1. This test only considers activities in the completion or post-completion phase. However, some publishers publish mid-term evaluation documents for activities in the implementation phase. It would be preferable to give credit for these publications, without penalising publishers who do not publish these.
2. As it is currently formulated, the test expects an evaluation document as soon as an activity progresses to the completion or post-completion phase. It may be preferable to allow some grace period before expecting an evaluation document (but while still considering only current data).

[YohannaLoucheur]

It may indeed be difficult to reconcile the use of "curent data" (meaning projects that have an end date or transactions within 12 monts) with the publication of final evaluations. Depending on an organization's business model, there may well be more than 12 months between the date when the project transitions to completion and the publication of the evaluation, without transactions during that period. Perhaps this indicator should use a different data basis, eg "Activities in post-completion phase with the last date (end date, or last transaction) within 36 months"?

On Issue 1, perhaps a fairly simple approach would be to check the presence of document type A07 in projects at the implementation phase, and add a small bonus to the indicator score.

[abdulriza]

Fully agree with both suggestions. Due to the second (without grace period) organizations might not be in position to share all evaluation documents at the time of the assessment and therefore, will lose some of the points allocated for evaluations. May be some of those lost points could be compensated for those organizations who have published mid-term evaluation documents? I think, this way we can give credit to organizations who publish mid-term evaluations without penalizing those who do not.

For example, what if we say the organization can score maximum of 50% of the lost points if they have published mid-term evaluation documents for ongoing activities? With the proposed test, an organization with 100 activities with completion or post-completion phase will score 4 out of 5 for evaluation, if they have only published evaluations for only 80 activities . But if they have published minimum of 20 mid-term evaluations for ongoing projects, then they will get compensated by 0.5 more points.

[agnessurry]

The assessment should consider lag time from completion to publication of project completion report.

[francescafo]

I fully agree on the time lag issue. I also suggest considering that sometimes evaluations are thematic or per programme and not at activity level. Considering the high number of activities published by the European Commission (all services) it would be impossible to have one single evaluation for each of our contract (= level 2 = activity level).

[andylolz]

On Issue 1, perhaps a fairly simple approach would be to check the presence of document type A07 in projects at the implementation phase, and add a small bonus to the indicator score.

Thanks for this suggestion, @YohannaLoucheur.

[andylolz]

On issue 2, Document Category code A07 is called ‘Review of project performance and evaluation’. So, we should be clear that this does not necessarily need to be a full evaluation – a review document that can be published early and is potentially of use would be sufficient. We will be reviewing the definition accordingly.

[andylolz]

@francescafo: we allow organisations to specify the hierarchy at which they provide an evaluation document. Publishers are not penalised for publishing evaluations at a higher level.

[YohannaLoucheur]

@francescafo: we allow organisations to specify the hierarchy at which they provide an evaluation document. Publishers are not penalised for publishing evaluations at a higher level.

It's unclear to me how this works (ie addresses the issue), since the test is done on the basis of activities.

[AndieVaughn]

It is unrealistic that 100% of programs and projects need an evaluation. A full score should be a percentage and not 100% in order to account for the fact that some activities do not require evaluations (beyond administrative costs).

Additionally, to echo comments above, it is unrealistic to assume a evaluation should be completed immediately upon the completion of an activity. There should be grace period before the presence of an evaluation is assessed.

[ToonVB]

Agree with @AndieVaughn : in fact, only C01 type of aid activities should be tested in our view, see our comments reg. objectives which I take the liberty to reiterate below. Secondly, an evaluation can indeed take a long time to be finalized.

We think that it's neither useful to require objectives in the case of (assessed or voluntary) core-support to multilateral organisations (B02). Moreover: in the case of core-support to Belgian NGO's (B01), in which we co-fund programs, we (will) require them to publish their activities in IATI format themselves, as UK and NL are already doing as well. So we will only publish the commitment and disbursement, leaving them to publish activities, objectives, results and so on. The same applies for disbursements type of aid B03 and B04. And more generally, in fact objectives and results are only formulated in the case of C01 - projects.

[andylolz]

Thanks for the comments here.

We intend to modify the test to:

For each current activity,
i​f `activity-status/@code` i​s one of (3, 4)
a​nd `default-aid-type/@code` i​s n​ot G01
o​r `document-link/category[@code="A07"]` is present
then `document-link/category[@code="A07"]` should be present

The effect being if an evaluation is present even for an ongoing project, credit is given.

On the second issue identified, we intend to reword the indicator definition to be clear that we are not necessarily expecting a full evaluation – a review document that can be published early and is potentially of use would be sufficient.