Additional Checks for IATI

robredpath commented 7 years ago

A separate run of the bdd-tester (or similar)
Should report how many activities passed(failed?) the test, even if it's zero
Exact test details will be provided before the sprint starts, likely to be relating to fields present/not present

robredpath commented 7 years ago

I've asked @stevieflow for exact details

robredpath commented 7 years ago

@stevieflow are you ok if this is cmdline only at the first iteration - ie, only available to the toolkit?

stevieflow commented 7 years ago

@robredpath seems fine

robredpath commented 7 years ago

I have the details for this, I'll pass them to @edugomez ASAP

robredpath commented 7 years ago

@edugomez

The work here is to add a separate run of bdd-tester to IATI CoVE. It only needs to give results when run via the cli. Note the main difference to other checks being that these should always report, even if no/all activities trigger the test.

The tests are:

Presence of AGROVOC classifications, at first using @rory09 's extension and later using the proposed element in 2.03. It may well be that it's possible to test for them both at the same time - I believe I talked with @Bjwebb about this so it's worth talking to him about implementation
Presence of geolocations. Talk to @rory09 or @stevieflow for examples of what this looks like.
Are org IDs using recognised prefixes from org-id.guide? It's fine for this to be an offline implementation if live lookups prove difficult.

@tobybatch would probably find it useful to know the output design for these earlier rather than later, but that's an assumption. Definitely worth checking in with him.

Bjwebb commented 7 years ago

Presence of AGROVOC classifications

We have some validation of the tag element, based on an XML schema definition - https://github.com/OpenDataServices/cove/issues/835#issuecomment-330526780 - but this checks for the correct attributes if a tag element is included.

However for our AGROVOC additional check we want to test that there exists at least one tag with specifically a AGROVOC classification; ie. with vocabulary being 98 or 99, with vocabulary-uri="http://aims.fao.org/aos/agrovoc/" and with a code attribute.

I don't think we need to check the code value itself. I'm not sure if we want to check that a narrative is present.

Bjwebb commented 7 years ago

Presence of geolocations

Talk to @rory09 or @stevieflow, but some useful links for that conversation are:

The iati standard location docs - http://iatistandard.org/202/activity-standard/iati-activities/iati-activity/location/
The open ag framework sub national location docs http://openagfunding.opendataservices.coop/en/latest/framework/#sub-national-location
The IATI dashboard checks of location - http://dashboard.iatistandard.org/comprehensiveness.html - ctrl-f for location
PWYF test of sub national location for the index. Old test - https://github.com/pwyf/data-quality-tester/blob/226712da9c160d2ac620c0e910237020d60ea672/test_definitions/pwyf-tests.yaml#L210; new test - https://github.com/pwyf/data-quality-tester/blob/develop/test_definitions/index_2017/local_level_monitoring/sub-national_locations.feature - seems to be missing..!?

Are org IDs using recognised prefixes from org-id.guide?

For 360Giving we already pull in the IATI OrganisationRegistrationAgency, which is the precursor to org-id.guide. So, we want something similar to this, but for org-id.guide instead. We also want to migrate 360 CoVE to org-id.guide soon. https://github.com/OpenDataServices/cove/issues/760

edugomez commented 6 years ago

@stevieflow @rory09, according to this proposed schema for the tag element by @Bjwebb, the element is not mandatory. So in the additional checks for Open Ag we should check for:

At least one openag:tag for activity
At least one openag:tag with attributes for Agrovoc, that is, vocabulary="98" or vocabulary="99" and @vocabulary-uri="http://aims.fao.org/aos/agrovoc/".

Is that correct? Underlying 'at least' to make sure that is what we want.

stevieflow commented 6 years ago

@edugomez thanks

So, this would only make sense within the context of the toolkit work with @tobybatch , I think that's what @robredpath means via https://github.com/OpenDataServices/cove/issues/834#issuecomment-330799923:

It only needs to give results when run via the cli. Note the main difference to other checks being that these should always report, even if no/all activities trigger the test.

Because it's highly unlikely that a regular user of CoVE will be anywhere near the AGROVOC foo.

Have I read this correctly?

edugomez commented 6 years ago

@stevieflow That is correct, the output of the iati-cli will always include Open Ag checks. BTW, now that you mentioned this:

It only needs to give results when run via the cli. Note the main difference to other checks being that these should always report, even if no/all activities trigger the test.

I guess that these should always report means that the result of additional checks for Open Ag will be empty (e.g. {...ruleset_errors_openag: {}, ...}) if all activities are compliant with the tests (and not activity_xxx: all checks passed, etc ...)

edugomez commented 6 years ago

Rules so far for openag:tag checks:

Feature: openag:tag element is expected and must contain specific attributes

  Scenario Outline: openag-tag: element is expected
    Given an Open Agriculture IATI activity
     Then at least one `openag:tag` element is expected

  Scenario Outline: openag-tag: element must have @vocabulary attribute with code for "maintained by the Reporting Organisation"
    Given `openag:tag` elements
     Then every `openag:tag` must have `vocabulary` attribute
       And every `vocabulary` must be equal to `98 or 99`

  Scenario Outline: openag-tag: element must have @vocabulary-uri attribute with Agrovoc URI
    Given `openag:tag` elements
     Then every `openag:tag` must have `vocabulary-uri` attribute
       And every `vocabulary-uri` must be equal to `http://aims.fao.org/aos/agrovoc/`

  Scenario Outline: openag-tag: element must have @code attribute
    Given `openag:tag` elements
     Then every `openag:tag` must have `code` attribute

For location:

Feature: location element must be present and must contain location-id with attributes

  Scenario Outline: location: element is expected
    Given an Open Agriculture IATI activity
     Then at least one `location` element is expected

  Scenario Outline: location: element must include <location-id>
    Given `location` elements
     Then every `location` must include `location-id` element

  Scenario Outline: location: element must use @code attribute
    Given 'location/location-id' elements
     Then every `location/location-id` must have `code` attribute

  Scenario Outline: location: element must use @vocabulary attribute
    Given 'location/location-id' elements
     Then every `location/location-id` must have `vocabulary` attribute

For organisations @ref

Feature: organisation identifiers must use org-id prefixes 

  Scenario Outline: reporting-org: @ref should have an org-id prefix
    Given `reporting-org` organisation
     then `ref` attribute should start with an org-id prefix

  Scenario Outline: transaction-provider-organisation: @ref should have an org-id prefix
    Given `transaction/provider-org/` organisation
     then `ref` attribute should start with an org-id prefix

  Scenario Outline: transaction-receiver-organisation: @ref should have an org-id prefix
    Given `other-identifier/owner-org/` organisation
     then `ref` attribute should start with an org-id prefix

  Scenario Outline: transaction-receiver-organisation: @ref should have an org-id prefix
    Given `transaction/receiver-org/` organisation
     then `ref` attribute should start with an org-id prefix

  Scenario Outline: participating-org: @ref should have an org-id prefix
    Given `participating-org` organisations
     then every `ref` attribute should start with an org-id prefix

edugomez commented 6 years ago

@tobybatch This is how the output would look like in the results.json from the iati-cli:


...

 "ruleset_errors_ag": [
    {
      "path": "/iati-activities/iati-activity[3]",
      "message": "the activity should include at least one <openag:tag> element",
      "rule": "element is expected",
      "id": "CC-CCC-789789-CC789"
    },
    {
      "path": "/iati-activities/iati-activity[1]",
      "message": "the activity should include at least one <location> element",
      "rule": "element is expected",
      "id": "AA-AAA-123123-AA123"
    },
    {
      "path": "/iati-activities/iati-activity[2]",
      "message": "the activity should include at least one <location> element",
      "rule": "element is expected",
      "id": "BB-BBB-456456-BB456"
    },
    {
      "path": "/iati-activities/iati-activity[3]",
      "message": "the activity should include at least one <location> element",
      "rule": "element is expected",
      "id": "CC-CCC-789789-CC789"
    },
    {
      "path": "/iati-activities/iati-activity[1]/openag:tag",
      "message": "@vocabulary attribute must be equal to \"98 or 99\" (it is \"95\")",
      "rule": "element must have @vocabulary attribute with code for \"maintained by the reporting organisation\"",
      "id": "AA-AAA-123123-AA123"
    },
    {
      "path": "/iati-activities/iati-activity[2]/openag:tag",
      "message": "openag:tag element must have @vocabulary attribute",
      "rule": "element must have @vocabulary attribute with code for \"maintained by the reporting organisation\"",
      "id": "BB-BBB-456456-BB456"
    },
    {
      "path": "/iati-activities/iati-activity[1]/openag:tag",
      "message": "@vocabulary-uri attribute must be equal to \"http://aims.fao.org/aos/agrovoc/\" (it is \"http://bad.org\")",
      "rule": "element must have @vocabulary-uri attribute with agrovoc uri",
      "id": "AA-AAA-123123-AA123"
    },
    {
      "path": "/iati-activities/iati-activity[2]/openag:tag",
      "message": "@vocabulary-uri attribute must be equal to \"http://aims.fao.org/aos/agrovoc/\" (it is \"http://bad.org\")",
      "rule": "element must have @vocabulary-uri attribute with agrovoc uri",
      "id": "BB-BBB-456456-BB456"
    }
  ] 

...

Bjwebb commented 6 years ago

This is now done, and deployed to http://iati.cove.opendataservices.coop/

OpenDataServices / cove