Principle #16 maintenance - automated validation

[beckyjackson]

FP 16 - Maintenance

Automated checks:

1. Has the ontology been recently updated?

Mechanism:

This is a bit of a subjective one. The principle page states that developers must respond within three months to requests and changes in scientific consensus, but I don't think we can automatically check that. We can look at the official PURL and see if the contents have been updated within the past year. I'm open to suggestions.

[beckyjackson]

It looks like the GitHub hosted files do not have a Last-Modified header entry. Another option is to just use the version IRI, assuming they use the date format.

[nataled]

What is currently being checked (updates) is not the same as the principle. The principle is maintenance in light of scientific advance, not simply 'updated regularly'. Obviously, meeting the principle can only be done with concomitant updates, but updates can be done without meeting the principle.

[matentzn]

Do we make it clear somewhere that our automated checks are often, like in this case, just an approximation of the OBO principle? I mean, I am perhaps a bit more lenient than @nataled, but I totally agree with the sentiment - testing release frequency is not the same as "keeping up to date with the scientific advance". Some ontologies are updated weekly, with few changes between releases, and others are updated twice per year (BTO) with huge numbers of terms added.

[cmungall]

@nataled - The automated check is necessary but not sufficient. We could mark these. But how do you propose to check for maintenance in the light of scientific advances?

I propose an additional check that uses the github API to determine some kind of reasonable responsivity. Obviously many ontologies are under-resourced and cannot keep up with tickets but we can come up with some reasonable metric

[cmungall]

@beckyjackson GO is showing up as X in the dashboard here which seems unusual

when I mouseover it says something about labels...?

[nataled]

@cmungall that's basically my point--necessary but not sufficient, and the EWG has discussed marking such cases (where the automated check cannot fully verify that the principle is being followed, just hint at it). At the moment all we have is manual vigilance. Responsivity is a different principle, one that is more easily checked in the way you said.

[jamesaoverton]

From the details page for GO http://obo-dashboard-test.ontodev.com/go/dashboard.html there's a link to the ROBOT report http://obo-dashboard-test.ontodev.com/go/robot_report.html. It looks like duplicate obsolete labels. I thought this was allowed by the change in https://github.com/ontodev/robot/issues/591 so I'm surprised to see it as an error here.

[jamesaoverton]

Sorry @cmungall, I was confused. The real problem was a missing cell ("Open") in the header row of the table, causing a mis-alignment of all the columns. That's now fixed. (The X is actually under "Naming", for the duplicate obsolete labels I was talking about.)

[jamesaoverton]

The current check uses the version IRI to get the date of the current release:

• SUCCESS up to one year ago
• INFO more than one year ago OR version IRI does not include the date (e.g. ChEBI, PR)
• WARN more than two years ago
• ERROR more than three years ago

This is crude but seems reasonable to me.

[nlharris]

Is there still an action item here or can this issue be closed?

[matentzn]

I would advocate to break this down into smaller action tickets. Testing responsiveness using the github API is hard if we cannot expect everyone to actually use their repos as anything but issue trackers (like PRO). If we can, we can use obohealth toolkit to obtain all that information (@cthoyt).

I will close it now, but please, participants, reopen with clear action items.