clean and normalize ERA SKOS vocabularies

[VladimirAlexiev]

A lot of the thesauri in https://github.com/Interoperable-data/ERA_vocabulary/tree/main/era-skos are just dumps of values retrieved from the RINF and ERATV databases, with no effort at curation, normalization or merging.

• numerous categorial values have only a notation (decimal number), with no label or explanation what it means
• numeric values like are converted to skos:Concepts without any cleaning or extracting structure

Here are some examples and doubts from era-skos-PlatformHeights.ttl:

• these are obviously the same thing, so should be unified skos:prefLabel "200-550" . skos:prefLabel "200-550 mm" .
• is this the same as 220-360, or does it perhaps mean a two-part platform with two discrete heights? skos:prefLabel "220 mm, 360 mm" .
• why the first number here is greater than the second? skos:prefLabel "380 mm, 220 mm" .
• why is this not expressed as 1470 (mm)? skos:prefLabel "1,47m" .

Expressing a combination of numbers as a Concept is wrong. Instead, the possibilities should be expressed in some RDF constructs, which should be checked with SHACL. For example:

• "200-550": `minHeight 200; maxHeight 550
• "380 mm, 220 mm": `multipleHeights (380 220);

Codifying an uncontrolled RINF/ERATV field as a skos:ConceptScheme is actually harmful since it gives data providers a bad signal that they should use these unclean values in the data they export.

I appreciate that cleaning up all these values is a big task. But the longer it's postponed, the greater harm to interoperability. It should be based on a solid plan:

• analyze all vocabularies
• find out and write down definitions
• decide when NOT to use Concepts (see the numeric example above)
• develop some data cleaning rules
• apply the rules to migrate existing RDF data
• apply the rules when creating new RDF data

[gatemezing]

This is actually a work in progress and some improvements will be visible in v3.1.0. In real-world scenario, and sometimes in big ORG, there is trade-off to meet between publishing data vs taking time to clean data before publishing. Data quality is a work in progress once we have a good governance in place. Thank you for the feedback.

[VladimirAlexiev]

@gatemezing great! Is there a scoping doc or spec that you can share?

[gatemezing]

No spec for now. There is a process to establish because we are talking about legacy data, with a field "other value" in the registry where the applicants could put whatever they wanted. There will be at least a first clean up in the official release of v3.1.0