INSPIRE / Add support to load codelists from Registry

[fxprunayre]

• [x] In the thesaurus import console, add an option to load from a registry.

A form is displayed with:

• [x] URL of the registry. By default the INSPIRE registry can be set using a dedicated button. User can choose a custom URL to point to another registry. Once selected, the list of language and item classes are updated.
• [x] User can then select the list of languages available (see http://inspire.ec.europa.eu/registry/help, “The languages currently available are” section). Default language selected is the current UI language. This can not be retrieved from the registry API so the list is hardcoded based on registry documentation page.
• [x] An item class can be selected. Default is the first one loaded. Item class can be "simple" like INSPIRE theme; in this case no subtype can be selected. If not like codelist, the list of codelists is loaded.

• [x] Add the possibility to load the list of codelist and metadata-codelist for the user to choose from a list (instead of pointing to a URL). The codelist is populated based on http://inspire.ec.europa.eu/codelist/codelist.en.json.

When submitting the form, the registry items are processed by the catalogue.

Processing to build a thesaurus from the codelist :

• [x] XML is the format use to import the codelist.
• [x] Download all languages requested
• [x] Combine all languages in a XML document to be XSLT processed
• [x] XSLT / Compose a SKOS thesaurus file to be used in the catalogue (similar to draft 19135 to SKOS conversion https://github.com/SPW-DIG/metawal-core-geonetwork/blob/metawal-3.4.x/web/src/main/webapp/xslt/services/thesaurus/iso19135-to-skos.xsl). Only import concept with status valid (ie. http://inspire.ec.europa.eu/registry/status/valid)
• [x] XSLT / Convert parents relation in Registry format into SKOS topConcept and broader/narrower links

• [x] API Changes:

• POST /api/0.1/registries/vocabularies : Upload a new thesaurus from a file

• PUT /api/0.1/registries/vocabularies : Upload a new thesaurus from URL or Registry

  • registryUrl - String (Mandatory) eg. http://inspire.ec.europa.eu/codelist/BreakLineTypeValue, registryLanguage - String[] (At least one)

• GET /api/0.1/registries/vocabularies/{thesaurusId} : Download a thesaurus

• DELETE /api/0.1/registries/vocabularies/{thesaurusId} : Remove a thesaurus

Dev branch: https://github.com/titellus/core-geonetwork/tree/feature/inspire-registry

Tests made with:

• INSPIRE Registry http://inspire.ec.europa.eu/registry
• French Registry http://registre.geocatalogue.fr/registry

[df-git]

You can find my answers inline:

• Can we retrieve this list of available languages from the registry API? In the current version of the INSPIRE registry service we do not have a format containing all the languages.
• Can we download a codelist in all available languages ? it looks like language parameter is mandatory in all API calls. We may be interested in having an export format with all available language We do not have a full export of a codelist in the current version of the service. This will be available in the next version which is planned for Q4 2018.
• Is ISO19135 a format that will be supported on the long run ? it is now flagged as “beta”. Is it safe to go that way ? I suggest not to use the ISO 19135 for this purpose because it is missing some information (like the parent / child relation) which is available in the Re3gistry XML or in the JSON

[fxprunayre]

Thanks @df-git for the recommendations.

[fxprunayre]

some information (like the parent / child relation) which is available in the Re3gistry XML

@df-git, Could you point us to a codelist using this type of relation in order to add support to it. Thanks.

[df-git]

This is an example: http://inspire.ec.europa.eu/metadata-codelist/PriorityDataset If you open the Re3gistry XML you can see the hierarchy. e.g.

...
<parents>
  <parent id="http://inspire.ec.europa.eu/metadata-codelist/PriorityDataset/dir-2002-49">
    <label xml:lang="en">Directive 2002/49/EC</label>
  </parent>
</parents>
...

[fxprunayre]

Thanks @df-git . Is there any elements which could help identifying top concepts ?

Another question, I suppose we should only concentrate on items with status http://inspire.ec.europa.eu/registry/status/valid ? And ignore others eg. Superseded

[df-git]

The top concepts have no parents. There isn't a specific element to identify them.

Yes, the ones to be considered are the "Valid" elements.

[fxprunayre]

Some new questions @df-git

• Could we retrieve the list of first URL path parameters ? like codelist, metadata-codelist, glossary ...
• When we retrieve the list of codelist, can we request only codelist having valid items (http://inspire.ec.europa.eu/codelist/codelist.en.xml) ?
• Could we limit the response to label and key as we don't really need to download 1.2Mo to display the list ?
• Could we know what is the translation status of a codelist ? When requesting in french, if not translation, english is returned. In our case, the user will ask for french and end with a thesaurus with english term. Could we warn user before building the thesaurus ?

[fxprunayre]

Also an "issue" related to valid item is that each Registry installation has its own concept of valid status eg. http://registre.geocatalogue.fr/codelist/drillingmethod/drillingmethod.en.xml

<status id="http://registre.geocatalogue.fr/registry/status/valid">
<label xml:lang="en">Valid</label>

The valid status id depends on the registry base url.

[df-git]

• The list of Registers is available here: http://inspire.ec.europa.eu/registry. Then, in some cases there is a hierarchical register, like the case of the code lists (http://inspire.ec.europa.eu/metadata-codelist) or the metadata code list (http://inspire.ec.europa.eu/metadata-codelist). To check if a register is a hierarchical register, you need to check the XML of the one of the codes in order to understand if there is the "containeditems" element. That means that you have one other level. We use to consider the elements in the first level of the hierarchical registers like registers (e.g. http://inspire.ec.europa.eu/codelist/AccessRestrictionValue could be considered as a register containing other elements for the SKOS export)
• All the formats returns all the items, not just valid. You can filter them using xpath.
• No, the formats comes with the complete list of fields
• You can use the xml:lang="en" attribute. An example: http://inspire.ec.europa.eu/codelist/ControlledActivityType/ControlledActivityType.fr.xml
• Even if the guideline is to point all the status to http://inspire.ec.europa.eu/registry/status/, the users can create their own status (here we have an example pointing to the "INSPIRE registry status"|: http://registry.inspire.gv.at/codelist).

[fxprunayre]

I noticed an issue related to the codelist date. For example, the INSPIRE themes were published the 2008-06-01 eg. https://github.com/geonetwork/util-gemet/blob/master/thesauri/inspire-theme.rdf#L13

But when creating the thesaurus from the registry, we are currently setting the date to "now".

Using the validator, we end up with

The content "2018-05-25" of element <DateOfPublication> does not match the required simple type. Value "2018-05-25" contravenes the enumeration facet "2008-06-01" of the type of element DateOfPublication at column 66, line 10

@df-git, is there any element indicating date of publication in Registry responses ? eg. http://inspire.ec.europa.eu/theme/theme.en.xml