[ALLI-7731] Refactor Skosmos enrichment.

NatLibFi / RecordManager

A metadata record management system written in PHP, intended to be used in conjunction with VuFind or another Solr-based discovery interface. Provides import, export, harvesting (OAI-PMH), normalization, deduplication and Solr index update functionality with support for multiple metadata formats. Also includes an OAI-PMH provider that can be used to access the data stored in RecordManager database. Functionality driven by simple command line programs for easy automation.

GNU General Public License v2.0

47 stars 31 forks source link

[ALLI-7731] Refactor Skosmos enrichment. #121

Closed EreMaijala closed 1 year ago

EreMaijala commented 1 year ago

Makes it possible to filter terms by language.
Uses JSON-LD records for locally stored entities as well as remotely fetched ones.
Uses a proper JSON-LD parser.
Simplifies the local repository import process.
Stores local records as PHP serialized objects for quick retrieval (it's slow to create an object from plain JSON-LD since it requires a lot of validation)
Unifies topic and geographic id handling across record classes and makes the SkosmosEnrichment base class work as a format-agnostic enrichment.
Retains backward-compatibility with OnkiLightEnrichment settings.

EreMaijala commented 1 year ago

Note that you can run enrichment without locally stored vocabularies, but it will be slow. To download and import vocabularies, do e.g.:

utils/update_skosmos_data.sh https://api.finto.fi/rest/v1/allars/data
utils/update_skosmos_data.sh https://api.finto.fi/rest/v1/kauno/data
utils/update_skosmos_data.sh https://api.finto.fi/rest/v1/koko/data
utils/update_skosmos_data.sh https://api.finto.fi/rest/v1/musa/data
utils/update_skosmos_data.sh https://api.finto.fi/rest/v1/ysa/data
utils/update_skosmos_data.sh https://api.finto.fi/rest/v1/yso/data