Validation of scientificMetadata

[sbliven]

Summary

Enable validation of scientificMetadata against a schema. The schema should be configurable for each site and default to no validation. In some cases multiple schema might be desirable, with different features available depending on the schema.

Motivation

• Boundary conditions (eg matching a proposal system format, interoperability with OpenEM databases, etc)
• Frontend visualization (tabular or more advanced)
• Semantic tagging (eg comment by @bpedersen2)
• Advanced search (eg ranges, searches with units)

Current Behaviour

Validation must be performed out-of-band by the ingestor tools/libraries

Expected Behaviour

• Configure a json schema for scientificMetadata
• isValid endpoints check this schema
• Creating and updating datasets fails if it doesn't match the schema
• Enable enhanced functionality if scientificMetadata matches a schema (eg matching a schema for quantities enables value+unit parsing by the frontend)

Questions

1. Should users be able to specify (optionally) a more restrictive schema? For instance, a schema for EM data. Should this be another top-level property, or something like scientificMetadata.@context (JSON-LD style).
2. What schemas are implicitly supported by SciCat currently (eg with specific frontend behavior)?

[sbliven]

Before implementing this we would have to agree on the schema technology and the spellings for the terms. I think that the schema language is not a major barrier to adoption, as there are very good tools for converting both schemas and data between different serializations (eg with LinkML).

JSON Schema

One option would be to add a json schema to the dataset model. Most people I've discussed this with seem to expect the schema to apply only to the scientificMetadata, not to the whole json.

{
    "pid": "123...",
    "schema": "https://osc-em.github.com/oscem_schema.json",
    "scientificMetadata": {
        [validate this]
    }
}

If "schema" is omited we would default to no validation for backwards compatibility.

JSON-LD

JSON-LD also has good support and is used in many metadata containers (eg RO-Crate). One nice feature is that the context (aka schema) can be either a shared schema included by URI or can be inlined into the data itself. We could adapt the dataset model so that it's valid JSON-LD by including a @context key linking a published minimal schema for scicat. This could even be overridden within scientificMetadata to specify a more restrictive schema.

LinkML

Would it make sense to validate LinkML directly?

[sbliven]

Notes from SciCatCon day 2 discussion:

• Agreement: scicat should have a field for the schema in datasets (defaulting to very permissive) and should validate this if set
• Agreement: validation should apply to the full dataset, with a base schema specifying scicat fields and derivative schemas adding additional constraints on scientificMetadata (this is supported by LinkML)
• Need to explore schema technologies more before deciding (JSON Schema, JSON-LD, LinkML)
• schemas can be used for validating features needed by search or frontend, for example enforcing (unit, value) tuples

[linupi]

Notes from the discussion on 4/7/2023

Next steps (independent):

• investigate how Swagger/OpenAPI schema can be ingested into LinkML and converted to Json Schema/ Json-LD
• Proposal: add (in first step static) @context to the response of 'Dataset - GET` endpoint (one option: add to DatasetClass so that it is stored in Mongo and part of the response)
• (in first step) add @context as one attribute to ScientificMetadata for now to specify the schema of ScientificMetadata
• try to build valid example Json-LD datasets (just files) e.g. for pizza to agree on what needs to be done SciCat before staring to code
• consider (in second step) MIME-Type dependent responds of GET endpoints

[sbliven]

Releated discussion: https://github.com/SciCatProject/scicat-backend-next/discussions/1308