[spike: 5 days] Evaluate LinkML for schema job suitability [revisit 11/1/2022]

[mogul]

Purpose

We want to know if LinkML is going to solve problems with managing schemas and using them in different contexts, but we're not sure if it's going to help.

Given above uncertainty, conducting prototyping is needed to provide factual knowledge to help us make a decision.

5 days of effort has been allocated and once compete, findings will be demonstrated and specific future actions will be decided.

Acceptance Criteria

[ACs should be clearly demo-able/verifiable whenever possible. Try specifying them using BDD.]

• [ ] WHEN 5 days expires \ THEN we are able to demonstrate what we've learned, and have a good idea whether LinkML will help with these needs:
  • single-source a schema for forms, Excel, and validation
  • maintain multiple schema variants without the risk of drift
  • in-progress validation rules differ from final submission validation rules and we don't want to maintain separate full schemas
  • generate a template Excel file from a schema
  • single-source validation for:
  • internal Excel file validation
  • front-end JSON validation (after converting Excel to JSON via excel-js)
  • back-end JSON validation

Background

Refer to #328. This is a good candidate story for lots of pairing/mobbing!

Sketch

• [ ] write LinkML for the general information section
• [ ] test generating JSON Schema from the LinkML
• [ ] write LinkML for in-progress vs final validation rule variants (for the general information section)
• [ ] generate an Excel file based on the schema
• [ ] identify a path for generating validation rules inside the Excel file (with or without LinkML)
• [ ] fill in the generated Excel file with stored JSON data
• [ ] ingest an Excel file and validate the data in it
• [ ] evaluate suitability of the Python library API

[mogul]

The LinkML folks have opened an issue about emitting conditional rules in the JSONSchema output. We'll hold off on LinkML for now and re-evaluate based on whether that has moved in a month.

[mogul]

Closing as the upstream team won't get to this soon and we have some XBRL stuff to look at.