Lectern - Data Dictionary Management and Validation

Lectern is Overture's Data Dictionary Schema Manager, providing a system for defining Schemas that will validate the structured data collected by an application. The core of Lectern is a web-server application that handles storage and version management of data dictionaries. Lectern data dictionaries are collections of schemas that define the structure of tabular data files (like TSV). This application provides functionality to validate the structure of data dictionaries, maintain a list of dictionary versions, and to compute the difference between dictionary versions.

Repository Structure

This repository is organized as a monorepo using pnpm-workspace and nx.

Note:

You will need to use pnpm instead of npm to manage dependencies in this code base. PNPM will take care of linking all modules together correctly.

Workspace Modules

The repository is organized with the following directory structure:

.
├── apps/
│   └── server 
└── packages/
    ├── client
    ├── common
    ├── dictionary
    └── validation

The modules in the monorepo are organized into two categories:

• apps/ - Standalone processes meant to be run. These are published to ghcr.io as container images.
• packages/ - Reusable packages shared between applications and other packages. Packages are published to NPM .

Component	Type	Package Name	Path	Published Location	Description
Lectern Server	Application	server	apps/server/		Lectern Server web application.
Lectern Client	Package	@overture-stack/lectern-client	packages/client		TypeScript Client to interact with Lectern Server and perform data validation.
common	Package	common	packages/common/	N/A	Non-specific but commonly reusable utilities. Includes shared Error classes.
dictionary	Package	dictionary	packages/dictionary/	N/A	Dictionary meta-schema definition, includes TS types, and Zod schemas. This also exports all utilities for getting the diff of two dictionaries, and for validating data records with a Dictionary.
validation	Package	@overture-stack/lectern-validation	packages/validation/		Validate data using Lectern Dictionaries.

Developer Instructions

You can install all dependencies for the entire repo from the root (as defined the pnpm-lock.yaml) with the command:

pnpm install

Using nx will ensure all local dependencies are built, in the correct sequence, when building, running, or testing any of the applications and packages in the repo. To run a package.json script from any module - after installing dependencies - use a command of the form pnpm nx <script> <package name>. For example, to build the module server can be done with the command:

pnpm nx build server

This will ensure that all dependencies of server are built in correct order before the server build is run.

Note that the full name of from the package must be used for this to work, so for the client the command would be:

pnpm nx build @overture-stack/lectern-client

For convenience, scripts have been added to the root level package.json to run build and test scripts for every service using short names. These follow the pattern pnpm <build|test>:<package short name/alias>. For example, the same build command can be performed by:

pnpm build:client

To work with any module in this repository, follow the instructions in the README provide in that module's directory.

Get started by running the Lecter Server application.

Common Commands

A few commonly reused scripts have been added to the root package.json. Run them from the root directory, or if you are in a sub directory then use pnpm -w.

For example, when your current working directory is not in the project root, you can still conveniently test every module in the monorepo with the command:

pnpm -w test:all

Build Everything

pnpm build:all

This will build all modules.

Test Everything

pnpm test:all

This will test everything, building all dependencies needed to fully test.

Additonal Content

In addition to the code for Lectern, this repository contains some useful reference material.

Meta-Schema

Lectern provides a meta-schema definition that describes the structure of Lectern Dictionaries. The generated JSON Schema formatted copy of this schema can be found at ./generated/DictionaryMetaSchema.json.

This can be used as a programing language agnostic schema for external applications that want to validate, generate, or interact with Lectern Dictionaries.

Note:

Don't manually update any files in the ./generated path. This content is programatically generated from the source code.

Support & Contributions

We welcome community contributions! Please follow our code of conduct

• Filing an issue
• Connect with us on Slack
• Add or Upvote a feature request