Research and coordinate document schema definition

[valiafetisov]

Goal

Agree on the document schema definition language

Context

Since our API need to be build on top of the document model structure developed in another repository, we need to agree on what information is enough to be exported from that package in order to make integration smooth and maintainable in the future. For this, I would like to do a small research on what approaches are available, then discuss the outcomes.

Tasks

• [x] Outline overall integration requirements (ie what needs to be shared)
• [x] Research schema definitions
  • [x] Come up with a list of qualities to judge each contender
  • [x] Propose possible contenders
• [x] Ask for feedback
• [ ] Discuss externally
• [ ] Outline conclusion

[valiafetisov]

Overall integration requirements

What we need

• Schema of the document that can be converted into the database schema
  • Array of root document types for each module (e.g. budget-statement)
  • Specified type of the relations (whenever there is a ref)
  • Initial/default values + required fields (if there are any, or will it always be covered by INIT operation?)
• List of operations
  • Operation name
  • Operation description?
  • List of arguments (assuming input validators are embedded in the business logic)
    • Argument type
    • Argument description?
    • List of roles which can execute it
• List of roles
  • Role name/id
  • Role description?

What we will provide

• Endpoint/mutations to login and fetch authenticated user object (or just the list of attached roles and a token)
• Endpoint/query to list supported document types (e.g. budget-statement) and operations (e.g. INIT) – basically the same schema returned back, so that frontend/integration can also know what to expect
• Endpoint/query to fetch a document (current state + previously applied operations including meta information)
• Endpoint/mutations to submit operations (ie execute operations as graphql mutations)

[valiafetisov]

List of qualities to judge each contender

• Type safe
  • It should not be possible to define schema that doesn't fit models, operations
  • Schema validation at build time
• Extensible
  • Ideally, it should be possible to declare not only basic types, but also other parameters related to fields, eg initial/default values, validators (eg, to validate ethereum address or email), meta information (field title, field description, etc), relations, etc
  • Easy way to deal with exceptions, extensibility that doesn't blow up the complexity
• Reusable for database
  • In order to avoid redefining the structure of the data multiply times, the schema should hold information on relations (and their qualities). Otherwise, this will require double work on defining the schema and x4 work on migrating data (since the migrations then should be separately written for data and for the database)
• Support schema migrations
  • When a schema structure is updated, the database structure also needs to be updated, i.e. by generating a migration or writing it by hand. In case this is not included into the document model, it will need to be done manually at least two times
• Simplicity of the schema definition
  • So the analyst doesn't have to learn a whole new language
• Editor support for the schema language

[valiafetisov]

Possible contenders

• Typescript types (via something like typescript-json-schema)
  • + Type safe
  • +/- Extensible
    • Typescript types are not available at runtime. However it's possible to compile them to json-schema via external libraries. The library also supports using jsdoc-style comments to extend types with extra data, altrough this might be too hacky
  • - Reusable for database
    • Unless we try to squeeze database types into jsdoc
  • - Support schema migrations
• JSON schema (via something like ajv)
  • +/- Type safe
    • - Limited support for type safety of the schema itself if declared within typescript
      • Limited due to no type support for metadata declaration in tooling
    • + Possible to create validators / type guards
  • + Extensible
    • Already supports description of fields
    • Possible to extend schema with metadata/user-defined keys
  • - Reusable for database
  • - Support schema migrations
• GraphQL schema language (using codegen)
  • - Type safe
    • Or at least until the schema is the ultimate source of truth and types are generated from it
    • The extensions are not type safe (eg @constraint mentioned below can not be safely validated while editing or at build time)
  • +/- Extensible
    • It's technically possible to extend schema statically via something like a custom directive
    • Declarative-only, example @constraint directive
      • This modifies GraphQL schema, basic scalars (Int, Float, String) are replaced by custom scalars
  • +/- Reusable for database
    • As mentioned above, it's technically possible to extend schema with a custom directive that will be interpreted by a custom codegen
  • - Support schema migrations
  • Overall
    • It does seems strange to define schema in another language to then generate something usable from it to then later on generate actual schema for typescript
• Prisma ORM schema language
  • Doesn't fit due to other unfulfilled pre-requirements
    • No native support for polymorphism/union types
    • No support for splitting Prisma schema into multiple files
• TypeORM model definition (together with something like typeorm-polymorphic)
  • +/- Type safe
    • The only missing type safety in typeorm is inability to validate db type over the ts type (due to limited ts support for decorators)
  • + Extensible
    • New decorators as well as arbitary methods can be added to the schema
  • + Reusable for database
    • As it defines database model itself, it can be used both on frontend using in-memory sqlite as well as for the API
  • + Support schema migrations
    • If changes are made to the schema itself, it's easy to automatically generate migrations (on the API side, where we care about consistency) and ignore this part on the frontend

[KirillDogadin-std]

I have a question regarding the set of qualities :

As mentioned in one of the posts you have written in past, when the document schema changes, we would have to also migrate the history of changes (operations) which are stored (i assume) in the database. Was this problem covered by one of the qualities you have listed? If so, it's not really clear how it is handled via one of the listed tools. E.g. if we take typeorm, introduce a document schema migration, how would migration of the (history-)operations be resolved?

[valiafetisov]

we would have to also migrate the history of changes (operations) which are stored (i assume) in the database

No, there is no such requirement, to my knowledge. We would only need to migrate the state, not operations. Each operation will store parameters passed to it as json, unstructured. I general I think it wouldn't make sense to migrate operations, since, for example it's possible that some operation that exist today will not exist in the future – this doesn't mean we need to remove it from the table of executed operations 🙂

[BracketJohn]

Thanks for the investigation - looks like TypeORM is the winner.

The setup would be:

• doc model team writes typeORM models that represents their data
• doc model team uses models as typing-indicators in their DB
• doc model team shares packaged typeORM models with switchboard api team
• switchboard api team uses typeORM models to:
  • provide API
  • setup & run DB
  • implement auth
  • ...

Qs:

• I guess this would force us to use https://github.com/typestack/class-validator if we want to have runtime-data-validation, e.g., on the API level, instead of zod?
• would graphql be generated from typeORM models somehow?
• any idea for a database playground that we can use for development (and later on production)?

[valiafetisov]

Notes:

• There is a generator that can create typeorm schema from graphql https://github.com/jjwtay/graphGenTypeorm (but it's 4 years old and have 7 stars, so most likely wouldn't work)
• Similar package but to generate from JSON schema to typeorm or to graphql https://github.com/kristianmandrup/json-schema-model-builder

[valiafetisov]

Document schema was defined in a separate repo packaged via npm as @acaldas/document-model-libs. This issue is therefore replaced with https://github.com/powerhouse-inc/switchboard-boilerplate/issues/54