Versioning/Migrating

[kalda341]

Hi,

I'm super interested to hear what you have in mind for versioning and migrations in this package, as I have a fairly complex system in production to manage this that I've built on top of io-ts. I know this is probably something you don't want to put much thought into at this point, but I'd love to get a discussion started well before you decide on any specific implementation.

Briefly, we have a schema version field on a struct which is used to discriminate between versions. We have a set of migrations, which are type checked in order to move from one version to the next (we don't do reverse migrations, though I would like to). After a migration is run from one version to the next we check the type at runtime with io-ts to make sure the result is valid. We can recursively call migrateNext which will take an arbitrary version and migrate it to the latest version.

Below is a bit of a mess (I've taken code from a lot of different places), but may be of interest if you're interested in how this works. I'd love to hear how this compares with what you have in mind. This is designed for a specific schema, though I suspect it could fairly easily be generalised.

const ProjectStateVersions = [
  V1.ProjectStateT,
  V2.ProjectStateT,
  V3.ProjectStateT,
] as const;

export import Current = V3;

type IoTsTypeofArray<Tuple extends readonly IoTs.Mixed[]> = {
  [K in keyof Tuple]: Tuple[K] extends IoTs.Mixed
    ? IoTs.TypeOf<Tuple[K]>
    : never;
};

export type ProjectStateVersions = TypeUtil.Writeable<
  IoTsTypeofArray<typeof ProjectStateVersions>
>;

export type AnyVersion = TypeUtil.ArrayInfer<ProjectStateVersions>;
export type LatestVersion = TypeUtil.Last<ProjectStateVersions>;

export const ProjectState = ProjectStateVersions[
  ProjectStateVersions.length - 1
] as IoTs.Type<LatestVersion>;
export type ProjectState = LatestVersion;

export const currentSchemaVersion: LatestVersion['schemaVersion'] =
  Current.schemaVersion;

// Index manipulation is a pretty nasty way to achieve this, but it works
// without excessive recursion which leads to type errors, even for a large
// number of ProjectStateVersions.
export type NextVersion<T extends AnyVersion> =
  ProjectStateVersions[T['schemaVersion']];
export type FutureVersion<T extends AnyVersion> = TypeUtil.ArrayInfer<
  TypeUtil.After<T, ProjectStateVersions>
>;

export const UnknownProjectState = IoTs.union(
  ProjectStateVersions as TypeUtil.Writeable<typeof ProjectStateVersions>,
  'UnknownProjectState'
);
export type UnknownProjectState = IoTs.TypeOf<typeof UnknownProjectState>;

export interface Migration<
  T extends AnyVersion,
  U extends NextVersion<T>
> {
  readonly from: T['schemaVersion'];
  readonly to: U['schemaVersion'];
  // We don't need dependencies or errors yet, but we'll reserve them for later
  readonly up: (
    projectState: T
  ) => RTE.ReaderTaskEither<MigrationDeps, never, U>;
  readonly codec: IoTs.Type<U, unknown>;
}

export const migrateUp =
  <T extends AnyVersion, U extends NextVersion<T>>(
    migration: Migration<T, U>
  ) =>
  (
    projectState: T
  ): RTE.ReaderTaskEither<MigrationDeps, MigrationError, U> =>
    F.pipe(
      migration.up(projectState),
      RTE.filterOrElseW(
        migration.codec.is,
        F.constant(
          new Types.MigrationError(
            migration.from,
            migration.to,
            `Failed validation after migration from schemaVersion ${migration.from} to schemaVersion ${migration.to}.`
          )
        )
      )
    );

const recursiveMigrate =
  <T extends AnyVersion, U extends NextVersion<T>>(
    migration: Migration<T, U>
  ) =>
  (
    projectState: T
  ): RTE.ReaderTaskEither<
    MigrationDeps,
    MigrationError,
    ProjectStates.ProjectState
  > =>
    F.pipe(projectState, migrateUp(migration), RTE.chain(migrateLatest));

export function migrateLatest(
  projectState: ProjectStates.AnyVersion
): RTE.ReaderTaskEither<
  MigrationDeps,
  MigrationError,
  ProjectState
> {
  switch (projectState.schemaVersion) {
    case 1:
      return recursiveMigrate(nonAddedDocumentMigration)(projectState);
    case 2:
      return recursiveMigrate(siteMigration)(projectState);
    case 3:
      return recursiveMigrate(activityStatusMigration)(projectState);
    case ProjectStates.currentSchemaVersion:
      return RTE.of(projectState);
  }
}

Thank you for all your work, I'm very excited to move to using the fp-ts org packages soon, though it will be a massive job as we are super heavily invested in the previous fp-ts, io-ts and monocle-ts packages!

[gcanti]

Hi @kalda341, sorry for the late answer, I'm still working on the core of /schema, I'll be thinking to this topic as soon as the project will be more fleshed out

[kalda341]

@gcanti no worries, I know it's not the highest priority. I'm watching your progress with the new packages excitedly!