Migration Orchestrator Service

[stefjoosten]

User story

As a migration engineer, I want to fire up the migration system to start the migration and clean up the old database and the migration system after the migration is completed. This is to prevent pollution in the landscape.

After the migration of system $A$ to $B$, the remainders of $A$ and the migration system have to be removed so that only $B$ remains. This involves deleting the right tables from the database. The Ampersand compiler provides these tables (see issue AmpersandTarski/Ampersand#1404).

Task

Define a cleanup service and make it available to the user under appropriate conditions.

[stefjoosten]

Choices

At the start of the migration, we bring up a migration system and a new system. All three work on the same database. At the end of the migration, we delete the old system and the migration system.

Let us make an orchestrator that does the firing up of the migration system and the cleanup when done. The Migration Orchestrator Service (MOS) is a separate component that lives during the migration to ensure that the migration proceeds as required in the migration specification. This choice allows us to deploy the MOS in a separate container. The MOS will be developed in a separate directory in the Prototype repo, so Git can synchronize differences between the MOS and the prototype framework. The MOS will be deployed in a branch of its own, so its release rhythm is independent of the release rhythm of the prototype framework.

[stefjoosten]

The Migration Orchestrator Service (MOS) shall implement the following migration process.

Process description

1. Let the existing information system $\mathscr{S}$ have data set $\mathscr{D}$ and schema $\mathscr{Z}$. A developer defines a new information system, $\mathscr{S}'$, which contains a data set of its own, $\mathscr{D}'$, a schema $\mathscr{Z}'$, and some functionality. After the migration, the resulting system will be $\mathscr{S}'$, enhanced with the triples from $\mathscr{D}$ that satisfy $\mathscr{Z}'$, and with additional changes made by the migration engineer (see step 3) and by users who have been using the system during the migration. $\mathscr{D}'$ may contain some new data, for instance, to initialize new features at the moment of transition (MoT). If schemas $\mathscr{Z}$ equals $\mathscr{Z}'$, the data migration is unnecessary, so in the sequel we assume they are different.
2. The migration engineer uses a software generator to generate a migration script, $\mathscr{M}$. $\mathscr{M}$ is a system that includes both $\mathscr{S}$ and $\mathscr{S}'$. $\mathscr{M}$ is meant to migrate the data and define the semantics during the migration phase. It contains enforcement rules that migrate existing data to the desired data set. It also contains data structures and rules that are needed for the duration of the migration phase. The generated script must be free of type errors, to ensure that satisfaction is defined for every rule $u$ and data set $\mathscr{D}$. The proof is available here.
3. In many cases, the schema $\mathscr{Z}$_{$\mathscr{M}$} will be enough to perform the migration. However, a migration engineer may have specific requirements that call for changes in $\mathscr{M}$. For that purpose, the generator generates the migration script as an Ampersand script, so the migration engineer can accommodate such requirements. This can be necessary, for example, to resolve data pollution that violates a constraint that was not in schema $\mathscr{Z}$_{$\mathscr{S}$}. She can either implement enforce rules or use the business to resolve the issues. She can test the resulting migration script, $\mathscr{M}'$ separately before taking the desired system to production. The migration engineer alters $\mathscr{Z}$ nor $\mathscr{Z}'$ while adapting $\mathscr{M}'$ to the specific needs of the current migration.
4. Then, the migration engineer deploys system $\mathscr{M}'$. This event starts the copying of data from $\mathscr{D}$ to $\mathscr{D}'$, as specified in $\mathscr{M}'$. During this phase, the ingress is still referring users to the existing system, so they will hardly notice that the desired system is being fired up and loaded with data. Users are still changing $\mathscr{D}$ (both inserts and deletes), but these changes are being transferred to $\mathscr{D}'$ as well. This step is complete when all data from $\mathscr{D}$ has been copied to $\mathscr{D}'$ and all invariants of the migration system are true.
5. At this moment, the ingress switches the incoming traffic from $\mathscr{S}$ to $\mathscr{M}'$, so $\mathscr{M}'$ effectively takes over. This event marks the MoT. Since $\mathscr{S}$ may still have some violations due to last-minute edits, it may take some (short but finite) time until the invariants of $\mathscr{S}$ are true. After that, dataset $\mathscr{D}$ will no longer change.
6. From this moment, users experience the functionality of the migration system $\mathscr{M}'$. This system must ensure that all invariants of $\mathscr{Z}'$ are true before the migration is complete. It no longer ensures the rules of $\mathscr{Z}$, which govern the existing system. Since $\mathscr{D}$ is no longer changing, the migration engineer can now focus on the remaining violations of $\mathscr{Z}'$. Users of the migration system will deal with process rules in the migration system that are still violated. Once all invariants of $\mathscr{M}'$ are true, the migration is complete. This state marks the moment of cleanup (MoC)
7. After the MoC, the ingress switches the incoming traffic from $\mathscr{M}'$ to $\mathscr{S}'$. The migration system is no longer needed and can be taken out of production, together with the existing system $\mathscr{S}$. The desired system remains in production and the migration is finished. This establishes the post-migration situation.