Feature: DB-agnostic data migrations

[ShishKabab]

Motivation: This is an annoying topic that we keep solving for every framework out there, while the basic operations can be quite simple and should be shareable across different frameworks, languages and database technologies out there.

Design considerations:

• The easy stuff should be easy and automatic: adding columns with default values, removing columns, etc.
• Since migrations need to happen at different points in the infrastructure, the migration should be declarative rather than using some API, so it can be either interpreted by an NPM lib, or compiled to things like SQL scripts. Also, these declared operations could be easy to interpret in different existing frameworks if you later decide to switch library or language.
• Declaritive stuff is nice, but give the option to fall back to JS code just in case
• Should provide both forward and backward migrations
• Should be made with automatic testing in mind
• Should be able to be applied to test data fixtures for automatic tests, so overhead maintaining test data can be reduced significantly
• Default CLI tools around this should be easy to use and give useful, human-readable warnings about potentially dangerous operations

The idea: Provide packages for migration through an API and CLI interfaces. Example:

import MigrationManager from `storex-migrations`

...
await storageManager.finishInitliazation()

const migrations = new MigrationManager(storageManager, [
    // Only the more complex migration logic happens here, the rest is generated automatically
    {
        version: new Date(...),
        forwardSteps: [
            {
                collection: 'users',
                operation: 'mergeFields',
                sourceFields: ['firstName', 'lastName'],
                targetField: 'displayName',
                value: {'${object.firstName} ${object.lastName}'}
            },
            ...
        ]
    }
])
await migrations.migrate() // Calls method on storageManager with a set of instructions to migrate data

Steps:

• [x] I've written a library similar to JSON logic, but cleaner, more extensible, more debugabble and it'll have some other cool stuff. Ask me to publish that as an NPM package so we can use that for declared logic operations for fields like value above. done
• [ ] Identify the most common operations needed during migration, and add them to this issue.
• [x] Write logic to calculate the difference between collection version and compile them into a list of operations to be consumed by the back-end (addColumn, addTable, etc.)
• [x] Integrate these above custom migration steps into that list of instructions
• [x] Implement the migration logic into different back-ends

Even if you only do one of these things, it'd already be big help :)

[ShishKabab]

Just published the first version of user-logic for data-driven logic that can be used to define simple data model manipulations during migrations: https://www.npmjs.com/package/user-logic

[ShishKabab]

Work started here: https://github.com/WorldBrain/storex-schema-migrations

[ShishKabab]

Basic implementation done, further development and issues on missing features should be done in the repo mentioned above.