56 Enhance migration tooling api

[Oyelowo]

Overview

This PR introduces a comprehensive overhaul of the Migration CLI, streamlining its functionality for a more intuitive and efficient user experience. It's a step forward in simplifying and unifying the migration process in SurrealDB environments.

• Streamlined CLI Interface: Redesigned to ensure consistency and ease of use. All database connection flags are now universally applicable at the top level.
• New Commands implemented: Introducing init, reset, list, and prune for expanded control over migration processes.
• Direct Execution Post-Generation: The gen, init, and reset commands now offer optional execution against the current live database instance post-migration generation.
• Simplified Migration Strategies: The gen, up and down commands have been refined for straightforward execution, eliminating the need for the --reversible flag during routine operations.
• Automated Testing Suite: Comprehensive integration and end-to-end tests cover all migration operations, including edge cases.
• Enhanced Renaming and Deletion Operations: Automatic and manual renaming and deletion processes have been optimized for better user control and understanding.

More details

Migration: Fully Automated Database Schema Migrations

Surreal ORM provides a powerful command-line interface (CLI) for automatically diffing and managing database migrations in a SurrealDB environment. This tool offers functionalities ranging from initializing migrations, generating migration files, applying migrations up or down, resetting migrations, listing migrations, pruning pending migration files and various other tasks to manage your database schema effectively.

It supports both CLI and Emdded-migrations. Embedded migration includes the migration files data into your binary at compile time and is accessible to your binary at runtime.

Usage

It involves several processes including gathering codebase resources, then setting those up for initializing and generating new migrations.

Step 1: Setting Up And Gathering Codebase Resources

use surreal_orm::*;

#[derive(Node, Serialize, Deserialize, Debug, Clone, Default)]
#[serde(rename_all = "camelCase")]
#[surreal_orm(table_name = "animal", schemafull)]
pub struct Animal {
    pub id: SurrealSimpleId<Self>,
    pub species: String,
    // #[surreal_orm(old_name = "oldField")] // Comment this line out to carry out a renaming operation
    pub attributes: Vec<String>,
    pub created_at: chrono::DateTime<Utc>,
    pub updated_at: chrono::DateTime<Utc>,
    pub velocity: u64,
}

impl TableResources for Animal {
    fn events_definitions() -> Vec<Raw> {
        let animal::Schema { species, velocity, .. } = Self::schema();

        let event1 = define_event("event1".to_string())
            .on_table("animal".to_string())
            .when(cond(species.eq("Homo Erectus")).and(velocity.gt(545)))
            .then(select(All).from(Crop::table_name()))
            .to_raw();

        vec![event1]
    }

    fn indexes_definitions() -> Vec<Raw> {
        let animal::Schema { species, velocity, .. } = Self::schema();

        let idx1 = define_index("species_speed_idx".to_string())
            .on_table(Self::table_name())
            .fields(arr![species, velocity])
            .unique()
            .to_raw();

        vec![idx1]
    }
}

#[derive(Edge, Serialize, Deserialize, Debug, Clone, Default)]
#[serde(rename_all = "camelCase")]
#[surreal_orm(table_name = "eats", schemafull)]
pub struct Eats<In: Node, Out: Node> {
    pub id: SurrealSimpleId<Self>,
    #[serde(rename = "in")]
    pub in_: In,
    pub out: Out,
    pub place: String,
    pub created_at: chrono::DateTime<Utc>,
}

pub type AnimalEatsCrop = Eats<Animal, Crop>;
impl TableResources for AnimalEatsCrop {}

#[derive(Debug, Clone)]
pub struct Resources;

impl DbResources for Resources {
    create_table_resources!(
        Animal,
        Crop,
        AnimalEatsCrop,
    );

    // Define other database resources here. They default to empty vecs
    fn analyzers(&self) -> Vec<Raw> {
        vec![]
    }

    fn functions(&self) -> Vec<Raw> {
        vec![]
    }

    fn params(&self) -> Vec<Raw> {
        vec![]
    }

    fn scopes(&self) -> Vec<Raw> {
        vec![]
    }

    fn tokens(&self) -> Vec<Raw> {
        vec![]
    }

    fn users(&self) -> Vec<Raw> {
        vec![]
    }
}

use surreal_orm::migrator::Migrator;

#[tokio::main]
async fn main() {
    Migrator::run(Resources).await;
}

Step 2: Running the CLI and/or Embedding Migrations

The CLI tool offers a range of commands, each with specific options and flags. Here's a quick overview:

1. Initialize Migrations:

   cargo run -- init --name "initial migration" -r

   This initializes the migrations directory with a reversible migration named "initial_migration". Omit the -r flag if you want up only migrations.

2. Generate Migrations:

   cargo run -- gen --name "add users table"

   Generates a new migration file named "add_users_table". Notice that we do not need to include the -r or --reversiable flag. Because we specified whether we want a reversible or non-reversible migration when we initialized, the migration type is automatically detected subsequently.

3. Apply Migrations Up:

   # Applies all pending till latest by default
   cargo run -- up
   
   # Applies all pending till latest
   cargo run -- up -l
   
   # Applies by the number specified
   cargo run -- up -n 5
   cargo run -- up --number 5
   
   # Applies till specified migration
   cargo run -- up -t "20240107015727114_create_first.up.surql"
   cargo run -- up --till "20240107015727114_create_first.up.surql"

   Applies pending migrations forward using various strategies: till latest, by number count and till a specified migration.

4. Rollback Migrations:

   # Rollback migration to previous by default
   cargo run -- down
   
   # Rollback all pending till previous
   cargo run -- down --previous
   
   # Rollback by the number specified
   cargo run -- down -n 5
   cargo run -- down --number 5
   
   # Rollback till specified migration
   cargo run -- down -t "20240107015727114_create_first.up.surql"
   cargo run -- down --till "20240107015727114_create_first.up.surql"
   
   # In addition, you can use the --prune flag to delete local migration 
   # files after rolling back. This can be useful in development for rapid changes.
   cargo run -- down -n 5 --prune

   Rolls back the last applied migration.

5. Reset Migrations:

   cargo run -- reset --name "initial_migration" -r

   Resets all migrations and initializes a new reversible migration named "initial_migration". Skip the -r or --reversible flag if you want up only migrations,

6. Prune Migrations:

   # List pending migrations by default
   cargo run -- prune

   Prune all pending unapplied migrations.

7. List Migrations:

   # List pending migrations by default
   cargo run -- ls
   cargo run -- list
   
   # List all migrations
   cargo run -- list --status all
   
   # List pending migrations
   cargo run -- list --status pending
   
   # List applied migrations
   cargo run -- list --status applied

   Lists migrations by their statuses i.e, all, pending and applied.

Advanced Migration CLI Usage

Advanced usage involves specifying additional flags and options to tailor the migration process to your specific needs. Here's how you can use these advanced features:

1. Custom Migration Directory:

   cargo run -- init --name "initial_migration" --dir "custom_migrations" -r

   Initializes migrations in a custom directory named "custom_migrations".

2. Verbose Output:

   cargo run -- up -vvv

   Runs migrations with 3 levels verbose output.

3. Database Connection Configuration:

   • URL: ws://localhost:8000
   • Database Name: test
   • Namespace: test
   • User: root
   • Password: root

   cargo run -- up --url "ws://localhost:8000" --db "mydb" --ns "myns" --user "username" --pass "password"

   Connects to the specified SurrealDB instance with custom credentials and applies migrations.

   Other supported urls types include:

   - ws://localhost:8000
   - wss://cloud.example.com
   - http://localhost:8000
   - https://cloud.example.com
   - mem:// 
   # or simply
   - memory
   - file://temp.db
   - indxdb://MyDatabase
   - tikv://localhost:2379
   - fdb://fdb.cluster

   This configuration enables the CLI to connect to different database backends including WebSocket, HTTP(S), In-Memory, File-Backend, and more.

Embedded Migrations

use surreal_orm::migrator::{
    self, config::DatabaseConnection, EmbeddedMigrationsOneWay, EmbeddedMigrationsTwoWay,
embed_migrations, Mode, UpdateStrategy,
};

// Embed migrations as constant
const MIGRATIONS_ONE_WAY: EmbeddedMigrationsOneWay =
    embed_migrations!("tests/migrations-oneway", one_way, strict);

let db = DatabaseConnection::default().setup().await.db().unwrap();
MIGRATIONS_ONE_WAY
    .run(db.clone(), UpdateStrategy::Latest, Mode::Strict)
    .await
    .unwrap();

const MIGRATIONS_TWO_WAY: EmbeddedMigrationsTwoWay =
    embed_migrations!("tests/migrations-twoway", two_way, strict);

MIGRATIONS_TWO_WAY
    .run(db.clone(), UpdateStrategy::Latest, Mode::Strict)
    .await
    .unwrap();