tractr / directus-sync

A CLI tool for synchronizing the schema and configuration of Directus across various environments.
GNU General Public License v3.0
209 stars 8 forks source link
devops directus directus-extension environments

Directus Sync

Directus 11.1.0

[!IMPORTANT] Latest version of directus-sync introduces breaking changes and is not compatible with Directus 10.x.x. If you are using Directus 10.x.x, please run npx directus-sync@2.2.0

[!NOTE] Help us improve Directus Sync by sharing your feedback! Take a quick survey about your usage here: https://forms.gle/LnaB89uVkZCDqRfGA

The directus-sync command-line interface (CLI) provides a set of tools for managing and synchronizing the schema and collections within Directus across different environments.

By leveraging Directus's REST API, it aligns closely with the native actions performed within the application, ensuring a high fidelity of operation.

Updates are granular, focusing on differential data changes rather than blunt table overwrites, which means only the necessary changes are applied, preserving the integrity and history of your data.

Moreover, directus-sync organizes backups into multiple files, significantly improving readability and making it easier to track and review changes. This thoughtful separation facilitates a smoother version control process, allowing for targeted updates and clearer oversight of your Directus configurations.

Table of Contents

Requirements

Usage

The CLI is available using the npx command.

npx directus-sync <command> [options]

Here's how to use each command in the CLI:

Commands

Pull

npx directus-sync pull

Retrieves the current schema and collections from Directus and stores them locally. This command does not modify the database.

It also retrieves the specifications (GraphQL & OpenAPI) and stores them locally. It gets specifications from the /server/specs/* endpoints:

Diff

npx directus-sync diff

Analyzes and describes the difference (diff) between your local schema and collections and the state of the Directus instance. This command is non-destructive and does not apply any changes to the database.

Push

npx directus-sync push

Applies the changes from your local environment to the Directus instance. This command pushes your local schema and collection configurations to Directus, updating the instance to reflect your local state.

Available options

Options are merged from the following sources, in order of precedence:

  1. CLI arguments
  2. Environment variables
  3. Configuration file
  4. Default values

CLI and environment variables

These options can be used with any command to configure the operation of directus-sync:

Configuration file

The directus-sync CLI also supports a configuration file. This file is optional. If it is not provided, the CLI will use the default values for the options.

The default paths for the configuration file are ./directus-sync.config.js, ./directus-sync.config.cjs or ./directus-sync.config.json. You can change this path using the --config-path option.

The configuration file can extend another configuration file using the extends property.

This is an example of a configuration file:

// ./directus-sync.config.js
module.exports = {
  extends: ['./directus-sync.config.base.js'],
  debug: true,
  directusUrl: 'https://directus.example.com',
  directusToken: 'my-directus-token',
  directusEmail: 'admin@example.com', // ignored if directusToken is provided
  directusPassword: 'my-directus-password', // ignored if directusToken is provided
  directusConfig: {
    clientOptions: {},  // see https://docs.directus.io/guides/sdk/getting-started.html#polyfilling
    restConfig: {}, // see https://docs.directus.io/packages/@directus/sdk/rest/interfaces/RestConfig.html
  },
  dumpPath: './directus-config',
  collectionsPath: 'collections',
  onlyCollections: ['roles', 'policies', 'permissions', 'settings'],
  excludeCollections: ['settings'],
  preserveIds: ['roles', 'panels'], // can be '*' or 'all' to preserve all ids, or an array of collections
  snapshotPath: 'snapshot',
  snapshot: true,
  split: true,
  specsPath: 'specs',
  specs: true,
};

Collections hooks

In addition to the CLI commands, directus-sync also supports hooks. Hooks are JavaScript functions that are executed at specific points during the synchronization process. They can be used to transform the data coming from Directus or going to Directus.

Hooks are defined in the configuration file using the hooks property. Under this property, you can define the collection name and the hook function to be executed. Available collection names are: dashboards, flows, folders, operations, panels, permissions, policies, presets, roles, settings and translations.

For each collection, available hook functions are: onQuery, onLoad, onSave, and onDump. These can be asynchronous functions.

During the pull command:

During the push command:

Simple example

Here is an example of a configuration file with hooks:

// ./directus-sync.config.js
module.exports = {
  hooks: {
    flows: {
      onDump: (flows) => {
        return flows.map((flow) => {
          flow.name = `🧊 ${flow.name}`;
          return flow;
        });
      },
      onSave: (flows) => {
        return flows.map((flow) => {
          flow.name = `🔥 ${flow.name}`;
          return flow;
        });
      },
      onLoad: (flows) => {
        return flows.map((flow) => {
          flow.name = flow.name.replace('🔥 ', '');
          return flow;
        });
      },
    },
  },
};

[!WARNING]
The dump hook is called after the mapping of the SyncIDs. This means that the data received by the hook is already tracked. If you filter out some elements, they will be deleted during the push command.

Filtering out elements

You can use onQuery hook to filter out elements. This hook is executed just before the query is sent to Directus, during the pull command.

In the example below, the flows and operations whose name starts with Test: are filtered out and will not be tracked.

// ./directus-sync.config.js
const testPrefix = 'Test:';

module.exports = {
  hooks: {
    flows: {
      onQuery: (query, client) => {
        query.filter = {
          ...query.filter,
          name: { _nstarts_with: testPrefix },
        };
        return query;
      },
    },
    operations: {
      onQuery: (query, client) => {
        query.filter = {
          ...query.filter,
          flow: { name: { _nstarts_with: testPrefix } },
        };
        return query;
      },
    },
  },
};

[!WARNING] Directus-Sync may alter the query after this hook. For example, for roles, the query excludes the current admin role.

Using the Directus client

The example below shows how to disable the flows whose name starts with Test: and add the flow name to the operation.

const { readFlow } = require('@directus/sdk');

const testPrefix = 'Test:';

module.exports = {
  hooks: {
    flows: {
      onDump: (flows) => {
        return flows.map((flow) => {
          flow.status = flow.name.startsWith(testPrefix)
            ? 'inactive'
            : 'active';
        });
      },
    },
    operations: {
      onDump: async (operations, client) => {
        for (const operation of operations) {
          const flow = await client.request(readFlow(operation.flow));
          if (flow) {
            operation.name = `${flow.name}: ${operation.name}`;
          }
        }
        return operations;
      },
    },
  },
};

Snapshot hooks

Like the collections hooks, the snapshot hooks are defined in the configuration file using the hooks.snapshot property. Under this property, you can define the hook functions to be executed.

Available hook functions are: onLoad, onSave:

[!NOTE] This function can be asynchronous. It receives the snapshot object and the Directus client as parameters and must return the snapshot object.

Here is an example of a configuration file that exclude some fields when loading the snapshot. This will be similar for the onSave hook.

// ./directus-sync.config.js
module.exports = {
  hooks: {
    snapshot: {
      /**
       * @param {Snapshot} snapshot
       * @param {DirectusClient} client
       */
      onLoad: async (snapshot, client) => {
        // Remove some fields from the snapshot
        const fieldsToExclude = {
          my_model: ['date_created', 'user_created'],
        };
        const collections = Object.keys(fieldsToExclude);
        const nodeFilter = (node) => {
          const { collection } = node;
          return !(collections.includes(collection) && fieldsToExclude[collection].includes(node.field));
        }
        snapshot.fields = snapshot.fields.filter(nodeFilter);
        snapshot.relations = snapshot.relations.filter(nodeFilter);

        return snapshot;
      },
    },
  },
};

[!NOTE] For more information about the snapshot object, see the Snapshot interface.

Helpers

Untrack

npx directus-sync helpers untrack --collection <collection> --id <id>

Removes tracking from an element within Directus. You must specify the collection and the ID of the element you wish to stop tracking.

Remove permission duplicates

Permissions should be unique regarding the role, collection, and action. Unfortunately, Directus does not enforce this uniqueness. This can lead to unexpected behavior, such as missing ids or other permissions fields. More details can be found in the Directus issue #21965.

If you have permission duplicates, you can use the following command to remove them.

npx directus-sync helpers remove-permission-duplicates --keep <keep>

This command will keep the last (or first) permission found and remove the others for each duplicated group role, collection, and action.

Lifecycle & hooks

Pull command

flowchart
subgraph Pull[Get elements - for each collection]
direction TB
B[Create query for all elements]
-->|onQuery hook|C[Add collection-specific filters]
--> D[Get elements from Directus]
--> E[Get or create SyncId for each element. Start tracking]
--> F[Remove original Id of each element]
-->|onDump hook|G[Keep elements in memory]
end
subgraph Post[Link elements - for each collection]
direction TB
H[Get all elements from memory]
--> I[Replace relations ids by SyncIds]
--> J[Remove ignore fields]
--> K[Sort elements]
-->|onSave hook|L[Save to JSON file]
end
A[Pull command] --> Pull --> Post --> Z[End]

Diff command

Coming soon

Push command

Coming soon

Tracked Elements

directus-sync tracks the following Directus collections:

For these collections, data changes are committed to the code, allowing for replication on other Directus instances. A mapping table links Directus instance IDs with SyncIDs, managed by the directus-extension-sync.

Roles & Policies

Roles and policies are tracked.
By default, Directus creates a default administrator role and two default policies: admin and public.

  1. To avoid recreating the default admin role, directus-sync uses the user's role as the default admin role.
  2. To avoid recreating the default public policy, directus-sync uses the first policy found with role = null.
  3. To avoid recreating the default admin policy, directus-sync uses the first policy linked to the admin role with admin_access = true.

Presets

Global and role based presets are tracked (even the administrator role based presets). However, the users' presets are not tracked. This is because the users are not tracked and any relation with the users will cause conflicts.

Dependency: directus-extension-sync

To utilize the directus-sync tool, it is imperative to install the directus-extension-sync extension on your Directus instance. This extension acts as a bridge between directus-sync and Directus, managing the crucial mapping table that correlates the SyncIDs with Directus's internal IDs.

Installation

The directus-extension-sync must be added to each Directus instance involved in the synchronization process, whether as a source or a destination. Follow the installation instructions provided in the directus-extension-sync repository to add this extension to your Directus setup.

How It Works

directus-sync operates on a tagging system similar to Terraform, where each trackable element within Directus is assigned a unique synchronization identifier (SyncID). This system is key to enabling version control for the configurations and schema within Directus. Here is a step-by-step explanation of how directus-sync functions:

Tagging and Tracking

Upon execution of the pull command, directus-sync will:

  1. Scan the specified Directus collections, which include dashboards, flows, folders, operations, panels, permissions, policies, presets, roles, settings and translations.
  2. Assign a SyncID to each element within these collections if it doesn't already have one.
  3. Commit the data of these collections into code, allowing for versioning and tracking of configuration changes.

This SyncID tagging facilitates the replication of configurations across different instances of Directus while maintaining the integrity and links between different entities.

[!NOTE] The original IDs of the flows are preserved to maintain the URLs of the webhook type flows. The original IDs of the folders are preserved to maintain the associations with fields of the file and image types.

[!TIP] You can use the --preserve-ids option to preserve the original ids of some collections. Eligible collections are collections using UUID: dashboards, operations, panels, roles and translations. If you have already used the pull command, you may use the untrack helper to remove the id tracking of an element before using this option.

Mapping Table

Since it's not possible to add tags directly to entities within Directus, directus-sync uses a mapping table that correlates the SyncIDs with the internal IDs used by Directus. This mapping is essential for the synchronization process, as it ensures that each element can be accurately identified and updated across different environments.

Synchronization Process

The synchronization process is split into two main commands:

Schema Management

The Directus schema, which defines the data modeling and user interface, is managed by the Directus API. However, for better code repository management, directus-sync stores the schema elements in separate files organized within a clear directory structure. This separation allows developers to easily track changes to the schema and apply version control principles to the database structure.

Non-Tracked Elements and Ignored Fields

Elements that are not meant to be tracked, such as user activities and logs, are not affected by the synchronization process. Certain fields are specifically ignored during synchronization because they are not relevant for version control purposes, such as creation dates and the identity of the user who created an entity.

Strengths of directus-sync

The strength of directus-sync lies in its ability to maintain consistent and reproducible configurations across multiple environments. It ensures that only the necessary changes are made, avoiding unnecessary recreation of configurations and maintaining the relationships between tracked and non-tracked entities. This selective updating is what makes directus-sync a robust tool for managing Directus instances in a team or multi-environment setup.

By following these mechanisms, directus-sync streamlines the development workflow, allowing for local changes to be efficiently deployed to various environments, all while keeping the Directus instances synchronized and version-controlled.

Directus upgrades

When upgrading Directus, it is important to update the configurations pulled by directus-sync. Here is a general guide to upgrading Directus using directus-sync:

  1. Run Directus on Version A: Start with your current Directus setup that is actively running on version A.
  2. Push Configuration (if needed): Use npx directus-sync push to push the latest configurations to your Directus instance. This step ensures that all your current settings are up-to-date on the server.
  3. Stop the Directus Server: Shut down your Directus server to prepare for the upgrade.
  4. Update to Version B: Change the Directus version to B in your configuration files or update scripts ( e.g., package.json, docker-compose.yml or Dockerfile).
  5. Restart and Migrate: Start the Directus server to initiate the upgrade. Directus will automatically run migration scripts necessary to update the database and apply new system configurations.
  6. Pull New Configuration: Once Directus is stable on version B, execute npx directus-sync pull to download the latest configuration snapshot. This action captures any changes or migrations that occurred during the upgrade from version A to B.

Use Cases

Troubleshooting

Donate

If Directus Sync has helped you in your projects, please consider donating to support its ongoing development and maintenance.

Donate