Snapshot diff always returns no changes to apply

[zguig52]

Hello,

I don't know if I am just too bad to use the tool but I cannot manage to synchronize Directus custom collections and so on that are named in directus-sync snapshot and are exported in snapshot folder.

Environment: Directus: 10.9.3 directus-extension-sync (on Directus Servers): 0.4.0 directus-sync (on client CLI): 1.0.0 ? npx directus-sync -V => unknown

What I do:

1. npx directus-sync pull -u https://sourceDirectusURL -e admin@email -p PASSWORD
2. npx directus-sync diff -u https://targetDirectusURL -e admin@email -p PASSWORD
3. npx directus-sync push -u https://targetDirectusURL -e admin@email -p PASSWORD

I always got for diff and push commands: INFO (110897): [snapshot] No changes to apply

For the collections which are exported in collections folder everything is working fine.

I even tried with the --no-split option in case there was a parser bug, but I always got a "no changes to apply" message and my custom collection are not created / updated on the target Directus instance.

Is it a problem with how I use it? Am I not using the right versions?

[EdouardDem]

Hi @zguig52,

A snapshot captures the schema structure of collections, including both Directus default collections and custom collections. It does not include the data within these collections. This aspect is covered by this feature in Directus.

What you find in the collections folder is actually the data dump from the Directus collections (such as flows, permissions, presets, etc.). As of now, there is no functionality to export the data from custom collections, but I am actively working on developing this feature (#1).

When you executed the diff command, were you expecting to see changes in the schema or the data?

By the way, running npx directus-sync -V should return 1.0.0. I will fix this issue.

[zguig52]

Hi @EdouardDem ,

Thanks for your feedback. I had understood that snapshot is only referring to custom collections structure/schema (fields, relations, etc). I was not expecting any data to be imported.

The thing is that the source directus instance was populated with many custom collections (for example articles collection - this is my dev environment), while the target instance was a fresh empty Directus installation (a qualification environment - just bootstrapped with initial admin user).

So I was expecting that push command will create all my custom collections STRUCTURE only. But I was just having always "no changes to apply" and no changes were done inside Directus.

What is strange is the the pull create as expected all the snapshot data inside my export folder (collections, fields, relations).

After inspecting a bit the code, I might suspect that there might be something buggy with Directus integrated diff detection endpoint.

Are you able to bootstrap structure of a fresh install with latest directus-sync and latest Directus?

[EdouardDem]

@zguig52 This is strange. I just deployed a Directus 10.9.3 yesterday, using directus-sync 1.0.0, and it worked as expected. All the schema was pushed.

After running the pull command, do you see all your custom collections in the directus-config/snapshot/collections folder ? It should look like this :

How many custom collections do you have ? Maybe there is a limitation with the number of collections.

Did you try the force options ? npx directus-sync push -f -u https://targetdirectusurl/ -e admin@email -p PASSWORD. This flag is passed to the /schema/diff and /schema/apply endpoints : https://docs.directus.io/reference/system/schema.html#query-parameters-1

[zguig52]

I confirm you that I see all my custom collections, fields and relations in the snapshot directory (and content looks good). I have for the moment 52 collections. Just tried with the force option, same result.

I inspected the logs of the target directus instance and I did not see the shema diff call done (it is running in debug mode). Here are directus logs after the call: api-tst-0 | [08:50:41] POST /auth/login 200 503ms api-tst-0 | [08:50:44] GET /directus-extension-sync/table/settings/sync_id/45c72aa7-3150-43b8-9a0b-27e2d99a6b51 404 4ms api-tst-0 | [08:50:44] GET /directus-extension-sync/table/settings 200 4ms api-tst-0 | [08:50:44] GET /directus-extension-sync/table/settings 200 4ms

I would have expected a POST /schema/diff just after the login.

So I have dig a bit more also on the proxy logs and found out the issue: proxy | {"level":"error","ts":1709456297.4242256,"logger":"http.handlers.waf","msg":"[client \"192.168.122.1\"] Coraza: Access denied (phase 2). SQL Injection Attack: SQL function name detected [file \"/opt/coraza/owasp-crs/REQUEST-942-APPLICATION-ATTACK-SQLI.conf\"] [line \"8751\"] [id \"942151\"] [rev \"\"] [msg \"SQL Injection Attack: SQL function name detected\"] [data \"Matched Data: current_timestamp( found within ARGS:json.fields.27.schema.default_value: current_timestamp()\"] [severity \"critical\"] [ver \"OWASP_CRS/4.0.0-rc2\"] [maturity \"0\"] [accuracy \"0\"] [tag \"application-multi\"] [tag \"language-multi\"] [tag \"platform-multi\"] [tag \"attack-sqli\"] [tag \"OWASP_CRS\"] [tag \"capec/1000/152/248/66\"] [tag \"PCI/6.5.2\"] [tag \"paranoia-level/1\"] [hostname \"\"] [uri \"/schema/diff?force=true\"] [unique_id \"hKrmQpCKpBHrzZgr\"]\n"} proxy | {"level":"error","ts":1709456298.351868,"logger":"http.handlers.waf","msg":"[client \"192.168.122.1\"] Coraza: Access denied (phase 2). Inbound Anomaly Score Exceeded (Total Score: 5) [file \"/opt/coraza/owasp-crs/REQUEST-949-BLOCKING-EVALUATION.conf\"] [line \"11377\"] [id \"949110\"] [rev \"\"] [msg \"Inbound Anomaly Score Exceeded (Total Score: 5)\"] [data \"\"] [severity \"emergency\"] [ver \"OWASP_CRS/4.0.0-rc2\"] [maturity \"0\"] [accuracy \"0\"] [tag \"anomaly-evaluation\"] [hostname \"\"] [uri \"/schema/diff?force=true\"] [unique_id \"hKrmQpCKpBHrzZgr\"]\n"}

The POST /schema/diff is blocked with a 403 reply. I did an exception to the WAF rules to let this pass and it works perfectly now. So this behavior had nothing to do with directus-sync.

The only point is that maybe you could improve the usage of directus sync cli with detecting the call error and displaying it?

I would have think to something like that (packages/cli/src/lib/services/snapshot/snapshot-client.ts): NOW

  /**
   * Diff the snapshot from the dump file.
   */
  async diff() {
    const diff = await this.diffSnapshot();
    if (!diff?.diff) {
      this.logger.info('No changes to apply');
    } 

SUGGESTION:

  /**
   * Diff the snapshot from the dump file.
   */
  async diff() {
    const diff = await this.diffSnapshot();
    if (!diff) {
      this.logger.error('Failed to call Directus diff endpoin');
    } else{
       if (!diff.diff) {
        this.logger.info('No changes to apply');
...

Thanks a lot for taking the time to investigate with me!

[EdouardDem]

@zguig52, thank you for the suggestion. I'll include this logging in the next version. Did the WAF block the request because the POST payload was too large? I will add a troubleshooting section to the README.

[zguig52]

He was detecting an SQL Injection Attack with an SQL function name found inside the POST body.

He didn't like the json.fields.27.schema.default_value: current_timestamp().

I had already found many false positive rules to be allowed with Directus calls. Here is the list of the rules that I found for the moment that get fired during normal operations (rules are part of OWASP_CRS/4.0.0-rc2):

      SecRuleRemoveById 930120
      SecRuleRemoveById 933150
      SecRuleRemoveById 933180
      SecRuleRemoveById 934100
{% if mode == "dev" %}
      # Allow localhost in body
      SecRuleRemoveById 934110
{% endif %}
      SecRuleRemoveById 941100
      SecRuleRemoveById 941160
      SecRuleRemoveById 942100
      SecRuleRemoveById 942151
      SecRuleRemoveById 942290

[EdouardDem]

Thanks for the details