search

doctrine / migrations

Doctrine Database Migrations Library
https://www.doctrine-project.org/projects/migrations.html
MIT License
4.68k stars 388 forks source link

Error `The metadata storage is not up to date` since 3.5.3 #1302

Closed dmaicher closed 1 year ago

dmaicher commented 1 year ago

Bug Report

Q A
BC Break yes?
Version 3.5.3

Summary

I just upgraded to 3.5.3 and I have an error when running app/console doctrine:migrations:up-to-date:

In MetadataStorageError.php line 13:

  The metadata storage is not up to date, please run the sync-metadata-storage command to fix this issue.

Unfortunately running doctrine:migrations:sync-metadata-storage does not help. It says [OK] Metadata storage synchronized but I get the same error when running up-to-date again.

How to reproduce

This is my metadata table dump

CREATE TABLE `migration_versions` (
  `version` varchar(191) COLLATE utf8_unicode_ci NOT NULL,
  `executed_at` datetime DEFAULT NULL,
  `execution_time` int(11) DEFAULT NULL,
  PRIMARY KEY (`version`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;

Expected behavior

It should not report an issue with out-of-sync storage

ochov1 commented 1 year ago

Here the same. php 7.4 symfony 5.4

ochov1 commented 1 year ago

My solution was to downgrade to 5.3.2

composer require doctrine/migrations:3.5.2

dmaicher commented 1 year ago

This is the table diff that is reported here:

^ Doctrine\DBAL\Schema\TableDiff^ {#2304
  +name: "migration_versions"
  +newName: false
  +addedColumns: []
  +changedColumns: array:1 [
    "version" => Doctrine\DBAL\Schema\ColumnDiff^ {#2313
      +oldColumnName: "version"
      +column: Doctrine\DBAL\Schema\Column^ {#2320
        #_name: "version"
        #_namespace: null
        #_quoted: false
        #_type: Doctrine\DBAL\Types\StringType^ {#2446}
        #_length: 191
        #_precision: 10
        #_scale: 0
        #_unsigned: false
        #_fixed: false
        #_notnull: true
        #_default: null
        #_autoincrement: false
        #_platformOptions: []
        #_columnDefinition: null
        #_comment: null
        #_customSchemaOptions: []
      }
      +changedProperties: []
      +fromColumn: Doctrine\DBAL\Schema\Column^ {#2280
        #_name: "version"
        #_namespace: null
        #_quoted: false
        #_type: Doctrine\DBAL\Types\StringType^ {#2446}
        #_length: 191
        #_precision: 10
        #_scale: 0
        #_unsigned: false
        #_fixed: false
        #_notnull: true
        #_default: null
        #_autoincrement: false
        #_platformOptions: array:2 [
          "charset" => "utf8"
          "collation" => "utf8_unicode_ci"
        ]
        #_columnDefinition: null
        #_comment: null
        #_customSchemaOptions: []
      }
    }
  ]
  +removedColumns: []
  +renamedColumns: []
  +addedIndexes: []
  +changedIndexes: []
  +removedIndexes: []
  +renamedIndexes: []
  +addedForeignKeys: []
  +changedForeignKeys: []
  +removedForeignKeys: []
  +fromTable: Doctrine\DBAL\Schema\Table^ {#2330
    #_name: "migration_versions"
    #_namespace: null
    #_quoted: false
    #_columns: array:3 [
      "version" => Doctrine\DBAL\Schema\Column^ {#2280}
      "executed_at" => Doctrine\DBAL\Schema\Column^ {#2340
        #_name: "executed_at"
        #_namespace: null
        #_quoted: false
        #_type: Doctrine\DBAL\Types\DateTimeType^ {#2481}
        #_length: 0
        #_precision: 10
        #_scale: 0
        #_unsigned: false
        #_fixed: false
        #_notnull: false
        #_default: null
        #_autoincrement: false
        #_platformOptions: []
        #_columnDefinition: null
        #_comment: null
        #_customSchemaOptions: []
      }
      "execution_time" => Doctrine\DBAL\Schema\Column^ {#2342
        #_name: "execution_time"
        #_namespace: null
        #_quoted: false
        #_type: Doctrine\DBAL\Types\IntegerType^ {#2451}
        #_length: null
        #_precision: 10
        #_scale: 0
        #_unsigned: false
        #_fixed: false
        #_notnull: false
        #_default: null
        #_autoincrement: false
        #_platformOptions: []
        #_columnDefinition: null
        #_comment: null
        #_customSchemaOptions: []
      }
    ]
    #_indexes: array:1 [
      "primary" => Doctrine\DBAL\Schema\Index^ {#2316
        #_name: "PRIMARY"
        #_namespace: null
        #_quoted: false
        #_columns: array:1 [
          "version" => Doctrine\DBAL\Schema\Identifier^ {#2311
            #_name: "version"
            #_namespace: null
            #_quoted: false
          }
        ]
        #_isUnique: true
        #_isPrimary: true
        #_flags: []
        -options: array:1 [
          "lengths" => array:1 [
            0 => null
          ]
        ]
      }
    ]
    #_primaryKeyName: "primary"
    #uniqueConstraints: []
    #_fkConstraints: []
    #_options: array:6 [
      "create_options" => []
      "engine" => "InnoDB"
      "collation" => "utf8_unicode_ci"
      "charset" => "utf8"
      "autoincrement" => null
      "comment" => ""
    ]
    #_schemaConfig: null
    -implicitIndexes: []
  }
}
colinpieper commented 1 year ago

Our Symfony 5.4 projects are also impacted by this issue, running MariaDB 10.6.9. Downgrading doctrine/migrations to 3.5.2 fixed the issue.

yani commented 1 year ago

Same happened over here.

I'm using the library natively with MariaDB and changing server_version to mariadb-x.x.x did not fix the issue.

Downgrading to 3.5.2 fixed it.

Kaaly commented 1 year ago

Same issue here. Seem coming from this commit : 315f255aa78c1179197759a6afa9608437ea2ef6

axi commented 1 year ago

The difference is on the "_platformOptions" for me. On database ($fromTable):

[
       "charset" => "utf8mb4"
       "collation" => "utf8mb4_unicode_ci"
]

On $toTable:

[]

I would say introduced in this commit 1f5851853825ec1b71a8b1a86a2747de87c3695b

Reverting method needsUpdate() from TableMetadataStorage fix the problem on my side


    private function needsUpdate(Table $expectedTable): ?TableDiff
    {
        if ($this->schemaUpToDate) {
            return null;
        }
        $currentTable = $this->schemaManager->listTableDetails($this->configuration->getTableName());
        $diff         = $this->schemaManager->createComparator()->diffTable($currentTable, $expectedTable);

        return $diff instanceof TableDiff ? $diff : null;
    }
WoutervanderLoopNL commented 1 year ago

Same here, not getting the charset en collation. so its see a diff every time

greg0ire commented 1 year ago

@axi (or anybody) there are 3 changes in that method, it might be interesting to revert only one of them.

greg0ire commented 1 year ago

Also @axi, Table does not have a _platformOptions property, does it? Only Column does: https://github.com/doctrine/dbal/blob/e106ee7a3869a2ca82b0993914ea55061be02554/src/Schema/Column.php#L45-L46

On which column(s) do you observe the diff?

stof commented 1 year ago

@greg0ire in the dump of the TableDiff in @dmaicher's comment above, the column appearing in changedColumns is the version column.

greg0ire commented 1 year ago

I think something must be calling setPlatformOptions or setPlatformOption on that column with these options at some point, and maybe the same should be done here somehow: https://github.com/doctrine/migrations/blob/01f89a1082b6971ba53ba76f95741fc8ad4a5fd2/lib/Doctrine/Migrations/Metadata/Storage/TableMetadataStorage.php#L241-L256

Can one of you determine if that's the case? I'm suspecting https://github.com/doctrine/dbal/blob/3.5.x/src/Platforms/MySQL/Comparator.php#L62

What's very interesting is the comment on that class: https://github.com/doctrine/dbal/blob/e106ee7a3869a2ca82b0993914ea55061be02554/src/Platforms/MySQL/Comparator.php#L15-L17

In MySQL, unless specified explicitly, the column's character set and collation are inherited from its containing table. So during comparison, an omitted value and the value that matches the default value of table in the desired schema must be considered equal.

If I understand correctly, the diff should be empty if utf8mb4 and utf8mb4_unicode_ci is what is used at the table level.

dmaicher commented 1 year ago

nice catch @greg0ire . As discussed on Slack the issue was the missing normalization of columns: https://github.com/doctrine/dbal/pull/5861

SchwarzDeveloping commented 1 year ago

Same here. Symfony 6.2, PHP 8.1, Mariadb 10.5.15 Downgrade to 3.5.2 fixed it for the moment. Took 10 hours, 5 tantrums and 8 project restarts to find this issue -.-

greg0ire commented 1 year ago

Please stop with the spam.

greg0ire commented 1 year ago

Fixed by https://github.com/doctrine/dbal/releases/tag/3.5.3 (don't get confused, it's another package, it just happens to have the same version number)

samuel-sol commented 1 year ago

@greg0ire I've tried updating again with composer update but the error persist. Forcing a downgrade to 3.5.2 and then reupgrading do 3.5.3 didn't work. removing the package and readding also did not fix.

Here is a var_dump part of the return from the method Doctrine\Migrations\Metadata\Storage\TableMetadataStorage::needsUpdate. You can see that the error is that for some reason he has "default => (string)NULL" instead of NULL.

object(Doctrine\DBAL\Schema\TableDiff)#1005 (14) {
  ["changedColumns"]=>
  array(1) {
    ["executed_at"]=>
    object(Doctrine\DBAL\Schema\ColumnDiff)#1004 (4) {
      ["oldColumnName"]=>
      string(11) "executed_at"
      ["column"]=>
      object(Doctrine\DBAL\Schema\Column)#1001 (16) {
        ["_name":protected]=>
        string(11) "executed_at"
        ["_namespace":protected]=>
        NULL
        ["_quoted":protected]=>
        bool(false)
        ["_type":protected]=>
        object(Doctrine\DBAL\Types\DateTimeType)#946 (0) {
        }
        ["_length":protected]=>
        NULL
        ["_precision":protected]=>
        int(10)
        ["_scale":protected]=>
        int(0)
        ["_unsigned":protected]=>
        bool(false)
        ["_fixed":protected]=>
        bool(false)
        ["_notnull":protected]=>
        bool(false)
        ["_default":protected]=>
        NULL
        ["_autoincrement":protected]=>
        bool(false)
        ["_platformOptions":protected]=>
        array(0) {
        }
        ["_columnDefinition":protected]=>
        NULL
        ["_comment":protected]=>
        NULL
        ["_customSchemaOptions":protected]=>
        array(0) {
        }
      }
      ["changedProperties"]=>
      array(1) {
        [0]=>
        string(7) "default"
      }
      ["fromColumn"]=>
      object(Doctrine\DBAL\Schema\Column)#996 (16) {
        ["_name":protected]=>
        string(11) "executed_at"
        ["_namespace":protected]=>
        NULL
        ["_quoted":protected]=>
        bool(false)
        ["_type":protected]=>
        object(Doctrine\DBAL\Types\DateTimeType)#946 (0) {
        }
        ["_length":protected]=>
        int(0)
        ["_precision":protected]=>
        int(10)
        ["_scale":protected]=>
        int(0)
        ["_unsigned":protected]=>
        bool(false)
        ["_fixed":protected]=>
        bool(false)
        ["_notnull":protected]=>
        bool(false)
        ["_default":protected]=>
        string(4) "NULL"
        ["_autoincrement":protected]=>
        bool(false)
        ["_platformOptions":protected]=>
        array(0) {
        }
        ["_columnDefinition":protected]=>
        NULL
        ["_comment":protected]=>
        NULL
        ["_customSchemaOptions":protected]=>
        array(0) {
        }
      }
    }
  }

and here is my composer.json

{
    "type": "project",
    "license": "proprietary",
    "minimum-stability": "stable",
    "prefer-stable": true,
    "require": {
        "php": ">=8.1.0",
        "ext-ctype": "*",
        "ext-iconv": "*",
        "ext-openssl": "*",
        "beberlei/doctrineextensions": "^1.3",
        "doctrine/annotations": "^1.13.3",
        "doctrine/doctrine-bundle": "^2.0",
        "doctrine/migrations": "^3.0",
        "doctrine/doctrine-migrations-bundle": "^3.0",
        "doctrine/dbal": "^3.0",
        "doctrine/orm": "2.14.0",
        "doctrine/persistence": "^2.5.0",
        "gedmo/doctrine-extensions": "^3.0.0",
        "knplabs/knp-menu-bundle": "^3.2",
        "laminas/laminas-code": "4.7.0",
        "phpdocumentor/reflection-docblock": "^5.3",
        "phpstan/phpdoc-parser": "^1.6.4",
        "stof/doctrine-extensions-bundle": "*",
        "symfony/asset": "6.*",
        "symfony/console": "6.*",
        "symfony/doctrine-messenger": "6.*",
        "symfony/dotenv": "6.*",
        "symfony/expression-language": "6.*",
        "symfony/flex": "^2.2",
        "symfony/form": "6.*",
        "symfony/framework-bundle": "6.*",
        "symfony/http-client": "6.*",
        "symfony/intl": "6.*",
        "symfony/ldap": "6.*",
        "symfony/mailer": "6.*",
        "symfony/maker-bundle": "^1.45",
        "symfony/mime": "6.*",
        "symfony/monolog-bundle": "^3.0",
        "symfony/notifier": "6.*",
        "symfony/process": "6.*",
        "symfony/property-access": "6.*",
        "symfony/property-info": "6.*",
        "symfony/proxy-manager-bridge": "6.*",
        "symfony/runtime": "6.*",
        "symfony/security-bundle": "6.*",
        "symfony/serializer": "6.*",
        "symfony/string": "6.*",
        "symfony/translation": "6.*",
        "symfony/twig-bundle": "6.*",
        "symfony/validator": "6.*",
        "symfony/web-link": "6.*",
        "symfony/webapp-meta": "^1.0",
        "symfony/webpack-encore-bundle": "^1.15",
        "symfony/yaml": "6.*",
        "twig/cssinliner-extra": "^3.3",
        "twig/extra-bundle": "^3.4",
        "twig/inky-extra": "^3.3",
        "twig/intl-extra": "^3.4",
        "twig/twig": "^3.0"
    },
    "config": {
        "allow-plugins": {
            "composer/package-versions-deprecated": true,
            "symfony/flex": true,
            "symfony/runtime": true
        },
        "optimize-autoloader": true,
        "preferred-install": {
            "*": "dist"
        },
        "sort-packages": true
    },
    "autoload": {
        "psr-4": {
            "App\\": "src/"
        }
    },
    "autoload-dev": {
        "psr-4": {
            "App\\Tests\\": "tests/"
        }
    },
    "replace": {
        "symfony/polyfill-ctype": "*",
        "symfony/polyfill-iconv": "*",
        "symfony/polyfill-php72": "*",
        "symfony/polyfill-php73": "*",
        "symfony/polyfill-php74": "*",
        "symfony/polyfill-php80": "*"
    },
    "scripts": {
        "auto-scripts": {
            "assets:install %PUBLIC_DIR%": "symfony-cmd"
        },
        "post-install-cmd": [
            "@auto-scripts"
        ],
        "post-update-cmd": [
            "@auto-scripts"
        ]
    },
    "conflict": {
        "symfony/symfony": "*"
    },
    "extra": {
        "symfony": {
            "allow-contrib": false,
            "require": "6.2.*"
        }
    },
    "require-dev": {
        "symfony/debug-bundle": "6.*",
        "symfony/stopwatch": "6.*",
        "symfony/var-dumper": "6.*",
        "symfony/web-profiler-bundle": "6.*"
    },
    "repositories": [
        {
            "type": "vcs",
            "url": "spoecial"
        },
        {
            "type": "path",
            "url": local"
        }
    ]
}
samuel-sol commented 1 year ago

Ok, removing from require-dev the symfony/web-profiler-bundle removes the issue, even though this package was not updated recently.

greg0ire commented 1 year ago

Lol what

greg0ire commented 1 year ago

@samuel-sol reading your first message, it sounds like you downgraded doctrine/migrations to 2.5.2, then upgraded again to 2.5.3, expecting the bug to be somehow fixed… if that's the case, I suggest you read https://github.com/doctrine/migrations/issues/1302#issuecomment-1380116851 again, but slowly.

TTATPuOT commented 1 year ago

Same problem here with MySQL 8.0.25, PHP 7.4. But im use old Doctrine 2.8.0.

spajxo commented 1 year ago

If you are using getsentry/sentry-symfony this could be related https://github.com/getsentry/sentry-symfony/issues/732

MichaelMackus commented 1 year ago

Same problem for me .... I'm on doctrine/migrations 3.6.0

kwisatz commented 6 months ago

EDIT: I realize that this is an old issue, but this just came up on my project and I cannot see in the comments above, a clear indication of what is causing this or how to go about finding the correct solution.

Fixed by https://github.com/doctrine/dbal/releases/tag/3.5.3 (don't get confused, it's another package, it just happens to have the same version number)

@greg0ire This does not seem to solve the problem in all cases. I am running dbal 3.8.4 and migrations 3.5.5 with the changes shown in https://github.com/doctrine/dbal/pull/5861/files, yet, the problem is still there or another problem with the same symptoms is. I have the full mariadb version string indicated in my DATABASE_URL environment variable (?serverVersion=mariadb-10.11.6&charset=utf8).

Here's a dump of the TableDiff object. As @samuel-sol mentioned, it seems to want to change null to "NULL" for the column defaults:

^ Doctrine\DBAL\Schema\TableDiff^ {#477
  +name: "migration_versions"
  +newName: false
  +addedColumns: []
  +changedColumns: array:1 [
    "executed_at" => Doctrine\DBAL\Schema\ColumnDiff^ {#488
      +oldColumnName: "executed_at"
      +column: Doctrine\DBAL\Schema\Column^ {#495
        #_name: "executed_at"
        #_namespace: null
        #_quoted: false
        #_type: Doctrine\DBAL\Types\DateTimeType^ {#515}
        #_length: null
        #_precision: 10
        #_scale: 0
        #_unsigned: false
        #_fixed: false
        #_notnull: false
        #_default: null
        #_autoincrement: false
        #_platformOptions: []
        #_columnDefinition: null
        #_comment: null
        #_customSchemaOptions: []
      }
      +changedProperties: array:1 [
        0 => "default"
      ]
      +fromColumn: Doctrine\DBAL\Schema\Column^ {#534
        #_name: "executed_at"
        #_namespace: null
        #_quoted: false
        #_type: Doctrine\DBAL\Types\DateTimeType^ {#515}
        #_length: 0
        #_precision: 10
        #_scale: 0
        #_unsigned: false
        #_fixed: false
        #_notnull: false
        #_default: "NULL"
        #_autoincrement: false
        #_platformOptions: []
        #_columnDefinition: null
        #_comment: null
        #_customSchemaOptions: []
      }
    }
  ]
  +removedColumns: []
  +renamedColumns: []
  +addedIndexes: []
  +changedIndexes: []
  +removedIndexes: []
  +renamedIndexes: []
  +addedForeignKeys: []
  +changedForeignKeys: []
  +removedForeignKeys: []
  +fromTable: Doctrine\DBAL\Schema\Table^ {#533
    #_name: "migration_versions"
    #_namespace: null
    #_quoted: false
    #_columns: array:3 [
      "version" => Doctrine\DBAL\Schema\Column^ {#528
        #_name: "version"
        #_namespace: null
        #_quoted: false
        #_type: Doctrine\DBAL\Types\StringType^ {#505}
        #_length: 191
        #_precision: 10
        #_scale: 0
        #_unsigned: false
        #_fixed: false
        #_notnull: true
        #_default: null
        #_autoincrement: false
        #_platformOptions: []
        #_columnDefinition: null
        #_comment: null
        #_customSchemaOptions: []
      }
      "executed_at" => Doctrine\DBAL\Schema\Column^ {#534}
      "execution_time" => Doctrine\DBAL\Schema\Column^ {#535
        #_name: "execution_time"
        #_namespace: null
        #_quoted: false
        #_type: Doctrine\DBAL\Types\IntegerType^ {#538}
        #_length: null
        #_precision: 10
        #_scale: 0
        #_unsigned: false
        #_fixed: false
        #_notnull: false
        #_default: "NULL"
        #_autoincrement: false
        #_platformOptions: []
        #_columnDefinition: null
        #_comment: null
        #_customSchemaOptions: []
      }
    ]
    #_indexes: array:1 [
      "primary" => Doctrine\DBAL\Schema\Index^ {#536
        #_name: "PRIMARY"
        #_namespace: null
        #_quoted: false
        #_columns: array:1 [
          "version" => Doctrine\DBAL\Schema\Identifier^ {#525
            #_name: "version"
            #_namespace: null
            #_quoted: false
          }
        ]
        #_isUnique: true
        #_isPrimary: true
        #_flags: []
        -options: array:1 [
          "lengths" => array:1 [
            0 => null
          ]
        ]
      }
    ]
    #_primaryKeyName: "primary"
    #uniqueConstraints: []
    #_fkConstraints: []
    #_options: array:6 [
      "create_options" => []
      "engine" => "InnoDB"
      "collation" => "utf8mb3_unicode_ci"
      "charset" => "utf8mb3"
      "autoincrement" => null
      "comment" => ""
    ]
    #_schemaConfig: null
    -implicitIndexes: []
  }
}
greg0ire commented 6 months ago

Since OP seems to be satisfied with this, it means this is a different issue with the same symptoms. You should open another issue if there isn't one already about this null vs "NULL" thingy.

kwisatz commented 6 months ago

@greg0ire I might. I have a feeling this is some incompatibility between some doctrine component and the mariadb version we're running. I've seen more instances of quoted NULL strings in the migration I was able to create by temporarily deactivating the needsUpdate check.

greg0ire commented 6 months ago

OK. In the meantime, I'll lock this, as the error message can be caused by many different reasons.