💬 RFC: Migration table is already locked

[xBumblebee]

🔖 Need

Hello, We're encountering an error while trying to bring up the environment we're testing:

Backend failed to start up MigrationLocked: Migration table is already locked at 
/data1/user/dist/backend/node_modules/knex/lib/migrations/migrate/Migrator.js:342:13 at processTicksAndRejections 
(node/process/task_queues:96:5) at async Migrator._runBatch 
(/data1/user/dist/backend/node_modules/knex/lib/migrations/migrate/Migrator.js:358:7).

We’ve checked the databases, but couldn’t identify any lock issue. Has anyone encountered this issue and found a solution?

🎉 Proposal

Backend failed to start up MigrationLocked: Migration table is already locked at 
/data1/user/dist/backend/node_modules/knex/lib/migrations/migrate/Migrator.js:342:13 at processTicksAndRejections 
(node/process/task_queues:96:5) at async Migrator._runBatch 
(/data1/user/dist/backend/node_modules/knex/lib/migrations/migrate/Migrator.js:358:7).

〽️ Alternatives

No response

❌ Risks

No response

👀 Have you spent some time to check if this RFC has been raised before?

• [X] I checked and didn't find similar issue

🏢 Have you read the Code of Conduct?

• [X] I have read the Code of Conduct

[freben]

Hey,

Is there more info in the tracebacks? As in, which plugin/service that's trying to run those migrations when they fail?

[xBumblebee]

Hey,

Is there more info in the tracebacks? As in, which plugin/service that's trying to run those migrations when they fail?

Loaded config from app-config.yaml We're encountering an error: Can't take lock to run migrations: Migration table is already locked. However, upon checking our database, there are no visible locked records.

There is no more info about it sadly. But we didnt commit any new plugins etc.

[Rugvip]

Hmm, what version of Backstage are you on @xBumblebee? The error messaging looks a bit strange. You can generate a summary to post here with yarn backstage-cli info

[xBumblebee]

Hmm, what version of Backstage are you on @xBumblebee? The error messaging looks a bit strange. You can generate a summary to post here with yarn backstage-cli info

here is the summary of when i run "run backstage-cli info"

OS:   Linux 4.18.0-477.27.1.el8_8.x86_64 - linux/x64
node: v16.15.1
yarn: 1.22.19
cli:  0.22.6 (installed)
backstage:  1.13.0

Dependencies:
  @backstage/app-defaults                                   1.3.0, 1.3.1
  @backstage/backend-app-api                                0.4.2
  @backstage/backend-common                                 0.18.4
  @backstage/backend-dev-utils                              0.1.1
  @backstage/backend-plugin-api                             0.5.1
  @backstage/backend-tasks                                  0.5.1
  @backstage/catalog-client                                 1.4.1
  @backstage/catalog-model                                  1.3.0
  @backstage/cli-common                                     0.1.12
  @backstage/cli-node                                       0.1.0
  @backstage/cli                                            0.22.7, 0.22.6
  @backstage/config-loader                                  1.2.0, 1.3.0
  @backstage/config                                         1.0.7
  @backstage/core-app-api                                   1.8.0, 1.7.0
  @backstage/core-components                                0.12.5, 0.13.0, 0.13.1
  @backstage/core-plugin-api                                1.5.1
  @backstage/dev-utils                                      1.0.15, 1.0.14
  @backstage/errors                                         1.1.5
  @backstage/eslint-plugin                                  0.1.3
  @backstage/integration-aws-node                           0.1.2
  @backstage/integration-react                              1.1.12, 1.1.13
  @backstage/integration                                    1.4.4, 1.4.5
  @backstage/plugin-api-docs                                0.9.2
  @backstage/plugin-app-backend                             0.3.44
  @backstage/plugin-auth-backend                            0.18.2
  @backstage/plugin-auth-node                               0.2.13
  @backstage/plugin-catalog-backend-module-bitbucket-server 0.1.9
  @backstage/plugin-catalog-backend                         1.9.0
  @backstage/plugin-catalog-common                          1.0.13
  @backstage/plugin-catalog-graph                           0.2.29
  @backstage/plugin-catalog-import                          0.9.7
  @backstage/plugin-catalog-node                            1.3.5
  @backstage/plugin-catalog-react                           1.5.0, 1.6.0
  @backstage/plugin-catalog                                 1.10.0
  @backstage/plugin-github-actions                          0.5.17
  @backstage/plugin-home                                    0.5.0
  @backstage/plugin-org                                     0.6.7
  @backstage/plugin-permission-common                       0.7.5
  @backstage/plugin-permission-node                         0.7.7
  @backstage/plugin-permission-react                        0.4.12
  @backstage/plugin-proxy-backend                           0.2.38
  @backstage/plugin-scaffolder-backend                      1.13.0
  @backstage/plugin-scaffolder-common                       1.2.7
  @backstage/plugin-scaffolder-node                         0.1.2
  @backstage/plugin-scaffolder-react                        1.3.0
  @backstage/plugin-scaffolder                              1.13.0
  @backstage/plugin-search-backend-module-catalog           0.1.0
  @backstage/plugin-search-backend-module-pg                0.5.5
  @backstage/plugin-search-backend-module-techdocs          0.1.0
  @backstage/plugin-search-backend-node                     1.2.0
  @backstage/plugin-search-backend                          1.3.0
  @backstage/plugin-search-common                           1.2.3
  @backstage/plugin-search-react                            1.5.2
  @backstage/plugin-search                                  1.2.0
  @backstage/plugin-sonarqube-backend                       0.1.9
  @backstage/plugin-sonarqube-react                         0.1.5
  @backstage/plugin-sonarqube                               0.6.6
  @backstage/plugin-tech-radar                              0.6.3
  @backstage/plugin-techdocs-backend                        1.6.1
  @backstage/plugin-techdocs-module-addons-contrib          1.0.12
  @backstage/plugin-techdocs-node                           1.7.0
  @backstage/plugin-techdocs-react                          1.1.5
  @backstage/plugin-techdocs                                1.6.1
  @backstage/plugin-user-settings-backend                   0.1.8
  @backstage/plugin-user-settings                           0.7.2
  @backstage/release-manifests                              0.0.9
  @backstage/test-utils                                     1.3.1, 1.3.0
  @backstage/theme                                          0.2.19, 0.3.0
  @backstage/types                                          1.0.2
  @backstage/version-bridge                                 1.0.4

Another info from the backend log:

catalog info Performing database migration type=plugin
Can't take lock to run migrations: Migration table is already locked
If you are sure migrations are not running you can release the lock manually by running 'knex migrate:unlock'

[jhaals]

This is a very old version of backstage and there has been fixes towards migrations so future upgrades might not have this problem 🙏

[xBumblebee]

This is a very old version of backstage and there has been fixes towards migrations so future upgrades might not have this problem 🙏

We updated to v1.19.0 , it still deosnt work. Is there a version which fixes migration problems? also we have db connection in app-config.yaml file:

 database:
    client: pg
    connection:
      host: ${HOST}
      port: ${PORT}
      user: ${USER}
      password: ${PASSWORD}
    knexConfig:
      ....

We already checked if there is something that locks the migration, but we didnt see anything. There is no problem.