Migration script hangs on the same query

[liquidsanek]

💡 Summary

Hello, as I tried to update my instance from version 12.98.0 to the latest version (12.110.1), the migration scripts hang constantly on the same query.

🥰 Expected Behavior

The project is built correctly, and the migration scripts are all executed.

🤬 Actual Behavior

There are errors during the build process:

Found 200 errors in 84 files.

Errors  Files
     1  node_modules/@types/express-serve-static-core/index.d.ts:501
     8  node_modules/htmlparser2/node_modules/domelementtype/lib/index.d.ts:20
     1  node_modules/ip-address/dist/lib/ipv4.d.ts:2
     1  node_modules/ip-address/dist/lib/ipv6.d.ts:3
     1  node_modules/ip-cidr/index.d.ts:2
     1  node_modules/pureimage/types/text.d.ts:1
     1  src/boot/index.ts:15
     3  src/boot/master.ts:9
     2  src/daemons/queue-stats.ts:4
     2  src/daemons/server-stats.ts:5
     1  src/db/postgre.ts:2
     1  src/db/redis.ts:1
     2  src/misc/antenna-cache.ts:17
     1  src/misc/app-lock.ts:3
     1  src/misc/emoji-regex.ts:1
     2  src/misc/extract-custom-emojis-from-mfm.ts:9
     2  src/misc/extract-hashtags.ts:6
     1  src/misc/extract-mentions.ts:10
     1  src/misc/fetch.ts:49
     1  src/misc/get-note-summary.ts:8
     1  src/misc/i18n.ts:16
     1  src/misc/schema.ts:161
     2  src/misc/webhook-cache.ts:19
     1  src/models/repositories/gallery-post.ts:16
     2  src/models/repositories/note.ts:221
     2  src/models/repositories/page.ts:21
     2  src/queue/processors/db/export-custom-emojis.ts:6
     1  src/queue/processors/db/import-custom-emojis.ts:4
     1  src/remote/activitypub/kernel/move/index.ts:1
     1  src/remote/activitypub/kernel/update/index.ts:26
     5  src/remote/activitypub/models/person.ts:62
     1  src/remote/activitypub/renderer/flag.ts:13
     1  src/remote/activitypub/renderer/note.ts:114
     4  src/remote/activitypub/resolver.ts:64
     2  src/server/activitypub.ts:61
     2  src/server/activitypub/followers.ts:18
     2  src/server/activitypub/following.ts:19
     3  src/server/activitypub/outbox.ts:23
     1  src/server/api/api-handler.ts:9
     1  src/server/api/common/signin.ts:37
     4  src/server/api/endpoints.ts:712
     1  src/server/api/endpoints/admin/announcements/list.ts:66
     1  src/server/api/endpoints/admin/drive/files.ts:41
     1  src/server/api/endpoints/admin/drive/show-file.ts:172
     1  src/server/api/endpoints/admin/emoji/list.ts:66
     1  src/server/api/endpoints/admin/meta.ts:319
     1  src/server/api/endpoints/admin/show-users.ts:49
     1  src/server/api/endpoints/drive/files.ts:36
     1  src/server/api/endpoints/drive/files/create.ts:51
     1  src/server/api/endpoints/drive/files/find-by-hash.ts:31
     1  src/server/api/endpoints/drive/files/find.ts:33
     3  src/server/api/endpoints/drive/files/show.ts:53
     2  src/server/api/endpoints/drive/files/update.ts:60
     1  src/server/api/endpoints/drive/files/upload-from-url.ts:40
     1  src/server/api/endpoints/drive/stream.ts:35
     2  src/server/api/endpoints/endpoint.ts:25
     1  src/server/api/endpoints/i/authorized-apps.ts:34
     6  src/server/api/endpoints/i/update.ts:123
     6  src/server/api/endpoints/messaging/messages.ts:72
     8  src/server/api/endpoints/messaging/messages/create.ts:94
     7  src/server/api/endpoints/notes/create.ts:188
     1  src/server/api/endpoints/notes/reactions.ts:59
     4  src/server/api/endpoints/notes/search-by-tag.ts:88
     2  src/server/api/endpoints/notes/translate.ts:86
     2  src/server/api/endpoints/pages/show.ts:51
     4  src/server/api/endpoints/users/followers.ts:68
     4  src/server/api/endpoints/users/following.ts:68
     2  src/server/api/private/signin.ts:17
     3  src/server/api/private/signup-pending.ts:7
     2  src/server/api/private/signup.ts:14
    13  src/server/api/service/discord.ts:149
    11  src/server/api/service/github.ts:147
     8  src/server/api/service/twitter.ts:128
     2  src/server/api/stream/channels/queue-stats.ts:4
     2  src/server/api/stream/channels/server-stats.ts:4
     2  src/server/api/stream/index.ts:196
     5  src/server/index.ts:60
     1  src/server/nodeinfo.ts:97
     1  src/server/web/manifest.ts:2
     1  src/server/web/url-preview.ts:34
     2  src/services/drive/add-file.ts:478
     4  src/services/logger.ts:3
     1  src/services/note/delete.ts:52
     2  src/services/user-cache.ts:11

I could post the full build logs if needed, I haven't yet because of how lengthy it is.

And the command yarn migrate hangs on the query:

Migration chartV61643967331284 has been executed successfully.
query: SELECT "userId", "mutedWords" FROM "user_profile" WHERE "userHost" IS NULL
query: UPDATE "user_profile" SET "mutedWords" = $1 WHERE "userId" = $2 -- PARAMETERS: ["[]","8pflayyvr4"]

📝 Steps to Reproduce

1. git checkout master
2. git pull
3. git submodule update --init
4. yarn cleanall
5. yarn install
6. NODE_ENV=production yarn build
7. yarn migrate

📌 Environment

Misskey version: Update from 12.98.0 to 12.110.1 Your OS: Debian 10 Your browser: / NodeJS's version: Tried with 16.8, then 16.15 and finally 18.x after being told to update Node

Thanks!

[Johann150]

The problem with errors during the build process is known and getting rid of all the errors is currently the 1st roadmap entry. Misskey should still run despite these build errors.

The query you posted does not strike me as anything out of the ordinary, or even anything that should take a long time to complete. I assume there is no further output after that? Maybe you can check if the PostgreSQL server logs have any more information.

[liquidsanek]

even anything that should take a long time to complete.

I've left it going for at least 20 minutes, and as you said, seeing the query, it really shouldn't take that long, which is why it seems odd to me.

I assume there is no further output after that?

Nothing. It just gets stuck on this query and all I can do is either wait indefinitely or interrupt the migration process.

Maybe you can check if the PostgreSQL server logs have any more information.

I checked and nothing related to said migration queries are showing up, only Misskey failing to do certain queries since the migration scripts are not fully executed yet, which was before I stopped Misskey's process. I can't find any relevant log about this issue.

[xianonn]

ここで止まっているみたいですね。 https://github.com/misskey-dev/misskey/blob/a29ff7b1fa45e01bc7c097e8d770a4c4eedf2a8f/packages/backend/migration/1644010796173-convert-hard-mutes.js#L56-L62

ちゃんと調べていないのですが 12.98.0 から 12.110.1 まで一気に migrate するとデッドロックして UPDATE できていないような気がします。

一旦 12.105.0 にアップデートしてから 12.110.1 にアップデートすれば成功しそうです。

[xianonn]

12.105.0 はビルドに失敗するのでダメでした。

12.110.1 のままでもトランザクションを無効にすることで migrate できました。 yarn migrate の代わりに下のコマンドを実行します。

cd packages/backend
npx typeorm migration:run -d ormconfig.js -t false
cd ../..

[xianonn]

一応データベースのバックアップを取ってから実行した方が良いかも。

[liquidsanek]

It worked! The instance started successfully, and no errors are appearing in the logs. Thanks for the help @xianonn ! :)

Closing the issue since it's solved now

[liquidsanek]

It worked! The instance started successfully, and no errors are appearing in the logs. Thanks for the help @xianonn ! :)

Closing the issue since it's solved now