search

matrix-org / dendrite

Dendrite is a second-generation Matrix homeserver written in Go!
https://matrix-org.github.io/dendrite/
Apache License 2.0
5.64k stars 663 forks source link

dendrite keeps crashing #3148

Open clandmeter opened 1 year ago

clandmeter commented 1 year ago

We are running dendrite monolith from official docker repo with version 0.13.1+49d75d3 with PG. We suddenly are experiencing server crashes with the following in our logs:

dendrite-dendrite-1  | time="2023-07-10T08:36:42.877813732Z" level=info msg="Server is ready" component=jetstream
dendrite-dendrite-1  | time="2023-07-10T08:36:43.106458129Z" level=debug msg="Updating server ACLs for \"!RcWPWcZrMeBxOGaalX:matrix.org\"" allow_ip_literals=false num_allowed=1 num_denied=623
dendrite-dendrite-1  | time="2023-07-10T08:36:43.179897070Z" level=debug msg="Setting DB connection limits" conn_max_lifetime=-1s data_source_name="postgresql://postgres/dendrite?sslmode=disable" max_idle_conns=5 max_open_conns=90
dendrite-dendrite-1  | time="2023-07-10T08:36:43.437272507Z" level=panic msg="roomserver output log: write room event failure" add="[$FqTXDcxpzANTjqqfdBMPfyWoZqAQkZKA39prqIHaaQQ $drRD4AYpfNyDjygDKQV3VNqcTwjLhCiqAJxz4h3ANls $fsBrKinoE0Q-9ZJvIvOInCJKM7yy2xcJ7TNw3_iKqrk $JA_gYx7b_mdv1nXl5lkSrpCi5QT481e2BfhA34TXbRs $jAk4qdFKOr9c4C5m0ltrHas7UupI_cJh6unsUi8h9lU $8SBbgT78yrC0IGCyk_dUYwqjuAM052UKnqzh0uNqIak $vj7I9TT5N6_Bre6aTLFJThymBJtSD0JY2Ix9zYpcFFw $CL0ULuBK5JobhkcCcRimilY-02oSTjWyn-v4he0CY-M $JnRiQ33ZTD53x2KmphyF1e3yG31nq1ighga27IizLxo]" del="[$F1mZXPwlz9CTNxSjgQXLLOCdL3KJejJjAgKcrmwZW9s $EZM5oaY963eFLV4Gm-5yuH6Skvn8qXl0sFYErlWqUa4 $drRD4AYpfNyDjygDKQV3VNqcTwjLhCiqAJxz4h3ANls $fsBrKinoE0Q-9ZJvIvOInCJKM7yy2xcJ7TNw3_iKqrk $JA_gYx7b_mdv1nXl5lkSrpCi5QT481e2BfhA34TXbRs $jAk4qdFKOr9c4C5m0ltrHas7UupI_cJh6unsUi8h9lU $F1mZXPwlz9CTNxSjgQXLLOCdL3KJejJjAgKcrmwZW9s $FO8sjr_wqlvLo0ijO2TqBBUej3lg5HhvRab8OeOerKo $J9qnDRKI2ghzzFjIIgdePHur88BuqehK-2R4jxIvYbg $rBVR-cSFwWE7j-Le1ZMAyniV3RheQCFdQyKRDyMzbfc $f1ai0vfQcvjwZqllXK98-_qPquysxXcDnNM0NuZZ-x4 $FqTXDcxpzANTjqqfdBMPfyWoZqAQkZKA39prqIHaaQQ]" error="missing state events" event="{\"auth_events\":[\"$F35rpro7JOOszmY1uvC02D9wbRoJknNdvZw-da2ZPfE\",\"$cUryPGSGk6SNtI1Ni6hyeufOUqcXkHlMwkwYjGMt4DA\",\"$CMCptS0SN1RcCvjNLgwG_8pK-3JLWANxkdSqQUhLgAU\"],\"content\":{\"body\":\"\\u003e \\u003c@fireglow488:matrix.org\\u003e Hi! I've tried to join this room from my homeserver (matrix.firc.de), but element said it was banned from this room. I'm not sure if I ever joined this room before from my server. What to do?\\n\\nhttps://federationtester.matrix.org/#matrix.firc.de\",\"format\":\"org.matrix.custom.html\",\"formatted_body\":\"\\u003cmx-reply\\u003e\\u003cblockquote\\u003e\\u003ca href=\\\"https://matrix.to/#/!RcWPWcZrMeBxOGaalX:matrix.org/$xvsSJKijHLKG6DzLBWLgffyQ-3_M7i_IElfWL2YnYn8?via=matrix.org\\u0026via=envs.net\\u0026via=dendrite.matrix.org\\\"\\u003eIn reply to\\u003c/a\\u003e \\u003ca href=\\\"https://matrix.to/#/@fireglow488:matrix.org\\\"\\u003e@fireglow488:matrix.org\\u003c/a\\u003e\\u003cbr\\u003eHi! I\\u0026#39;ve tried to join this room from my homeserver (matrix.firc.de), but element said it was banned from this room. I\\u0026#39;m not sure if I ever joined this room before from my server. What to do? \\u003c/blockquote\\u003e\\u003c/mx-reply\\u003ehttps://federationtester.matrix.org/#matrix.firc.de\",\"m.relates_to\":{\"m.in_reply_to\":{\"event_id\":\"$xvsSJKijHLKG6DzLBWLgffyQ-3_M7i_IElfWL2YnYn8\"}},\"msgtype\":\"m.text\"},\"depth\":32215,\"hashes\":{\"sha256\":\"sUUOrAa4+gQSjyFxe0oErRDV9CHCYch0xq89Yq6QE4E\"},\"origin\":\"aguiarvieira.pt\",\"origin_server_ts\":1688819576127,\"prev_events\":[\"$wd1L92k-1ILw5bViz2NaA-2LepYFnLhRoqNEd7itPus\"],\"room_id\":\"!RcWPWcZrMeBxOGaalX:matrix.org\",\"sender\":\"@daedric:aguiarvieira.pt\",\"signatures\":{\"aguiarvieira.pt\":{\"ed25519:a_Yofy\":\"iOSgP/dAoqKWzsS7hVHhIKpAo2iazt87uIahrh7E/FB98kMl8C2Xm3+hF8ZeHNbYsebSWb2Y+5EZcVbrjfZ0BA\"}},\"type\":\"m.room.message\"}" event_id="$BIy1wVT--iTI-suA2aLU827hFwy9VToQ_D4hBM0egDU"
dendrite-dendrite-1  | panic: (*logrus.Entry) 0xc00049ce70
dendrite-dendrite-1  |
dendrite-dendrite-1  | goroutine 2804 [running]:
dendrite-dendrite-1  | github.com/sirupsen/logrus.(*Entry).log(0xc00049ce00, 0x0, {0xc003e7c150, 0x2f})
dendrite-dendrite-1  |  github.com/sirupsen/logrus@v1.9.3/entry.go:260 +0x4d6
dendrite-dendrite-1  | github.com/sirupsen/logrus.(*Entry).Log(0xc00049ce00, 0x0, {0xc00011ce08?, 0xe54679?, 0x0?})
dendrite-dendrite-1  |  github.com/sirupsen/logrus@v1.9.3/entry.go:304 +0x4f
dendrite-dendrite-1  | github.com/sirupsen/logrus.(*Entry).Panic(...)
dendrite-dendrite-1  |  github.com/sirupsen/logrus@v1.9.3/entry.go:342
dendrite-dendrite-1  | github.com/MFAshby/stdemuxerhook.(*StdDemuxerHook).Fire(0x0?, 0xc00049ccb0)
dendrite-dendrite-1  |  github.com/MFAshby/stdemuxerhook@v1.0.0/stdemuxerhook.go:58 +0x19e
dendrite-dendrite-1  | github.com/sirupsen/logrus.LevelHooks.Fire(0xc00011cf48?, 0x11cf18?, 0x5?)
dendrite-dendrite-1  |  github.com/sirupsen/logrus@v1.9.3/hooks.go:28 +0x7a
dendrite-dendrite-1  | github.com/sirupsen/logrus.(*Entry).fireHooks(0xc00049ccb0)
dendrite-dendrite-1  |  github.com/sirupsen/logrus@v1.9.3/entry.go:280 +0x1f0
dendrite-dendrite-1  | github.com/sirupsen/logrus.(*Entry).log(0xc00049cc40, 0x0, {0xc003e7c120, 0x2f})
dendrite-dendrite-1  |  github.com/sirupsen/logrus@v1.9.3/entry.go:242 +0x3c5
dendrite-dendrite-1  | github.com/sirupsen/logrus.(*Entry).Log(0xc00049cc40, 0x0, {0xc00011d208?, 0x0?, 0x0?})
dendrite-dendrite-1  |  github.com/sirupsen/logrus@v1.9.3/entry.go:304 +0x4f
dendrite-dendrite-1  | github.com/sirupsen/logrus.(*Entry).Logf(0xc00049cc40, 0x0, {0x185d4b4?, 0x5?}, {0x0?, 0xc005fbee10?, 0x9?})
dendrite-dendrite-1  |  github.com/sirupsen/logrus@v1.9.3/entry.go:349 +0x85
dendrite-dendrite-1  | github.com/sirupsen/logrus.(*Entry).Panicf(...)
dendrite-dendrite-1  |  github.com/sirupsen/logrus@v1.9.3/entry.go:387
dendrite-dendrite-1  | github.com/matrix-org/dendrite/federationapi/consumers.(*OutputRoomEventConsumer).onMessage(0xc0061275f0, {0x1a9b468, 0xc0062b6de0}, {0xc0062bc2a8?, 0xc005ea5ca0?, 0xc00599ba10?})
dendrite-dendrite-1  |  github.com/matrix-org/dendrite/federationapi/consumers/roomserver.go:121 +0x915
dendrite-dendrite-1  | github.com/matrix-org/dendrite/setup/jetstream.JetStreamConsumer.func2()
dendrite-dendrite-1  |  github.com/matrix-org/dendrite/setup/jetstream/helpers.go:100 +0x454
dendrite-dendrite-1  | created by github.com/matrix-org/dendrite/setup/jetstream.JetStreamConsumer
dendrite-dendrite-1  |  github.com/matrix-org/dendrite/setup/jetstream/helpers.go:44 +0x31e
dendrite-dendrite-1 exited with code 2

The server keeps crashing and we are unable to start it anymore.

S7evinK commented 1 year ago

As a quick fix for this, you should be able to

rm -rf '/var/lib/dendrite/jetstream/jetstream/$G/streams/DendriteOutputRoomEvent'

(where /var/lib/dendrite/jetstream is the path logged on startup by JetStream)

I'll be looking into this.

clandmeter commented 1 year ago

Hi @S7evinK thanks for the quick response. This does allow us to run dendrite again but we are unable to send messages. I think the error msg is: InputRoomEventsResponse: missing prev events and no other servers to ask

clandmeter commented 1 year ago

Error seems back and is crashing again:

dendrite-dendrite-1  | time="2023-07-10T11:21:39.037156000Z" level=info msg="Enabled perspective key fetcher" num_public_keys=2 server_name=matrix.org
dendrite-dendrite-1  | time="2023-07-10T11:21:39.037565750Z" level=debug msg="Setting DB connection limits" conn_max_lifetime=-1s data_source_name="postgresql://postgres/dendrite?sslmode=disable" max_idle_conns=5 max_open_conns=90
dendrite-dendrite-1  | time="2023-07-10T11:21:39.044022352Z" level=panic msg="roomserver output log: write room event failure" add="[$drRD4AYpfNyDjygDKQV3VNqcTwjLhCiqAJxz4h3ANls $fsBrKinoE0Q-9ZJvIvOInCJKM7yy2xcJ7TNw3_iKqrk $-dAY6HkPYtnHzudXRyyo6-QGfTPp63qVUrR2Xknr5lM $JA_gYx7b_mdv1nXl5lkSrpCi5QT481e2BfhA34TXbRs $jAk4qdFKOr9c4C5m0ltrHas7UupI_cJh6unsUi8h9lU $UlH7BJppYfDbJVj1KHpYBIbS87wE53YDl1XtgT80wJQ $QEv60DEzDCXSVEi6qPPZvDFIRHVT2J0ZQIeTLGLkefI $J9qnDRKI2ghzzFjIIgdePHur88BuqehK-2R4jxIvYbg $pdj8qXFMi2krM5T6Rzc5ZKnCZmXa4FAKEdCDI0WwGmM $rBVR-cSFwWE7j-Le1ZMAyniV3RheQCFdQyKRDyMzbfc $f1ai0vfQcvjwZqllXK98-_qPquysxXcDnNM0NuZZ-x4 $AvislyUVZA4G5W26ggE-OQtxdlmlhnmNxtonhFC4Tp0 $FqTXDcxpzANTjqqfdBMPfyWoZqAQkZKA39prqIHaaQQ $XZ7GsgRmSSYyYUxClmUJQbO7JRzti6x_egMPgZyLBik]" del="[$FqTXDcxpzANTjqqfdBMPfyWoZqAQkZKA39prqIHaaQQ $drRD4AYpfNyDjygDKQV3VNqcTwjLhCiqAJxz4h3ANls $fsBrKinoE0Q-9ZJvIvOInCJKM7yy2xcJ7TNw3_iKqrk $TUYHF65dYTFc27juZpnONqJjcNlhwsPCWcdO0rt7a8Q $JA_gYx7b_mdv1nXl5lkSrpCi5QT481e2BfhA34TXbRs $jAk4qdFKOr9c4C5m0ltrHas7UupI_cJh6unsUi8h9lU $iTEdcXhAj38I64PyNI8UFvJfSnCWeG8McyyPYPlfSuk $AevmQYRGhEhvq7mEmDq-15U9hW5NaFpgPrLb--Toq5g $UEx_FgHaHIpEorwbh4B9MrlhfsMpXCzdukrgPeKGi2U $CL0ULuBK5JobhkcCcRimilY-02oSTjWyn-v4he0CY-M $JnRiQ33ZTD53x2KmphyF1e3yG31nq1ighga27IizLxo $5o7ff9rBQxGqtSQb0DGXgQs5xhFJV6BhBGgzrC7WWrE]" error="missing state events" event="{\"auth_events\":[\"$F35rpro7JOOszmY1uvC02D9wbRoJknNdvZw-da2ZPfE\",\"$cUryPGSGk6SNtI1Ni6hyeufOUqcXkHlMwkwYjGMt4DA\",\"$9fggFqLoYjzFF_5G59ldsjz2d6nZcX0Yf9yRhltmmNQ\"],\"content\":{\"body\":\"😀\",\"msgtype\":\"m.text\"},\"depth\":32257,\"hashes\":{\"sha256\":\"SOj3RFvnifGYWvIEVKSsFZMc5jfepDZw9UsiZCpDZxA\"},\"origin\":\"l.llsss.top\",\"origin_server_ts\":1688906926706,\"prev_events\":[\"$Ya1mXQ_aFihCiMthizXIhReeYWhm0gzRVLG1CKCM08Q\"],\"room_id\":\"!RcWPWcZrMeBxOGaalX:matrix.org\",\"sender\":\"@l:l.llsss.top\",\"signatures\":{\"l.llsss.top\":{\"ed25519:ap4ihY\":\"Ly+ymynJiZnGQ0HhFaypk/vjLTFRPFS9WpnDnu20PjiCxRWeaZm1HX6ctxyQCrMjGT41RM63SbMyzDml3cD5BQ\"}},\"type\":\"m.room.message\"}" event_id="$OJxUnAr2xYpI18bUGfMGl3melOUX0qyZ13v7KcC-H04"
dendrite-dendrite-1  | panic: (*logrus.Entry) 0xc000261b90
dendrite-dendrite-1  |
dendrite-dendrite-1  | goroutine 286 [running]:
dendrite-dendrite-1  | github.com/sirupsen/logrus.(*Entry).log(0xc000261b20, 0x0, {0xc004241b90, 0x2f})
dendrite-dendrite-1  |  github.com/sirupsen/logrus@v1.9.3/entry.go:260 +0x4d6
dendrite-dendrite-1  | github.com/sirupsen/logrus.(*Entry).Log(0xc000261b20, 0x0, {0xc00017ae08?, 0xe54679?, 0x0?})
dendrite-dendrite-1  |  github.com/sirupsen/logrus@v1.9.3/entry.go:304 +0x4f
dendrite-dendrite-1  | github.com/sirupsen/logrus.(*Entry).Panic(...)
dendrite-dendrite-1  |  github.com/sirupsen/logrus@v1.9.3/entry.go:342
dendrite-dendrite-1  | github.com/MFAshby/stdemuxerhook.(*StdDemuxerHook).Fire(0x0?, 0xc000261a40)
dendrite-dendrite-1  |  github.com/MFAshby/stdemuxerhook@v1.0.0/stdemuxerhook.go:58 +0x19e
dendrite-dendrite-1  | github.com/sirupsen/logrus.LevelHooks.Fire(0xc00017af48?, 0x17af18?, 0x4?)
dendrite-dendrite-1  |  github.com/sirupsen/logrus@v1.9.3/hooks.go:28 +0x7a
dendrite-dendrite-1  | github.com/sirupsen/logrus.(*Entry).fireHooks(0xc000261a40)
dendrite-dendrite-1  |  github.com/sirupsen/logrus@v1.9.3/entry.go:280 +0x1f0
dendrite-dendrite-1  | github.com/sirupsen/logrus.(*Entry).log(0xc0002619d0, 0x0, {0xc004241b60, 0x2f})
dendrite-dendrite-1  |  github.com/sirupsen/logrus@v1.9.3/entry.go:242 +0x3c5
dendrite-dendrite-1  | github.com/sirupsen/logrus.(*Entry).Log(0xc0002619d0, 0x0, {0xc00017b208?, 0x0?, 0x0?})
dendrite-dendrite-1  |  github.com/sirupsen/logrus@v1.9.3/entry.go:304 +0x4f
dendrite-dendrite-1  | github.com/sirupsen/logrus.(*Entry).Logf(0xc0002619d0, 0x0, {0x185d4b4?, 0x5?}, {0x0?, 0xc00432e500?, 0xe?})
dendrite-dendrite-1  |  github.com/sirupsen/logrus@v1.9.3/entry.go:349 +0x85
dendrite-dendrite-1  | github.com/sirupsen/logrus.(*Entry).Panicf(...)
dendrite-dendrite-1  |  github.com/sirupsen/logrus@v1.9.3/entry.go:387
dendrite-dendrite-1  | github.com/matrix-org/dendrite/federationapi/consumers.(*OutputRoomEventConsumer).onMessage(0xc0042dc000, {0x1a9b468, 0xc0042d8210}, {0xc0042ae610?, 0xc0035cedb0?, 0xc0004c1a10?})
dendrite-dendrite-1  |  github.com/matrix-org/dendrite/federationapi/consumers/roomserver.go:121 +0x915
dendrite-dendrite-1  | github.com/matrix-org/dendrite/setup/jetstream.JetStreamConsumer.func2()
dendrite-dendrite-1  |  github.com/matrix-org/dendrite/setup/jetstream/helpers.go:100 +0x454
dendrite-dendrite-1  | created by github.com/matrix-org/dendrite/setup/jetstream.JetStreamConsumer
dendrite-dendrite-1  |  github.com/matrix-org/dendrite/setup/jetstream/helpers.go:44 +0x31e
dendrite-dendrite-1 exited with code 2
NyaaaWhatsUpDoc commented 1 year ago

Unsure if related, but mysellf I have been seeing the following panic on attempted startup since upgrading:

INFO[2023-07-10T13:32:05.386098778Z] Dendrite version 0.13.1+49d75d3
PANI[2023-07-10T13:32:10.599699370Z] failed to connect to room server db           error="unable to insert executed migrations: unable to prepare insert statement: database is locked"
panic: (*logrus.Entry) 0xc0003ee3f0

goroutine 1 [running]:
github.com/sirupsen/logrus.(*Entry).log(0xc0003ee380, 0x0, {0xc0000ac2a0, 0x23})
        github.com/sirupsen/logrus@v1.9.3/entry.go:260 +0x4d6
github.com/sirupsen/logrus.(*Entry).Log(0xc0003ee380, 0x0, {0xc00018f318?, 0xf425e5?, 0x0?})
        github.com/sirupsen/logrus@v1.9.3/entry.go:304 +0x4f
github.com/sirupsen/logrus.(*Entry).Panic(...)
        github.com/sirupsen/logrus@v1.9.3/entry.go:342
github.com/MFAshby/stdemuxerhook.(*StdDemuxerHook).Fire(0x0?, 0xc0003ee230)
        github.com/MFAshby/stdemuxerhook@v1.0.0/stdemuxerhook.go:58 +0x19e
github.com/sirupsen/logrus.LevelHooks.Fire(0xc00018f458?, 0x18f428?, 0x1?)
        github.com/sirupsen/logrus@v1.9.3/hooks.go:28 +0x7a
github.com/sirupsen/logrus.(*Entry).fireHooks(0xc0003ee230)
        github.com/sirupsen/logrus@v1.9.3/entry.go:280 +0x1f0
github.com/sirupsen/logrus.(*Entry).log(0xc0003ee1c0, 0x0, {0xc0000ac270, 0x23})
        github.com/sirupsen/logrus@v1.9.3/entry.go:242 +0x3c5
github.com/sirupsen/logrus.(*Entry).Log(0xc0003ee1c0, 0x0, {0xc00018f718?, 0x0?, 0x0?})
        github.com/sirupsen/logrus@v1.9.3/entry.go:304 +0x4f
github.com/sirupsen/logrus.(*Entry).Logf(0xc0003ee1c0, 0x0, {0x1843ced?, 0x161a740?}, {0x0?, 0x5a?, 0x2?})
        github.com/sirupsen/logrus@v1.9.3/entry.go:349 +0x85
github.com/sirupsen/logrus.(*Entry).Panicf(...)
        github.com/sirupsen/logrus@v1.9.3/entry.go:387
github.com/matrix-org/dendrite/roomserver.NewInternalAPI(0xc0001deaf0, 0xc00024f100, {0x0, {0x0, 0x0}, {{0x0, 0x0}, 0x5a, 0x2, 0xffffffffffffffff}, ...}, ...)
        github.com/matrix-org/dendrite/roomserver/roomserver.go:41 +0x145
main.main()
        github.com/matrix-org/dendrite/cmd/dendrite/main.go:152 +0xc25
S7evinK commented 1 year ago

@NyaaaWhatsUpDoc This is unrelated and should already be fixed by https://github.com/matrix-org/dendrite/pull/3144

S7evinK commented 1 year ago

@clandmeter Would you mind trying the main docker image? This should hopefully be fixed now.

clandmeter commented 1 year ago

Does not seem to change much. Anything else i need to do except change to main tag?

dendrite-dendrite-1  | time="2023-07-14T09:07:32.520029600Z" level=info msg="Starting external listener on :8008"
dendrite-dendrite-1  | time="2023-07-14T09:07:32.870980356Z" level=panic msg="roomserver output log: write room event failure" add="[$drRD4AYpfNyDjygDKQV3VNqcTwjLhCiqAJxz4h3ANls $fsBrKinoE0Q-9ZJvIvOInCJKM7yy2xcJ7TNw3_iKqrk $-dAY6HkPYtnHzudXRyyo6-QGfTPp63qVUrR2Xknr5lM $JA_gYx7b_mdv1nXl5lkSrpCi5QT481e2BfhA34TXbRs $jAk4qdFKOr9c4C5m0ltrHas7UupI_cJh6unsUi8h9lU $UlH7BJppYfDbJVj1KHpYBIbS87wE53YDl1XtgT80wJQ $QEv60DEzDCXSVEi6qPPZvDFIRHVT2J0ZQIeTLGLkefI $J9qnDRKI2ghzzFjIIgdePHur88BuqehK-2R4jxIvYbg $pdj8qXFMi2krM5T6Rzc5ZKnCZmXa4FAKEdCDI0WwGmM $rBVR-cSFwWE7j-Le1ZMAyniV3RheQCFdQyKRDyMzbfc $f1ai0vfQcvjwZqllXK98-_qPquysxXcDnNM0NuZZ-x4 $AvislyUVZA4G5W26ggE-OQtxdlmlhnmNxtonhFC4Tp0 $FqTXDcxpzANTjqqfdBMPfyWoZqAQkZKA39prqIHaaQQ $XZ7GsgRmSSYyYUxClmUJQbO7JRzti6x_egMPgZyLBik]" del="[$FqTXDcxpzANTjqqfdBMPfyWoZqAQkZKA39prqIHaaQQ $drRD4AYpfNyDjygDKQV3VNqcTwjLhCiqAJxz4h3ANls $fsBrKinoE0Q-9ZJvIvOInCJKM7yy2xcJ7TNw3_iKqrk $TUYHF65dYTFc27juZpnONqJjcNlhwsPCWcdO0rt7a8Q $JA_gYx7b_mdv1nXl5lkSrpCi5QT481e2BfhA34TXbRs $jAk4qdFKOr9c4C5m0ltrHas7UupI_cJh6unsUi8h9lU $iTEdcXhAj38I64PyNI8UFvJfSnCWeG8McyyPYPlfSuk $AevmQYRGhEhvq7mEmDq-15U9hW5NaFpgPrLb--Toq5g $UEx_FgHaHIpEorwbh4B9MrlhfsMpXCzdukrgPeKGi2U $CL0ULuBK5JobhkcCcRimilY-02oSTjWyn-v4he0CY-M $JnRiQ33ZTD53x2KmphyF1e3yG31nq1ighga27IizLxo $5o7ff9rBQxGqtSQb0DGXgQs5xhFJV6BhBGgzrC7WWrE]" error="missing state events" event="{\"auth_events\":[\"$F35rpro7JOOszmY1uvC02D9wbRoJknNdvZw-da2ZPfE\",\"$cUryPGSGk6SNtI1Ni6hyeufOUqcXkHlMwkwYjGMt4DA\",\"$9fggFqLoYjzFF_5G59ldsjz2d6nZcX0Yf9yRhltmmNQ\"],\"content\":{\"body\":\"😀\",\"msgtype\":\"m.text\"},\"depth\":32257,\"hashes\":{\"sha256\":\"SOj3RFvnifGYWvIEVKSsFZMc5jfepDZw9UsiZCpDZxA\"},\"origin\":\"l.llsss.top\",\"origin_server_ts\":1688906926706,\"prev_events\":[\"$Ya1mXQ_aFihCiMthizXIhReeYWhm0gzRVLG1CKCM08Q\"],\"room_id\":\"!RcWPWcZrMeBxOGaalX:matrix.org\",\"sender\":\"@l:l.llsss.top\",\"signatures\":{\"l.llsss.top\":{\"ed25519:ap4ihY\":\"Ly+ymynJiZnGQ0HhFaypk/vjLTFRPFS9WpnDnu20PjiCxRWeaZm1HX6ctxyQCrMjGT41RM63SbMyzDml3cD5BQ\"}},\"type\":\"m.room.message\"}" event_id="$OJxUnAr2xYpI18bUGfMGl3melOUX0qyZ13v7KcC-H04"
dendrite-dendrite-1  | panic: (*logrus.Entry) 0xc00497c3f0
dendrite-dendrite-1  |
dendrite-dendrite-1  | goroutine 258 [running]:
dendrite-dendrite-1  | github.com/sirupsen/logrus.(*Entry).log(0xc00497c380, 0x0, {0xc004c96450, 0x2f})
dendrite-dendrite-1  |  github.com/sirupsen/logrus@v1.9.3/entry.go:260 +0x4d6
dendrite-dendrite-1  | github.com/sirupsen/logrus.(*Entry).Log(0xc00497c380, 0x0, {0xc0000b4e08?, 0xe4e599?, 0x0?})
dendrite-dendrite-1  |  github.com/sirupsen/logrus@v1.9.3/entry.go:304 +0x4f
dendrite-dendrite-1  | github.com/sirupsen/logrus.(*Entry).Panic(...)
dendrite-dendrite-1  |  github.com/sirupsen/logrus@v1.9.3/entry.go:342
dendrite-dendrite-1  | github.com/MFAshby/stdemuxerhook.(*StdDemuxerHook).Fire(0x0?, 0xc00497c2a0)
dendrite-dendrite-1  |  github.com/MFAshby/stdemuxerhook@v1.0.0/stdemuxerhook.go:58 +0x19e
dendrite-dendrite-1  | github.com/sirupsen/logrus.LevelHooks.Fire(0xc0000b4f48?, 0xb4f18?, 0x2?)
dendrite-dendrite-1  |  github.com/sirupsen/logrus@v1.9.3/hooks.go:28 +0x7a
dendrite-dendrite-1  | github.com/sirupsen/logrus.(*Entry).fireHooks(0xc00497c2a0)
dendrite-dendrite-1  |  github.com/sirupsen/logrus@v1.9.3/entry.go:280 +0x1f0
dendrite-dendrite-1  | github.com/sirupsen/logrus.(*Entry).log(0xc00497c230, 0x0, {0xc004c96420, 0x2f})
dendrite-dendrite-1  |  github.com/sirupsen/logrus@v1.9.3/entry.go:242 +0x3c5
dendrite-dendrite-1  | github.com/sirupsen/logrus.(*Entry).Log(0xc00497c230, 0x0, {0xc0000b5208?, 0x0?, 0x0?})
dendrite-dendrite-1  |  github.com/sirupsen/logrus@v1.9.3/entry.go:304 +0x4f
dendrite-dendrite-1  | github.com/sirupsen/logrus.(*Entry).Logf(0xc00497c230, 0x0, {0x1857518?, 0x5?}, {0x0?, 0xc004968dc0?, 0xe?})
dendrite-dendrite-1  |  github.com/sirupsen/logrus@v1.9.3/entry.go:349 +0x85
dendrite-dendrite-1  | github.com/sirupsen/logrus.(*Entry).Panicf(...)
dendrite-dendrite-1  |  github.com/sirupsen/logrus@v1.9.3/entry.go:387
dendrite-dendrite-1  | github.com/matrix-org/dendrite/federationapi/consumers.(*OutputRoomEventConsumer).onMessage(0xc00371cf30, {0x1a957a8, 0xc003be0930}, {0xc000015d10?, 0xc004911e90?, 0xc0002e1a10?})
dendrite-dendrite-1  |  github.com/matrix-org/dendrite/federationapi/consumers/roomserver.go:123 +0x915
dendrite-dendrite-1  | github.com/matrix-org/dendrite/setup/jetstream.JetStreamConsumer.func2()
dendrite-dendrite-1  |  github.com/matrix-org/dendrite/setup/jetstream/helpers.go:100 +0x454
dendrite-dendrite-1  | created by github.com/matrix-org/dendrite/setup/jetstream.JetStreamConsumer
dendrite-dendrite-1  |  github.com/matrix-org/dendrite/setup/jetstream/helpers.go:44 +0x31e
dendrite-dendrite-1 exited with code 2
Ikke commented 1 year ago

Just noticed the same error message in the dendrite test suite:

FAIL github.com/matrix-org/dendrite/clientapi 100.071s 
time="2023-08-02T11:16:00Z" level=info msg="Enabling server notices at /_synapse/admin/v1/send_server_notice"
time="2023-08-02T11:16:01Z" level=info msg=PerformDeviceCreation context=missing device_id="" display_name=0xc00021f2a8 localpart=_server
time="2023-08-02T11:16:01Z" level=info msg=PerformDeviceCreation device_id="" display_name="" localpart=1 req.id=8t08xqu4WQ0j req.method=POST req.path=/_matrix/client/v3/login
time="2023-08-02T11:16:03Z" level=info msg=PerformDeviceCreation device_id="" display_name="" localpart=2 req.id=GhE0kqNGfSpI req.method=POST req.path=/_matrix/client/v3/login
time="2023-08-02T11:16:18Z" level=info msg="Enabling server notices at /_synapse/admin/v1/send_server_notice"
time="2023-08-02T11:16:18Z" level=info msg=PerformDeviceCreation context=missing device_id="" display_name=0xc0003ca8a8 localpart=_server
time="2023-08-02T11:16:19Z" level=info msg=PerformDeviceCreation device_id="" display_name="" localpart=3 req.id=JFB3uOlJr5A1 req.method=POST req.path=/_matrix/client/v3/login
time="2023-08-02T11:16:20Z" level=info msg=PerformDeviceCreation device_id="" display_name="" localpart=4 req.id=UXvZU6b8zgIi req.method=POST req.path=/_matrix/client/v3/login
time="2023-08-02T11:16:44Z" level=info msg="Enabling server notices at /_synapse/admin/v1/send_server_notice"
time="2023-08-02T11:16:44Z" level=info msg=PerformDeviceCreation context=missing device_id="" display_name=0xc000586fa8 localpart=_server
time="2023-08-02T11:16:45Z" level=info msg=PerformDeviceCreation device_id="" display_name="" localpart=5 req.id=QqFsmyFvlDP4 req.method=POST req.path=/_matrix/client/v3/login
time="2023-08-02T11:16:46Z" level=info msg=PerformDeviceCreation device_id="" display_name="" localpart=6 req.id=9HTM4D2U1NdP req.method=POST req.path=/_matrix/client/v3/login
time="2023-08-02T11:16:50Z" level=info msg="Enabling server notices at /_synapse/admin/v1/send_server_notice"
time="2023-08-02T11:16:51Z" level=info msg=PerformDeviceCreation context=missing device_id="" display_name=0xc000587da8 localpart=_server
time="2023-08-02T11:16:52Z" level=info msg=PerformDeviceCreation device_id="" display_name="" localpart=7 req.id=Kxat8g6ei5Il req.method=POST req.path=/_matrix/client/v3/login
time="2023-08-02T11:16:52Z" level=info msg=PerformDeviceCreation device_id="" display_name="" localpart=8 req.id=dTBVMta5mUPj req.method=POST req.path=/_matrix/client/v3/login
time="2023-08-02T11:16:58Z" level=info msg="Enabling server notices at /_synapse/admin/v1/send_server_notice"
time="2023-08-02T11:16:58Z" level=info msg=PerformDeviceCreation context=missing device_id="" display_name=0xc0019901a8 localpart=_server
time="2023-08-02T11:16:59Z" level=info msg=PerformDeviceCreation device_id="" display_name="" localpart=9 req.id=HXFHGja5Gyuz req.method=POST req.path=/_matrix/client/v3/login
time="2023-08-02T11:17:00Z" level=info msg=PerformDeviceCreation device_id="" display_name="" localpart=10 req.id=brAyljywILxQ req.method=POST req.path=/_matrix/client/v3/login
time="2023-08-02T11:17:00Z" level=info msg="Cleaning old notifications"
time="2023-08-02T11:17:00Z" level=error msg="Failed to clean old notifications" error="sqlutil.WithTransaction.Begin: sql: database is closed"
time="2023-08-02T11:17:11Z" level=info msg="Enabling server notices at /_synapse/admin/v1/send_server_notice"
time="2023-08-02T11:17:11Z" level=info msg=PerformDeviceCreation context=missing device_id="" display_name=0xc00040a4a8 localpart=_server
time="2023-08-02T11:17:11Z" level=info msg=PerformDeviceCreation device_id="" display_name="" localpart=11 req.id=eSA0IdJkFFJt req.method=POST req.path=/_matrix/client/v3/login
time="2023-08-02T11:17:13Z" level=info msg=PerformDeviceCreation device_id="" display_name="" localpart=12 req.id=JeIh2BN1UKM6 req.method=POST req.path=/_matrix/client/v3/login
time="2023-08-02T11:17:14Z" level=info msg=PerformDeviceCreation device_id="" display_name="" localpart=vhuser req.id=ztLxI74nVT1e req.method=POST req.path=/_matrix/client/v3/login
time="2023-08-02T11:17:18Z" level=info msg="Cleaning old notifications"
time="2023-08-02T11:17:18Z" level=error msg="Failed to clean old notifications" error="sqlutil.WithTransaction.Begin: sql: database is closed"
time="2023-08-02T11:17:29Z" level=info msg="Enabled perspective key fetcher" num_public_keys=2 server_name=matrix.org
time="2023-08-02T11:17:32Z" level=info msg="Enabling server notices at /_synapse/admin/v1/send_server_notice"
time="2023-08-02T11:17:32Z" level=info msg=PerformDeviceCreation context=missing device_id="" display_name=0xc0005876a8 localpart=_server
time="2023-08-02T11:17:33Z" level=info msg=PerformDeviceCreation device_id="" display_name="" localpart=13 req.id=SBhYO9O3ZnHn req.method=POST req.path=/_matrix/client/v3/login
time="2023-08-02T11:17:34Z" level=warning msg="Evacuated room, purging room from roomserver now" evacuated_users=1 room_id="!1:test"
time="2023-08-02T11:17:34Z" level=warning msg="Purging room from roomserver" room_id="!1:test"
time="2023-08-02T11:17:34Z" level=warning msg="Room purged from roomserver, informing other components" room_id="!1:test"
time="2023-08-02T11:17:34Z" level=warning msg="Failed to evacuate room before purging" error="room does not exist" room_id="!doesnotexist:localhost"
time="2023-08-02T11:17:34Z" level=warning msg="Purging room from federation API" room_id="!1:test"
time="2023-08-02T11:17:34Z" level=warning msg="Room purged from federation API" room_id="!1:test"
time="2023-08-02T11:17:34Z" level=panic msg="roomserver output log: write new event failure" add="[$uaHByYO0K1SQlIVpEXSLNdYjjqkRr76Y1KPxgt3FZo4]" del="[$XhiKUU8FMg1tjoimJ0dbHgats5gpBJKmuXliiiSvwqY]" error="disk I/O error" event="{\"auth_events\":[\"$1mJkXNxeyQcmoNd7js4maIqgmrD5-5hruuqenV7VEbs\",\"$VRSXPuIdgAbLZGQroZcW-hAXVVYmfNdfVx14VlBbtE4\",\"$XhiKUU8FMg1tjoimJ0dbHgats5gpBJKmuXliiiSvwqY\"],\"content\":{\"membership\":\"leave\"},\"depth\":7,\"hashes\":{\"sha256\":\"cqjqdooNZ7pznErxHFEyhydZ63ZvIYE2cKN9Gg2STqQ\"},\"origin\":\"test\",\"origin_server_ts\":1690975054095,\"prev_events\":[\"$Ywxd1AjLmqWqR6eHTRS39JVRROiECyxdjFhOGwkMh20\"],\"prev_state\":[],\"room_id\":\"!1:test\",\"sender\":\"@13:test\",\"signatures\":{\"test\":{\"ed25519:auto\":\"0B5CxYOsFFqbnXNYPQui5WE1ZzQmlaVdhKS/shr3mjWBUgyaiCTUSFERx9iF7UwVflvkGfjKee1JTTZRwAM1Cw\"}},\"state_key\":\"@13:test\",\"type\":\"m.room.member\",\"unsigned\":{\"prev_content\":{\"membership\":\"join\"},\"prev_sender\":\"@13:test\",\"replaces_state\":\"$XhiKUU8FMg1tjoimJ0dbHgats5gpBJKmuXliiiSvwqY\"}}" event_id="$uaHByYO0K1SQlIVpEXSLNdYjjqkRr76Y1KPxgt3FZo4"
panic: (*logrus.Entry) 0xc003c7c070

goroutine 1564 [running]:
github.com/sirupsen/logrus.(*Entry).log(0xc003c7c000, 0x0, {0xc003c74060, 0x2e})
        github.com/sirupsen/logrus@v1.9.3/entry.go:260 +0x5c6
github.com/sirupsen/logrus.(*Entry).Log(0xc003c7c000, 0x0, {0xc0053aaa08, 0x1, 0x1})
        github.com/sirupsen/logrus@v1.9.3/entry.go:304 +0x9c
github.com/sirupsen/logrus.(*Entry).Logf(0xc003c7c000, 0x0, {0x2aa3bf4a9ea, 0x2e}, {0x0, 0x0, 0x0})
        github.com/sirupsen/logrus@v1.9.3/entry.go:349 +0xe8
github.com/sirupsen/logrus.(*Entry).Panicf(...)
        github.com/sirupsen/logrus@v1.9.3/entry.go:387
github.com/matrix-org/dendrite/syncapi/consumers.(*OutputRoomEventConsumer).onNewRoomEvent(0xc003a9d180, {0x2aa3c542d60, 0xc003b85440}, {0xc0056c1740, 0x0, {0xc0056bd580, 0x1, 0x4}, {0xc0056bd5c0, 0x1, ...}, ...})
        github.com/matrix-org/dendrite/syncapi/consumers/roomserver.go:280 +0x117c
github.com/matrix-org/dendrite/syncapi/consumers.(*OutputRoomEventConsumer).onMessage(0xc003a9d180, {0x2aa3c542d60, 0xc003b85440}, {0xc00566caf8, 0x1, 0x1})
        github.com/matrix-org/dendrite/syncapi/consumers/roomserver.go:119 +0x67a
github.com/matrix-org/dendrite/setup/jetstream.JetStreamConsumer.func2()
        github.com/matrix-org/dendrite/setup/jetstream/helpers.go:100 +0x522
created by github.com/matrix-org/dendrite/setup/jetstream.JetStreamConsumer
        github.com/matrix-org/dendrite/setup/jetstream/helpers.go:44 +0x31a
FAIL    github.com/matrix-org/dendrite/clientapi        100.071s
clandmeter commented 1 year ago

@S7evinK sorry to bump this issue, but we are currently stuck with a broken instance. Is there any way to get this working again, or does this need a new installation?

array-in-a-matrix commented 1 year ago

Can this issue be closed? it was fixed in i believe this commit https://github.com/matrix-org/dendrite/commit/3e314e028e5b580d0ddaa7a46d862c5a8ac351a6. Any release after v0.13.1 should not have this issue.

Ikke commented 1 year ago

It's still crashing for us:

# dendrite --version
0.13.2+a721294

Is there anything we need or can do to fix it with the latest version?

stacktrace ``` time="2023-08-24T14:56:31.010312786Z" level=error msg="roomserver output log: failed to process event" error="event $jAk4qdFKOr9c4C5m0ltrHas7UupI_cJh6unsUi8h9lU is missing" type=new_room_event time="2023-08-24T14:56:31.012777705Z" level=error msg="roomserver output log: failed to process event" error="event $jAk4qdFKOr9c4C5m0ltrHas7UupI_cJh6unsUi8h9lU is missing" type=new_room_event time="2023-08-24T14:56:31.013658915Z" level=panic msg="roomserver output log: write room event failure" add="[$drRD4AYpfNyDjygDKQV3VNqcTwjLhCiqAJxz4h3ANls $fsBrKinoE0Q-9ZJvIvOInCJKM7yy2xcJ7TNw3_iKqrk $-dAY6HkPYtnHzudXRyyo6-QGfTPp63qVUrR2Xknr5lM $JA_gYx7b_mdv1nXl5lkSrpCi5QT481e2BfhA34TXbRs $jAk4qdFKOr9c4C5m0ltrHas7UupI_cJh6unsUi8h9lU $UlH7BJppYfDbJVj1KHpYBIbS87wE53YDl1XtgT80wJQ $QEv60DEzDCXSVEi6qPPZvDFIRHVT2J0ZQIeTLGLkefI $J9qnDRKI2ghzzFjIIgdePHur88BuqehK-2R4jxIvYbg $pdj8qXFMi2krM5T6Rzc5ZKnCZmXa4FAKEdCDI0WwGmM $rBVR-cSFwWE7j-Le1ZMAyniV3RheQCFdQyKRDyMzbfc $f1ai0vfQcvjwZqllXK98-_qPquysxXcDnNM0NuZZ-x4 $AvislyUVZA4G5W26ggE-OQtxdlmlhnmNxtonhFC4Tp0 $FqTXDcxpzANTjqqfdBMPfyWoZqAQkZKA39prqIHaaQQ $XZ7GsgRmSSYyYUxClmUJQbO7JRzti6x_egMPgZyLBik]" del="[$FqTXDcxpzANTjqqfdBMPfyWoZqAQkZKA39prqIHaaQQ $drRD4AYpfNyDjygDKQV3VNqcTwjLhCiqAJxz4h3ANls $fsBrKinoE0Q-9ZJvIvOInCJKM7yy2xcJ7TNw3_iKqrk $TUYHF65dYTFc27juZpnONqJjcNlhwsPCWcdO0rt7a8Q $JA_gYx7b_mdv1nXl5lkSrpCi5QT481e2BfhA34TXbRs $jAk4qdFKOr9c4C5m0ltrHas7UupI_cJh6unsUi8h9lU $iTEdcXhAj38I64PyNI8UFvJfSnCWeG8McyyPYPlfSuk $AevmQYRGhEhvq7mEmDq-15U9hW5NaFpgPrLb--Toq5g $UEx_FgHaHIpEorwbh4B9MrlhfsMpXCzdukrgPeKGi2U $CL0ULuBK5JobhkcCcRimilY-02oSTjWyn-v4he0CY-M $JnRiQ33ZTD53x2KmphyF1e3yG31nq1ighga27IizLxo $5o7ff9rBQxGqtSQb0DGXgQs5xhFJV6BhBGgzrC7WWrE]" error="missing state events" event="{\"auth_events\":[\"$F35rpro7JOOszmY1uvC02D9wbRoJknNdvZw-da2ZPfE\",\"$cUryPGSGk6SNtI1Ni6hyeufOUqcXkHlMwkwYjGMt4DA\",\"$9fggFqLoYjzFF_5G59ldsjz2d6nZcX0Yf9yRhltmmNQ\"],\"content\":{\"body\":\"😀\",\"msgtype\":\"m.text\"},\"depth\":32257,\"hashes\":{\"sha256\":\"SOj3RFvnifGYWvIEVKSsFZMc5jfepDZw9UsiZCpDZxA\"},\"origin\":\"l.llsss.top\",\"origin_server_ts\":1688906926706,\"prev_events\":[\"$Ya1mXQ_aFihCiMthizXIhReeYWhm0gzRVLG1CKCM08Q\"],\"room_id\":\"!RcWPWcZrMeBxOGaalX:matrix.org\",\"sender\":\"@l:l.llsss.top\",\"signatures\":{\"l.llsss.top\":{\"ed25519:ap4ihY\":\"Ly+ymynJiZnGQ0HhFaypk/vjLTFRPFS9WpnDnu20PjiCxRWeaZm1HX6ctxyQCrMjGT41RM63SbMyzDml3cD5BQ\"}},\"type\":\"m.room.message\"}" event_id="$OJxUnAr2xYpI18bUGfMGl3melOUX0qyZ13v7KcC-H04" panic: (*logrus.Entry) 0xc00d158540 goroutine 6692 [running]: github.com/sirupsen/logrus.(*Entry).log(0xc00d1584d0, 0x0, {0xc007fc0a80, 0x2f}) github.com/sirupsen/logrus@v1.9.3/entry.go:260 +0x4d6 github.com/sirupsen/logrus.(*Entry).Log(0xc00d1584d0, 0x0, {0xc00016ee08?, 0xe50519?, 0x0?}) github.com/sirupsen/logrus@v1.9.3/entry.go:304 +0x4f github.com/sirupsen/logrus.(*Entry).Panic(...) github.com/sirupsen/logrus@v1.9.3/entry.go:342 github.com/MFAshby/stdemuxerhook.(*StdDemuxerHook).Fire(0x0?, 0xc00d158460) github.com/MFAshby/stdemuxerhook@v1.0.0/stdemuxerhook.go:58 +0x19e github.com/sirupsen/logrus.LevelHooks.Fire(0xc00016ef48?, 0x16ef18?, 0x0?) github.com/sirupsen/logrus@v1.9.3/hooks.go:28 +0x7a github.com/sirupsen/logrus.(*Entry).fireHooks(0xc00d158460) github.com/sirupsen/logrus@v1.9.3/entry.go:280 +0x1f0 github.com/sirupsen/logrus.(*Entry).log(0xc00d1583f0, 0x0, {0xc007fc0a50, 0x2f}) github.com/sirupsen/logrus@v1.9.3/entry.go:242 +0x3c5 github.com/sirupsen/logrus.(*Entry).Log(0xc00d1583f0, 0x0, {0xc00016f208?, 0x0?, 0x0?}) github.com/sirupsen/logrus@v1.9.3/entry.go:304 +0x4f github.com/sirupsen/logrus.(*Entry).Logf(0xc00d1583f0, 0x0, {0x185e841?, 0x5?}, {0x0?, 0xc00594eb40?, 0xe?}) github.com/sirupsen/logrus@v1.9.3/entry.go:349 +0x85 github.com/sirupsen/logrus.(*Entry).Panicf(...) github.com/sirupsen/logrus@v1.9.3/entry.go:387 github.com/matrix-org/dendrite/federationapi/consumers.(*OutputRoomEventConsumer).onMessage(0xc00b178360, {0x1aa1768, 0xc00718b020}, {0xc000286650?, 0xc00d1507e0?, 0xc00b03fa10?}) github.com/matrix-org/dendrite/federationapi/consumers/roomserver.go:123 +0x915 github.com/matrix-org/dendrite/setup/jetstream.JetStreamConsumer.func2() github.com/matrix-org/dendrite/setup/jetstream/helpers.go:100 +0x454 created by github.com/matrix-org/dendrite/setup/jetstream.JetStreamConsumer github.com/matrix-org/dendrite/setup/jetstream/helpers.go:44 +0x31e ```
hugeblank commented 11 months ago

I ran into this issue, not in #3228, but later on. After crashing the first time, repeated crashes were a segfault: dendrite_crash.log hoping that maybe this will add some context to this issue and it will get solved!

hugeblank commented 11 months ago

As a quick fix for this, you should be able to

rm -rf '/var/lib/dendrite/jetstream/jetstream/$G/streams/DendriteOutputRoomEvent'

(where /var/lib/dendrite/jetstream is the path logged on startup by JetStream)

I'll be looking into this.

just like @clandmeter, removing this file prevents outbound federated messages from being sent. I've gone from a potentially temporarily corrupted crashing instance to a half functional permanently broken instance.

Update: After waiting 20 minutes, outbound messages suddenly started working again.

array-in-a-matrix commented 11 months ago

As a quick fix for this, you should be able to

rm -rf '/var/lib/dendrite/jetstream/jetstream/$G/streams/DendriteOutputRoomEvent'

(where /var/lib/dendrite/jetstream is the path logged on startup by JetStream) I'll be looking into this.

just like @clandmeter, removing this file prevents outbound federated messages from being sent. I've gone from a potentially temporarily corrupted crashing instance to a half functional permanently broken instance.

When I was having this issue, removing the jetstream directory would only temporarily fix the issue. After a couple minutes the server would regenerate the files and crash again.

hugeblank commented 11 months ago

When I was having this issue, removing the jetstream directory would only temporarily fix the issue. After a couple minutes the server would regenerate the files and crash again.

In my case, it hasn't done that yet. I'll keep this message updated when/if it does recur.

Ikke commented 11 months ago

For us it crashes again after a couple of minutes, just as with @array-in-a-matrix

Ikke commented 11 months ago

For us it crashes again after a couple of minutes, just as with @array-in-a-matrix

After removing the DendriteOutputRoomEvent directory a couple of times and restarting it, it does not seem to crash anymore, though there are still some other issues left.

jcc10 commented 2 weeks ago

github.com/matrix-org/dendrite/roomserver.NewInternalAPI(

I would like to reply that I am getting the same error on a fresh install using the docker-compose file. Really sucks since I was hoping to try setting up a server.

How would one attempt the manual patch in docker?