Open matrixbot opened 3 weeks ago
matrixbot
commented
3 weeks ago This comment was originally posted by @S7evinK at https://github.com/matrix-org/dendrite/issues/3148#issuecomment-1628601322.
As a quick fix for this, you should be able to
rm -rf '/var/lib/dendrite/jetstream/jetstream/$G/streams/DendriteOutputRoomEvent'(where /var/lib/dendrite/jetstream is the path logged on startup by JetStream)
I'll be looking into this.
matrixbot
commented
3 weeks ago This comment was originally posted by @clandmeter at https://github.com/matrix-org/dendrite/issues/3148#issuecomment-1628732237.
Hi @S7evinK thanks for the quick response.
This does allow us to run dendrite again but we are unable to send messages.
I think the error msg is: InputRoomEventsResponse: missing prev events and no other servers to ask
matrixbot
commented
3 weeks ago This comment was originally posted by @clandmeter at https://github.com/matrix-org/dendrite/issues/3148#issuecomment-1628766807.
Error seems back and is crashing again:
dendrite-dendrite-1 | time="2023-07-10T11:21:39.037156000Z" level=info msg="Enabled perspective key fetcher" num_public_keys=2 server_name=matrix.org
dendrite-dendrite-1 | time="2023-07-10T11:21:39.037565750Z" level=debug msg="Setting DB connection limits" conn_max_lifetime=-1s data_source_name="postgresql://postgres/dendrite?sslmode=disable" max_idle_conns=5 max_open_conns=90
dendrite-dendrite-1 | time="2023-07-10T11:21:39.044022352Z" level=panic msg="roomserver output log: write room event failure" add="[$drRD4AYpfNyDjygDKQV3VNqcTwjLhCiqAJxz4h3ANls $fsBrKinoE0Q-9ZJvIvOInCJKM7yy2xcJ7TNw3_iKqrk $-dAY6HkPYtnHzudXRyyo6-QGfTPp63qVUrR2Xknr5lM $JA_gYx7b_mdv1nXl5lkSrpCi5QT481e2BfhA34TXbRs $jAk4qdFKOr9c4C5m0ltrHas7UupI_cJh6unsUi8h9lU $UlH7BJppYfDbJVj1KHpYBIbS87wE53YDl1XtgT80wJQ $QEv60DEzDCXSVEi6qPPZvDFIRHVT2J0ZQIeTLGLkefI $J9qnDRKI2ghzzFjIIgdePHur88BuqehK-2R4jxIvYbg $pdj8qXFMi2krM5T6Rzc5ZKnCZmXa4FAKEdCDI0WwGmM $rBVR-cSFwWE7j-Le1ZMAyniV3RheQCFdQyKRDyMzbfc $f1ai0vfQcvjwZqllXK98-_qPquysxXcDnNM0NuZZ-x4 $AvislyUVZA4G5W26ggE-OQtxdlmlhnmNxtonhFC4Tp0 $FqTXDcxpzANTjqqfdBMPfyWoZqAQkZKA39prqIHaaQQ $XZ7GsgRmSSYyYUxClmUJQbO7JRzti6x_egMPgZyLBik]" del="[$FqTXDcxpzANTjqqfdBMPfyWoZqAQkZKA39prqIHaaQQ $drRD4AYpfNyDjygDKQV3VNqcTwjLhCiqAJxz4h3ANls $fsBrKinoE0Q-9ZJvIvOInCJKM7yy2xcJ7TNw3_iKqrk $TUYHF65dYTFc27juZpnONqJjcNlhwsPCWcdO0rt7a8Q $JA_gYx7b_mdv1nXl5lkSrpCi5QT481e2BfhA34TXbRs $jAk4qdFKOr9c4C5m0ltrHas7UupI_cJh6unsUi8h9lU $iTEdcXhAj38I64PyNI8UFvJfSnCWeG8McyyPYPlfSuk $AevmQYRGhEhvq7mEmDq-15U9hW5NaFpgPrLb--Toq5g $UEx_FgHaHIpEorwbh4B9MrlhfsMpXCzdukrgPeKGi2U $CL0ULuBK5JobhkcCcRimilY-02oSTjWyn-v4he0CY-M $JnRiQ33ZTD53x2KmphyF1e3yG31nq1ighga27IizLxo $5o7ff9rBQxGqtSQb0DGXgQs5xhFJV6BhBGgzrC7WWrE]" error="missing state events" event="{\"auth_events\":[\"$F35rpro7JOOszmY1uvC02D9wbRoJknNdvZw-da2ZPfE\",\"$cUryPGSGk6SNtI1Ni6hyeufOUqcXkHlMwkwYjGMt4DA\",\"$9fggFqLoYjzFF_5G59ldsjz2d6nZcX0Yf9yRhltmmNQ\"],\"content\":{\"body\":\"😀\",\"msgtype\":\"m.text\"},\"depth\":32257,\"hashes\":{\"sha256\":\"SOj3RFvnifGYWvIEVKSsFZMc5jfepDZw9UsiZCpDZxA\"},\"origin\":\"l.llsss.top\",\"origin_server_ts\":1688906926706,\"prev_events\":[\"$Ya1mXQ_aFihCiMthizXIhReeYWhm0gzRVLG1CKCM08Q\"],\"room_id\":\"!RcWPWcZrMeBxOGaalX:matrix.org\",\"sender\":\"@l:l.llsss.top\",\"signatures\":{\"l.llsss.top\":{\"ed25519:ap4ihY\":\"Ly+ymynJiZnGQ0HhFaypk/vjLTFRPFS9WpnDnu20PjiCxRWeaZm1HX6ctxyQCrMjGT41RM63SbMyzDml3cD5BQ\"}},\"type\":\"m.room.message\"}" event_id="$OJxUnAr2xYpI18bUGfMGl3melOUX0qyZ13v7KcC-H04"
dendrite-dendrite-1 | panic: (*logrus.Entry) 0xc000261b90
dendrite-dendrite-1 |
dendrite-dendrite-1 | goroutine 286 [running]:
dendrite-dendrite-1 | github.com/sirupsen/logrus.(*Entry).log(0xc000261b20, 0x0, {0xc004241b90, 0x2f})
dendrite-dendrite-1 | github.com/sirupsen/logrus@v1.9.3/entry.go:260 +0x4d6
dendrite-dendrite-1 | github.com/sirupsen/logrus.(*Entry).Log(0xc000261b20, 0x0, {0xc00017ae08?, 0xe54679?, 0x0?})
dendrite-dendrite-1 | github.com/sirupsen/logrus@v1.9.3/entry.go:304 +0x4f
dendrite-dendrite-1 | github.com/sirupsen/logrus.(*Entry).Panic(...)
dendrite-dendrite-1 | github.com/sirupsen/logrus@v1.9.3/entry.go:342
dendrite-dendrite-1 | github.com/MFAshby/stdemuxerhook.(*StdDemuxerHook).Fire(0x0?, 0xc000261a40)
dendrite-dendrite-1 | github.com/MFAshby/stdemuxerhook@v1.0.0/stdemuxerhook.go:58 +0x19e
dendrite-dendrite-1 | github.com/sirupsen/logrus.LevelHooks.Fire(0xc00017af48?, 0x17af18?, 0x4?)
dendrite-dendrite-1 | github.com/sirupsen/logrus@v1.9.3/hooks.go:28 +0x7a
dendrite-dendrite-1 | github.com/sirupsen/logrus.(*Entry).fireHooks(0xc000261a40)
dendrite-dendrite-1 | github.com/sirupsen/logrus@v1.9.3/entry.go:280 +0x1f0
dendrite-dendrite-1 | github.com/sirupsen/logrus.(*Entry).log(0xc0002619d0, 0x0, {0xc004241b60, 0x2f})
dendrite-dendrite-1 | github.com/sirupsen/logrus@v1.9.3/entry.go:242 +0x3c5
dendrite-dendrite-1 | github.com/sirupsen/logrus.(*Entry).Log(0xc0002619d0, 0x0, {0xc00017b208?, 0x0?, 0x0?})
dendrite-dendrite-1 | github.com/sirupsen/logrus@v1.9.3/entry.go:304 +0x4f
dendrite-dendrite-1 | github.com/sirupsen/logrus.(*Entry).Logf(0xc0002619d0, 0x0, {0x185d4b4?, 0x5?}, {0x0?, 0xc00432e500?, 0xe?})
dendrite-dendrite-1 | github.com/sirupsen/logrus@v1.9.3/entry.go:349 +0x85
dendrite-dendrite-1 | github.com/sirupsen/logrus.(*Entry).Panicf(...)
dendrite-dendrite-1 | github.com/sirupsen/logrus@v1.9.3/entry.go:387
dendrite-dendrite-1 | github.com/matrix-org/dendrite/federationapi/consumers.(*OutputRoomEventConsumer).onMessage(0xc0042dc000, {0x1a9b468, 0xc0042d8210}, {0xc0042ae610?, 0xc0035cedb0?, 0xc0004c1a10?})
dendrite-dendrite-1 | github.com/matrix-org/dendrite/federationapi/consumers/roomserver.go:121 +0x915
dendrite-dendrite-1 | github.com/matrix-org/dendrite/setup/jetstream.JetStreamConsumer.func2()
dendrite-dendrite-1 | github.com/matrix-org/dendrite/setup/jetstream/helpers.go:100 +0x454
dendrite-dendrite-1 | created by github.com/matrix-org/dendrite/setup/jetstream.JetStreamConsumer
dendrite-dendrite-1 | github.com/matrix-org/dendrite/setup/jetstream/helpers.go:44 +0x31e
dendrite-dendrite-1 exited with code 2This comment was originally posted by @NyaaaWhatsUpDoc at https://github.com/matrix-org/dendrite/issues/3148#issuecomment-1628976084.
Unsure if related, but mysellf I have been seeing the following panic on attempted startup since upgrading:
INFO[2023-07-10T13:32:05.386098778Z] Dendrite version 0.13.1+49d75d3
PANI[2023-07-10T13:32:10.599699370Z] failed to connect to room server db error="unable to insert executed migrations: unable to prepare insert statement: database is locked"
panic: (*logrus.Entry) 0xc0003ee3f0
goroutine 1 [running]:
github.com/sirupsen/logrus.(*Entry).log(0xc0003ee380, 0x0, {0xc0000ac2a0, 0x23})
github.com/sirupsen/logrus@v1.9.3/entry.go:260 +0x4d6
github.com/sirupsen/logrus.(*Entry).Log(0xc0003ee380, 0x0, {0xc00018f318?, 0xf425e5?, 0x0?})
github.com/sirupsen/logrus@v1.9.3/entry.go:304 +0x4f
github.com/sirupsen/logrus.(*Entry).Panic(...)
github.com/sirupsen/logrus@v1.9.3/entry.go:342
github.com/MFAshby/stdemuxerhook.(*StdDemuxerHook).Fire(0x0?, 0xc0003ee230)
github.com/MFAshby/stdemuxerhook@v1.0.0/stdemuxerhook.go:58 +0x19e
github.com/sirupsen/logrus.LevelHooks.Fire(0xc00018f458?, 0x18f428?, 0x1?)
github.com/sirupsen/logrus@v1.9.3/hooks.go:28 +0x7a
github.com/sirupsen/logrus.(*Entry).fireHooks(0xc0003ee230)
github.com/sirupsen/logrus@v1.9.3/entry.go:280 +0x1f0
github.com/sirupsen/logrus.(*Entry).log(0xc0003ee1c0, 0x0, {0xc0000ac270, 0x23})
github.com/sirupsen/logrus@v1.9.3/entry.go:242 +0x3c5
github.com/sirupsen/logrus.(*Entry).Log(0xc0003ee1c0, 0x0, {0xc00018f718?, 0x0?, 0x0?})
github.com/sirupsen/logrus@v1.9.3/entry.go:304 +0x4f
github.com/sirupsen/logrus.(*Entry).Logf(0xc0003ee1c0, 0x0, {0x1843ced?, 0x161a740?}, {0x0?, 0x5a?, 0x2?})
github.com/sirupsen/logrus@v1.9.3/entry.go:349 +0x85
github.com/sirupsen/logrus.(*Entry).Panicf(...)
github.com/sirupsen/logrus@v1.9.3/entry.go:387
github.com/matrix-org/dendrite/roomserver.NewInternalAPI(0xc0001deaf0, 0xc00024f100, {0x0, {0x0, 0x0}, {{0x0, 0x0}, 0x5a, 0x2, 0xffffffffffffffff}, ...}, ...)
github.com/matrix-org/dendrite/roomserver/roomserver.go:41 +0x145
main.main()
github.com/matrix-org/dendrite/cmd/dendrite/main.go:152 +0xc25This comment was originally posted by @S7evinK at https://github.com/matrix-org/dendrite/issues/3148#issuecomment-1630266320.
@NyaaaWhatsUpDoc This is unrelated and should already be fixed by https://github.com/matrix-org/dendrite/pull/3144
matrixbot
commented
3 weeks ago This comment was originally posted by @S7evinK at https://github.com/matrix-org/dendrite/issues/3148#issuecomment-1635540530.
@clandmeter Would you mind trying the main docker image? This should hopefully be fixed now.
matrixbot
commented
3 weeks ago This comment was originally posted by @clandmeter at https://github.com/matrix-org/dendrite/issues/3148#issuecomment-1635550190.
Does not seem to change much. Anything else i need to do except change to main tag?
dendrite-dendrite-1 | time="2023-07-14T09:07:32.520029600Z" level=info msg="Starting external listener on :8008"
dendrite-dendrite-1 | time="2023-07-14T09:07:32.870980356Z" level=panic msg="roomserver output log: write room event failure" add="[$drRD4AYpfNyDjygDKQV3VNqcTwjLhCiqAJxz4h3ANls $fsBrKinoE0Q-9ZJvIvOInCJKM7yy2xcJ7TNw3_iKqrk $-dAY6HkPYtnHzudXRyyo6-QGfTPp63qVUrR2Xknr5lM $JA_gYx7b_mdv1nXl5lkSrpCi5QT481e2BfhA34TXbRs $jAk4qdFKOr9c4C5m0ltrHas7UupI_cJh6unsUi8h9lU $UlH7BJppYfDbJVj1KHpYBIbS87wE53YDl1XtgT80wJQ $QEv60DEzDCXSVEi6qPPZvDFIRHVT2J0ZQIeTLGLkefI $J9qnDRKI2ghzzFjIIgdePHur88BuqehK-2R4jxIvYbg $pdj8qXFMi2krM5T6Rzc5ZKnCZmXa4FAKEdCDI0WwGmM $rBVR-cSFwWE7j-Le1ZMAyniV3RheQCFdQyKRDyMzbfc $f1ai0vfQcvjwZqllXK98-_qPquysxXcDnNM0NuZZ-x4 $AvislyUVZA4G5W26ggE-OQtxdlmlhnmNxtonhFC4Tp0 $FqTXDcxpzANTjqqfdBMPfyWoZqAQkZKA39prqIHaaQQ $XZ7GsgRmSSYyYUxClmUJQbO7JRzti6x_egMPgZyLBik]" del="[$FqTXDcxpzANTjqqfdBMPfyWoZqAQkZKA39prqIHaaQQ $drRD4AYpfNyDjygDKQV3VNqcTwjLhCiqAJxz4h3ANls $fsBrKinoE0Q-9ZJvIvOInCJKM7yy2xcJ7TNw3_iKqrk $TUYHF65dYTFc27juZpnONqJjcNlhwsPCWcdO0rt7a8Q $JA_gYx7b_mdv1nXl5lkSrpCi5QT481e2BfhA34TXbRs $jAk4qdFKOr9c4C5m0ltrHas7UupI_cJh6unsUi8h9lU $iTEdcXhAj38I64PyNI8UFvJfSnCWeG8McyyPYPlfSuk $AevmQYRGhEhvq7mEmDq-15U9hW5NaFpgPrLb--Toq5g $UEx_FgHaHIpEorwbh4B9MrlhfsMpXCzdukrgPeKGi2U $CL0ULuBK5JobhkcCcRimilY-02oSTjWyn-v4he0CY-M $JnRiQ33ZTD53x2KmphyF1e3yG31nq1ighga27IizLxo $5o7ff9rBQxGqtSQb0DGXgQs5xhFJV6BhBGgzrC7WWrE]" error="missing state events" event="{\"auth_events\":[\"$F35rpro7JOOszmY1uvC02D9wbRoJknNdvZw-da2ZPfE\",\"$cUryPGSGk6SNtI1Ni6hyeufOUqcXkHlMwkwYjGMt4DA\",\"$9fggFqLoYjzFF_5G59ldsjz2d6nZcX0Yf9yRhltmmNQ\"],\"content\":{\"body\":\"😀\",\"msgtype\":\"m.text\"},\"depth\":32257,\"hashes\":{\"sha256\":\"SOj3RFvnifGYWvIEVKSsFZMc5jfepDZw9UsiZCpDZxA\"},\"origin\":\"l.llsss.top\",\"origin_server_ts\":1688906926706,\"prev_events\":[\"$Ya1mXQ_aFihCiMthizXIhReeYWhm0gzRVLG1CKCM08Q\"],\"room_id\":\"!RcWPWcZrMeBxOGaalX:matrix.org\",\"sender\":\"@l:l.llsss.top\",\"signatures\":{\"l.llsss.top\":{\"ed25519:ap4ihY\":\"Ly+ymynJiZnGQ0HhFaypk/vjLTFRPFS9WpnDnu20PjiCxRWeaZm1HX6ctxyQCrMjGT41RM63SbMyzDml3cD5BQ\"}},\"type\":\"m.room.message\"}" event_id="$OJxUnAr2xYpI18bUGfMGl3melOUX0qyZ13v7KcC-H04"
dendrite-dendrite-1 | panic: (*logrus.Entry) 0xc00497c3f0
dendrite-dendrite-1 |
dendrite-dendrite-1 | goroutine 258 [running]:
dendrite-dendrite-1 | github.com/sirupsen/logrus.(*Entry).log(0xc00497c380, 0x0, {0xc004c96450, 0x2f})
dendrite-dendrite-1 | github.com/sirupsen/logrus@v1.9.3/entry.go:260 +0x4d6
dendrite-dendrite-1 | github.com/sirupsen/logrus.(*Entry).Log(0xc00497c380, 0x0, {0xc0000b4e08?, 0xe4e599?, 0x0?})
dendrite-dendrite-1 | github.com/sirupsen/logrus@v1.9.3/entry.go:304 +0x4f
dendrite-dendrite-1 | github.com/sirupsen/logrus.(*Entry).Panic(...)
dendrite-dendrite-1 | github.com/sirupsen/logrus@v1.9.3/entry.go:342
dendrite-dendrite-1 | github.com/MFAshby/stdemuxerhook.(*StdDemuxerHook).Fire(0x0?, 0xc00497c2a0)
dendrite-dendrite-1 | github.com/MFAshby/stdemuxerhook@v1.0.0/stdemuxerhook.go:58 +0x19e
dendrite-dendrite-1 | github.com/sirupsen/logrus.LevelHooks.Fire(0xc0000b4f48?, 0xb4f18?, 0x2?)
dendrite-dendrite-1 | github.com/sirupsen/logrus@v1.9.3/hooks.go:28 +0x7a
dendrite-dendrite-1 | github.com/sirupsen/logrus.(*Entry).fireHooks(0xc00497c2a0)
dendrite-dendrite-1 | github.com/sirupsen/logrus@v1.9.3/entry.go:280 +0x1f0
dendrite-dendrite-1 | github.com/sirupsen/logrus.(*Entry).log(0xc00497c230, 0x0, {0xc004c96420, 0x2f})
dendrite-dendrite-1 | github.com/sirupsen/logrus@v1.9.3/entry.go:242 +0x3c5
dendrite-dendrite-1 | github.com/sirupsen/logrus.(*Entry).Log(0xc00497c230, 0x0, {0xc0000b5208?, 0x0?, 0x0?})
dendrite-dendrite-1 | github.com/sirupsen/logrus@v1.9.3/entry.go:304 +0x4f
dendrite-dendrite-1 | github.com/sirupsen/logrus.(*Entry).Logf(0xc00497c230, 0x0, {0x1857518?, 0x5?}, {0x0?, 0xc004968dc0?, 0xe?})
dendrite-dendrite-1 | github.com/sirupsen/logrus@v1.9.3/entry.go:349 +0x85
dendrite-dendrite-1 | github.com/sirupsen/logrus.(*Entry).Panicf(...)
dendrite-dendrite-1 | github.com/sirupsen/logrus@v1.9.3/entry.go:387
dendrite-dendrite-1 | github.com/matrix-org/dendrite/federationapi/consumers.(*OutputRoomEventConsumer).onMessage(0xc00371cf30, {0x1a957a8, 0xc003be0930}, {0xc000015d10?, 0xc004911e90?, 0xc0002e1a10?})
dendrite-dendrite-1 | github.com/matrix-org/dendrite/federationapi/consumers/roomserver.go:123 +0x915
dendrite-dendrite-1 | github.com/matrix-org/dendrite/setup/jetstream.JetStreamConsumer.func2()
dendrite-dendrite-1 | github.com/matrix-org/dendrite/setup/jetstream/helpers.go:100 +0x454
dendrite-dendrite-1 | created by github.com/matrix-org/dendrite/setup/jetstream.JetStreamConsumer
dendrite-dendrite-1 | github.com/matrix-org/dendrite/setup/jetstream/helpers.go:44 +0x31e
dendrite-dendrite-1 exited with code 2This comment was originally posted by @Ikke at https://github.com/matrix-org/dendrite/issues/3148#issuecomment-1662439591.
Just noticed the same error message in the dendrite test suite:
time="2023-08-02T11:16:00Z" level=info msg="Enabling server notices at /_synapse/admin/v1/send_server_notice" time="2023-08-02T11:16:01Z" level=info msg=PerformDeviceCreation context=missing device_id="" display_name=0xc00021f2a8 localpart=_server time="2023-08-02T11:16:01Z" level=info msg=PerformDeviceCreation device_id=" " display_name=" " localpart=1 req.id=8t08xqu4WQ0j req.method=POST req.path=/_matrix/client/v3/login time="2023-08-02T11:16:03Z" level=info msg=PerformDeviceCreation device_id=" " display_name=" " localpart=2 req.id=GhE0kqNGfSpI req.method=POST req.path=/_matrix/client/v3/login time="2023-08-02T11:16:18Z" level=info msg="Enabling server notices at /_synapse/admin/v1/send_server_notice" time="2023-08-02T11:16:18Z" level=info msg=PerformDeviceCreation context=missing device_id=" " display_name=0xc0003ca8a8 localpart=_server time="2023-08-02T11:16:19Z" level=info msg=PerformDeviceCreation device_id=" " display_name=" " localpart=3 req.id=JFB3uOlJr5A1 req.method=POST req.path=/_matrix/client/v3/login time="2023-08-02T11:16:20Z" level=info msg=PerformDeviceCreation device_id=" " display_name=" " localpart=4 req.id=UXvZU6b8zgIi req.method=POST req.path=/_matrix/client/v3/login time="2023-08-02T11:16:44Z" level=info msg="Enabling server notices at /_synapse/admin/v1/send_server_notice" time="2023-08-02T11:16:44Z" level=info msg=PerformDeviceCreation context=missing device_id=" " display_name=0xc000586fa8 localpart=_server time="2023-08-02T11:16:45Z" level=info msg=PerformDeviceCreation device_id=" " display_name=" " localpart=5 req.id=QqFsmyFvlDP4 req.method=POST req.path=/_matrix/client/v3/login time="2023-08-02T11:16:46Z" level=info msg=PerformDeviceCreation device_id=" " display_name=" " localpart=6 req.id=9HTM4D2U1NdP req.method=POST req.path=/_matrix/client/v3/login time="2023-08-02T11:16:50Z" level=info msg="Enabling server notices at /_synapse/admin/v1/send_server_notice" time="2023-08-02T11:16:51Z" level=info msg=PerformDeviceCreation context=missing device_id=" " display_name=0xc000587da8 localpart=_server time="2023-08-02T11:16:52Z" level=info msg=PerformDeviceCreation device_id=" " display_name=" " localpart=7 req.id=Kxat8g6ei5Il req.method=POST req.path=/_matrix/client/v3/login time="2023-08-02T11:16:52Z" level=info msg=PerformDeviceCreation device_id=" " display_name=" " localpart=8 req.id=dTBVMta5mUPj req.method=POST req.path=/_matrix/client/v3/login time="2023-08-02T11:16:58Z" level=info msg="Enabling server notices at /_synapse/admin/v1/send_server_notice" time="2023-08-02T11:16:58Z" level=info msg=PerformDeviceCreation context=missing device_id=" " display_name=0xc0019901a8 localpart=_server time="2023-08-02T11:16:59Z" level=info msg=PerformDeviceCreation device_id=" " display_name=" " localpart=9 req.id=HXFHGja5Gyuz req.method=POST req.path=/_matrix/client/v3/login time="2023-08-02T11:17:00Z" level=info msg=PerformDeviceCreation device_id=" " display_name=" " localpart=10 req.id=brAyljywILxQ req.method=POST req.path=/_matrix/client/v3/login time="2023-08-02T11:17:00Z" level=info msg="Cleaning old notifications" time="2023-08-02T11:17:00Z" level=error msg="Failed to clean old notifications" error="sqlutil.WithTransaction.Begin: sql: database is closed" time="2023-08-02T11:17:11Z" level=info msg="Enabling server notices at /_synapse/admin/v1/send_server_notice" time="2023-08-02T11:17:11Z" level=info msg=PerformDeviceCreation context=missing device_id=" " display_name=0xc00040a4a8 localpart=_server time="2023-08-02T11:17:11Z" level=info msg=PerformDeviceCreation device_id=" " display_name=" " localpart=11 req.id=eSA0IdJkFFJt req.method=POST req.path=/_matrix/client/v3/login time="2023-08-02T11:17:13Z" level=info msg=PerformDeviceCreation device_id=" " display_name=" " localpart=12 req.id=JeIh2BN1UKM6 req.method=POST req.path=/_matrix/client/v3/login time="2023-08-02T11:17:14Z" level=info msg=PerformDeviceCreation device_id=" " display_name=" " localpart=vhuser req.id=ztLxI74nVT1e req.method=POST req.path=/_matrix/client/v3/login time="2023-08-02T11:17:18Z" level=info msg="Cleaning old notifications" time="2023-08-02T11:17:18Z" level=error msg="Failed to clean old notifications" error="sqlutil.WithTransaction.Begin: sql: database is closed" time="2023-08-02T11:17:29Z" level=info msg="Enabled perspective key fetcher" num_public_keys=2 server_name=matrix.org time="2023-08-02T11:17:32Z" level=info msg="Enabling server notices at /_synapse/admin/v1/send_server_notice" time="2023-08-02T11:17:32Z" level=info msg=PerformDeviceCreation context=missing device_id=" " display_name=0xc0005876a8 localpart=_server time="2023-08-02T11:17:33Z" level=info msg=PerformDeviceCreation device_id=" " display_name=" " localpart=13 req.id=SBhYO9O3ZnHn req.method=POST req.path=/_matrix/client/v3/login time="2023-08-02T11:17:34Z" level=warning msg="Evacuated room, purging room from roomserver now" evacuated_users=1 room_id="!1:test" time="2023-08-02T11:17:34Z" level=warning msg="Purging room from roomserver" room_id="!1:test" time="2023-08-02T11:17:34Z" level=warning msg="Room purged from roomserver, informing other components" room_id="!1:test" time="2023-08-02T11:17:34Z" level=warning msg="Failed to evacuate room before purging" error="room does not exist" room_id="!doesnotexist:localhost" time="2023-08-02T11:17:34Z" level=warning msg="Purging room from federation API" room_id="!1:test" time="2023-08-02T11:17:34Z" level=warning msg="Room purged from federation API" room_id="!1:test" time="2023-08-02T11:17:34Z" level=panic msg="roomserver output log: write new event failure" add="[$uaHByYO0K1SQlIVpEXSLNdYjjqkRr76Y1KPxgt3FZo4]" del="[$XhiKUU8FMg1tjoimJ0dbHgats5gpBJKmuXliiiSvwqY]" error="disk I/O error" event="{\"auth_events\":[\"$1mJkXNxeyQcmoNd7js4maIqgmrD5-5hruuqenV7VEbs\",\"$VRSXPuIdgAbLZGQroZcW-hAXVVYmfNdfVx14VlBbtE4\",\"$XhiKUU8FMg1tjoimJ0dbHgats5gpBJKmuXliiiSvwqY\"],\"content\":{\"membership\":\"leave\"},\"depth\":7,\"hashes\":{\"sha256\":\"cqjqdooNZ7pznErxHFEyhydZ63ZvIYE2cKN9Gg2STqQ\"},\"origin\":\"test\",\"origin_server_ts\":1690975054095,\"prev_events\":[\"$Ywxd1AjLmqWqR6eHTRS39JVRROiECyxdjFhOGwkMh20\"],\"prev_state\":[],\"room_id\":\"!1:test\",\"sender\":\"@13:test\",\"signatures\":{\"test\":{\"ed25519:auto\":\"0B5CxYOsFFqbnXNYPQui5WE1ZzQmlaVdhKS/shr3mjWBUgyaiCTUSFERx9iF7UwVflvkGfjKee1JTTZRwAM1Cw\"}},\"state_key\":\"@13:test\",\"type\":\"m.room.member\",\"unsigned\":{\"prev_content\":{\"membership\":\"join\"},\"prev_sender\":\"@13:test\",\"replaces_state\":\"$XhiKUU8FMg1tjoimJ0dbHgats5gpBJKmuXliiiSvwqY\"}}" event_id="$uaHByYO0K1SQlIVpEXSLNdYjjqkRr76Y1KPxgt3FZo4" panic: (*logrus.Entry) 0xc003c7c070 goroutine 1564 [running]: github.com/sirupsen/logrus.(*Entry).log(0xc003c7c000, 0x0, {0xc003c74060, 0x2e}) github.com/sirupsen/logrus@v1.9.3/entry.go:260 +0x5c6 github.com/sirupsen/logrus.(*Entry).Log(0xc003c7c000, 0x0, {0xc0053aaa08, 0x1, 0x1}) github.com/sirupsen/logrus@v1.9.3/entry.go:304 +0x9c github.com/sirupsen/logrus.(*Entry).Logf(0xc003c7c000, 0x0, {0x2aa3bf4a9ea, 0x2e}, {0x0, 0x0, 0x0}) github.com/sirupsen/logrus@v1.9.3/entry.go:349 +0xe8 github.com/sirupsen/logrus.(*Entry).Panicf(...) github.com/sirupsen/logrus@v1.9.3/entry.go:387 github.com/matrix-org/dendrite/syncapi/consumers.(*OutputRoomEventConsumer).onNewRoomEvent(0xc003a9d180, {0x2aa3c542d60, 0xc003b85440}, {0xc0056c1740, 0x0, {0xc0056bd580, 0x1, 0x4}, {0xc0056bd5c0, 0x1, ...}, ...}) github.com/matrix-org/dendrite/syncapi/consumers/roomserver.go:280 +0x117c github.com/matrix-org/dendrite/syncapi/consumers.(*OutputRoomEventConsumer).onMessage(0xc003a9d180, {0x2aa3c542d60, 0xc003b85440}, {0xc00566caf8, 0x1, 0x1}) github.com/matrix-org/dendrite/syncapi/consumers/roomserver.go:119 +0x67a github.com/matrix-org/dendrite/setup/jetstream.JetStreamConsumer.func2() github.com/matrix-org/dendrite/setup/jetstream/helpers.go:100 +0x522 created by github.com/matrix-org/dendrite/setup/jetstream.JetStreamConsumer github.com/matrix-org/dendrite/setup/jetstream/helpers.go:44 +0x31a FAIL github.com/matrix-org/dendrite/clientapi 100.071s
matrixbot
commented
3 weeks ago This comment was originally posted by @clandmeter at https://github.com/matrix-org/dendrite/issues/3148#issuecomment-1676809780.
@S7evinK sorry to bump this issue, but we are currently stuck with a broken instance. Is there any way to get this working again, or does this need a new installation?
matrixbot
commented
3 weeks ago This comment was originally posted by @array-in-a-matrix at https://github.com/matrix-org/dendrite/issues/3148#issuecomment-1691806260.
Can this issue be closed? it was fixed in i believe this commit https://github.com/matrix-org/dendrite/commit/3e314e028e5b580d0ddaa7a46d862c5a8ac351a6. Any release after v0.13.1 should not have this issue.
matrixbot
commented
3 weeks ago This comment was originally posted by @Ikke at https://github.com/matrix-org/dendrite/issues/3148#issuecomment-1691847059.
It's still crashing for us:
# dendrite --version
0.13.2+a721294Is there anything we need or can do to fix it with the latest version?
matrixbot
commented
3 weeks ago This comment was originally posted by @hugeblank at https://github.com/matrix-org/dendrite/issues/3148#issuecomment-1747250375.
I ran into this issue, not in #3228, but later on. After crashing the first time, repeated crashes were a segfault: dendrite_crash.log hoping that maybe this will add some context to this issue and it will get solved!
matrixbot
commented
3 weeks ago This comment was originally posted by @hugeblank at https://github.com/matrix-org/dendrite/issues/3148#issuecomment-1747266463.
As a quick fix for this, you should be able to
rm -rf '/var/lib/dendrite/jetstream/jetstream/$G/streams/DendriteOutputRoomEvent'(where
/var/lib/dendrite/jetstreamis the path logged on startup by JetStream)I'll be looking into this.
just like @clandmeter, removing this file prevents outbound federated messages from being sent. I've gone from a potentially temporarily corrupted crashing instance to a half functional permanently broken instance.
Update: After waiting 20 minutes, outbound messages suddenly started working again.
matrixbot
commented
3 weeks ago This comment was originally posted by @array-in-a-matrix at https://github.com/matrix-org/dendrite/issues/3148#issuecomment-1747275296.
As a quick fix for this, you should be able to
rm -rf '/var/lib/dendrite/jetstream/jetstream/$G/streams/DendriteOutputRoomEvent'(where
/var/lib/dendrite/jetstreamis the path logged on startup by JetStream) I'll be looking into this.just like @clandmeter, removing this file prevents outbound federated messages from being sent. I've gone from a potentially temporarily corrupted crashing instance to a half functional permanently broken instance.
When I was having this issue, removing the jetstream directory would only temporarily fix the issue. After a couple minutes the server would regenerate the files and crash again.
matrixbot
commented
3 weeks ago This comment was originally posted by @hugeblank at https://github.com/matrix-org/dendrite/issues/3148#issuecomment-1747279887.
When I was having this issue, removing the
jetstreamdirectory would only temporarily fix the issue. After a couple minutes the server would regenerate the files and crash again.
In my case, it hasn't done that yet. I'll keep this message updated when/if it does recur.
matrixbot
commented
3 weeks ago This comment was originally posted by @Ikke at https://github.com/matrix-org/dendrite/issues/3148#issuecomment-1748887465.
For us it crashes again after a couple of minutes, just as with @array-in-a-matrix
matrixbot
commented
3 weeks ago This comment was originally posted by @Ikke at https://github.com/matrix-org/dendrite/issues/3148#issuecomment-1749195058.
For us it crashes again after a couple of minutes, just as with @array-in-a-matrix
After removing the DendriteOutputRoomEvent directory a couple of times and restarting it, it does not seem to crash anymore, though there are still some other issues left.
matrixbot
commented
3 weeks ago This comment was originally posted by @jcc10 at https://github.com/matrix-org/dendrite/issues/3148#issuecomment-2306014896.
github.com/matrix-org/dendrite/roomserver.NewInternalAPI(
I would like to reply that I am getting the same error on a fresh install using the docker-compose file. Really sucks since I was hoping to try setting up a server.
How would one attempt the manual patch in docker?
matrixbot
commented
3 weeks ago This comment was originally posted by @alistair23 at https://github.com/matrix-org/dendrite/issues/3148#issuecomment-2401380176.
Same issue here
Oct 09 04:36:25 debian-1cpu-1gb-de-fra1 dendrite[129328]: time="2024-10-09T04:36:25.680217944Z" level=info msg="User requested to room join" room_id="#remarkable:matrix.org" ser>
Oct 09 04:36:36 debian-1cpu-1gb-de-fra1 dendrite[129328]: time="2024-10-09T04:36:36.242609247Z" level=info msg="Signalling other goroutines waiting for this goroutine to generat>
Oct 09 04:36:36 debian-1cpu-1gb-de-fra1 dendrite[129328]: time="2024-10-09T04:36:36.383662235Z" level=info msg="Signalling other goroutines waiting for this goroutine to generat>
Oct 09 04:36:36 debian-1cpu-1gb-de-fra1 dendrite[129328]: time="2024-10-09T04:36:36.564797016Z" level=info msg="Signalling other goroutines waiting for this goroutine to generat>
Oct 09 04:36:36 debian-1cpu-1gb-de-fra1 dendrite[129328]: time="2024-10-09T04:36:36.575276320Z" level=info msg="Signalling other goroutines waiting for this goroutine to generat>
Oct 09 04:36:36 debian-1cpu-1gb-de-fra1 dendrite[129328]: time="2024-10-09T04:36:36.961927671Z" level=info msg="Signalling other goroutines waiting for this goroutine to generat>
Oct 09 04:36:37 debian-1cpu-1gb-de-fra1 dendrite[129328]: time="2024-10-09T04:36:37.361856775Z" level=info msg="Signalling other goroutines waiting for this goroutine to generat>
Oct 09 04:36:43 debian-1cpu-1gb-de-fra1 dendrite[129328]: time="2024-10-09T04:36:43.511506677Z" level=info msg="Checking event signatures for 11326 events of room state" context>
Oct 09 04:36:46 debian-1cpu-1gb-de-fra1 dendrite[129328]: time="2024-10-09T04:36:46.099267296Z" level=info msg="Joined federated room with 2027 hosts" room="!qdAzZzmQKYzCeRzqOH:>
Oct 09 04:36:47 debian-1cpu-1gb-de-fra1 dendrite[129328]: time="2024-10-09T04:36:47.843926995Z" level=info msg="Submitting \"m.room.member\" event to roomserver with state snaps>
Oct 09 04:36:54 debian-1cpu-1gb-de-fra1 dendrite[129328]: time="2024-10-09T04:36:54.660875017Z" level=info msg="Starting queue due to pending events or forceWakeup"
Oct 09 04:37:17 debian-1cpu-1gb-de-fra1 dendrite[129328]: time="2024-10-09T04:37:17.491675640Z" level=info msg="User joined room successfully" room_id="#remarkable:matrix.org" s>
Oct 09 04:37:22 debian-1cpu-1gb-de-fra1 dendrite[129328]: time="2024-10-09T04:37:22.328104884Z" level=panic msg="roomserver output log: write new event failure" add="[$154762599>
Oct 09 04:37:22 debian-1cpu-1gb-de-fra1 dendrite[129328]: 64316CJbma:matrix.org $16313114664840vNOUC:chaosfield.at $1608195391638181sefah:matrix.org $159779810631628xyYTU:t2bot.>
Oct 09 04:37:22 debian-1cpu-1gb-de-fra1 dendrite[129328]: 7vnrFv:libera.chat $165056716799189YycIB:t2bot.io $166056061395940ZRedR:matrix.org $1650679185142874aISLS:t2bot.io $165>
Oct 09 04:37:22 debian-1cpu-1gb-de-fra1 dendrite[129328]: 5198698904TyHeU:matrix.org $171975454892320cZhuR:t2bot.io $171977613993244LuuvF:t2bot.io $172550123213729QAbuQ:t2bot.io>
Oct 09 04:37:22 debian-1cpu-1gb-de-fra1 dendrite[129328]: panic: (*logrus.Entry) 0xc00ff64690
Oct 09 04:37:22 debian-1cpu-1gb-de-fra1 dendrite[129328]: goroutine 392 [running]:
Oct 09 04:37:22 debian-1cpu-1gb-de-fra1 dendrite[129328]: github.com/sirupsen/logrus.(*Entry).log(0xc00ff64620, 0x0, {0xc004e25bc0, 0x2e})
Oct 09 04:37:22 debian-1cpu-1gb-de-fra1 dendrite[129328]: /opt/dendrite/go/pkg/mod/github.com/sirupsen/logrus@v1.9.3/entry.go:260 +0x491
Oct 09 04:37:22 debian-1cpu-1gb-de-fra1 dendrite[129328]: github.com/sirupsen/logrus.(*Entry).Log(0xc00ff64620, 0x0, {0xc00fe52288?, 0x13dbc1c?, 0x0?})
Oct 09 04:37:22 debian-1cpu-1gb-de-fra1 dendrite[129328]: /opt/dendrite/go/pkg/mod/github.com/sirupsen/logrus@v1.9.3/entry.go:304 +0x48
Oct 09 04:37:22 debian-1cpu-1gb-de-fra1 dendrite[129328]: github.com/sirupsen/logrus.(*Entry).Panic(...)
Oct 09 04:37:22 debian-1cpu-1gb-de-fra1 dendrite[129328]: /opt/dendrite/go/pkg/mod/github.com/sirupsen/logrus@v1.9.3/entry.go:342
Oct 09 04:37:22 debian-1cpu-1gb-de-fra1 dendrite[129328]: github.com/MFAshby/stdemuxerhook.(*StdDemuxerHook).Fire(0x0?, 0xc00ff64540)
Oct 09 04:37:22 debian-1cpu-1gb-de-fra1 dendrite[129328]: /opt/dendrite/go/pkg/mod/github.com/!m!f!ashby/stdemuxerhook@v1.0.0/stdemuxerhook.go:58 +0x198
Oct 09 04:37:22 debian-1cpu-1gb-de-fra1 dendrite[129328]: github.com/sirupsen/logrus.LevelHooks.Fire(0xc00fe523c8?, 0xfe52398?, 0x4?)
Oct 09 04:37:22 debian-1cpu-1gb-de-fra1 dendrite[129328]: /opt/dendrite/go/pkg/mod/github.com/sirupsen/logrus@v1.9.3/hooks.go:28 +0x70
Oct 09 04:37:22 debian-1cpu-1gb-de-fra1 dendrite[129328]: github.com/sirupsen/logrus.(*Entry).fireHooks(0xc00ff64540)
Oct 09 04:37:22 debian-1cpu-1gb-de-fra1 dendrite[129328]: /opt/dendrite/go/pkg/mod/github.com/sirupsen/logrus@v1.9.3/entry.go:280 +0x1fa
Oct 09 04:37:22 debian-1cpu-1gb-de-fra1 dendrite[129328]: github.com/sirupsen/logrus.(*Entry).log(0xc00ff644d0, 0x0, {0xc004e25b90, 0x2e})
Oct 09 04:37:22 debian-1cpu-1gb-de-fra1 dendrite[129328]: /opt/dendrite/go/pkg/mod/github.com/sirupsen/logrus@v1.9.3/entry.go:242 +0x399
Oct 09 04:37:22 debian-1cpu-1gb-de-fra1 dendrite[129328]: github.com/sirupsen/logrus.(*Entry).Log(0xc00ff644d0, 0x0, {0xc00fe52678?, 0x0?, 0x0?})
Oct 09 04:37:22 debian-1cpu-1gb-de-fra1 dendrite[129328]: /opt/dendrite/go/pkg/mod/github.com/sirupsen/logrus@v1.9.3/entry.go:304 +0x48
Oct 09 04:37:22 debian-1cpu-1gb-de-fra1 dendrite[129328]: github.com/sirupsen/logrus.(*Entry).Logf(0xc00ff644d0, 0x0, {0x18f65bd?, 0x3?}, {0x0?, 0x0?, 0xc009df62f0?})
Oct 09 04:37:22 debian-1cpu-1gb-de-fra1 dendrite[129328]: /opt/dendrite/go/pkg/mod/github.com/sirupsen/logrus@v1.9.3/entry.go:349 +0x7c
Oct 09 04:37:22 debian-1cpu-1gb-de-fra1 dendrite[129328]: github.com/sirupsen/logrus.(*Entry).Panicf(...)
Oct 09 04:37:22 debian-1cpu-1gb-de-fra1 dendrite[129328]: /opt/dendrite/go/pkg/mod/github.com/sirupsen/logrus@v1.9.3/entry.go:387
Oct 09 04:37:22 debian-1cpu-1gb-de-fra1 dendrite[129328]: github.com/matrix-org/dendrite/syncapi/consumers.(*OutputRoomEventConsumer).onNewRoomEvent(0xc000139550, {0x1b7fc60, 0x>
Oct 09 04:37:22 debian-1cpu-1gb-de-fra1 dendrite[129328]: /opt/dendrite/syncapi/consumers/roomserver.go:290 +0x1038
Oct 09 04:37:22 debian-1cpu-1gb-de-fra1 dendrite[129328]: github.com/matrix-org/dendrite/syncapi/consumers.(*OutputRoomEventConsumer).onMessage(0xc000139550, {0xc007b1e320?, 0x1>
Oct 09 04:37:22 debian-1cpu-1gb-de-fra1 dendrite[129328]: /opt/dendrite/syncapi/consumers/roomserver.go:125 +0x618
Oct 09 04:37:22 debian-1cpu-1gb-de-fra1 dendrite[129328]: github.com/matrix-org/dendrite/setup/jetstream.JetStreamConsumer.func2()
Oct 09 04:37:22 debian-1cpu-1gb-de-fra1 dendrite[129328]: /opt/dendrite/setup/jetstream/helpers.go:100 +0x454
Oct 09 04:37:22 debian-1cpu-1gb-de-fra1 dendrite[129328]: created by github.com/matrix-org/dendrite/setup/jetstream.JetStreamConsumer in goroutine 1
Oct 09 04:37:22 debian-1cpu-1gb-de-fra1 dendrite[129328]: /opt/dendrite/setup/jetstream/helpers.go:44 +0x2f2
Oct 09 04:37:22 debian-1cpu-1gb-de-fra1 systemd[1]: dendrite.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
This issue was originally created by @clandmeter at https://github.com/matrix-org/dendrite/issues/3148.
We are running dendrite monolith from official docker repo with version 0.13.1+49d75d3 with PG. We suddenly are experiencing server crashes with the following in our logs:
The server keeps crashing and we are unable to start it anymore.