search

matrix-org / dendrite

Dendrite is a second-generation Matrix homeserver written in Go!
https://matrix-org.github.io/dendrite/
Apache License 2.0
5.75k stars 676 forks source link

Panic write room event duplicate key federatonsender_joined_hosts_event_id_idx #1862

Closed carroarmato0 closed 3 years ago

carroarmato0 commented 3 years ago

Background information

Description

Dendrite crashes in a loop due to the following panic:

time="2021-05-19T16:08:26.405991420Z" level=info msg="Dendrite version 0.3.11" func=github.com/matrix-org/dendrite/setup.NewBaseDendrite file="github.com/matrix-org/dendrite/setup/base.go:110"
time="2021-05-19T16:08:26.406516093Z" level=debug msg="Setting DB connection limits" logrus_error=", can not add field \"MaxOpenConns\"" func=github.com/matrix-org/dendrite/internal/sqlutil.Open file="github.com/matrix-org/dendrite/internal/sqlutil/trace.go:132" dataSourceName="postgres://postgres/dendrite?sslmode=disable"
time="2021-05-19T16:08:26.423064089Z" level=debug msg="Setting DB connection limits" logrus_error=", can not add field \"ConnMaxLifetime\"" func=github.com/matrix-org/dendrite/internal/sqlutil.Open file="github.com/matrix-org/dendrite/internal/sqlutil/trace.go:132" dataSourceName="postgres://postgres/dendrite?sslmode=disable"
time="2021-05-19T16:08:26.430696035Z" level=info msg="Enabled perspective key fetcher" func=github.com/matrix-org/dendrite/signingkeyserver.NewInternalAPI file="github.com/matrix-org/dendrite/signingkeyserver/signingkeyserver.go:103" num_public_keys=2 server_name=matrix.org
time="2021-05-19T16:08:26.438151489Z" level=debug msg="Setting DB connection limits" logrus_error=", can not add field \"ConnMaxLifetime\"" func=github.com/matrix-org/dendrite/internal/sqlutil.Open file="github.com/matrix-org/dendrite/internal/sqlutil/trace.go:132" dataSourceName="postgres://postgres/dendrite?sslmode=disable"
time="2021-05-19T16:08:26.500722413Z" level=debug msg="Updating server ACLs for \"!yomrOFwgFXzmeMAbzX:matrix.org\"" func="github.com/matrix-org/dendrite/roomserver/acls.(*ServerACLs).OnServerACLUpdate" file="github.com/matrix-org/dendrite/roomserver/acls/acls.go:117" allow_ip_literals=false num_allowed=1 num_denied=26
time="2021-05-19T16:08:26.510384401Z" level=debug msg="Updating server ACLs for \"!HiMrQwoxwdnWgrTVly:matrix.org\"" func="github.com/matrix-org/dendrite/roomserver/acls.(*ServerACLs).OnServerACLUpdate" file="github.com/matrix-org/dendrite/roomserver/acls/acls.go:117" allow_ip_literals=false num_allowed=1 num_denied=31
time="2021-05-19T16:08:26.510536054Z" level=debug msg="Setting DB connection limits" logrus_error=", can not add field \"MaxOpenConns\"" func=github.com/matrix-org/dendrite/internal/sqlutil.Open file="github.com/matrix-org/dendrite/internal/sqlutil/trace.go:132" dataSourceName="postgres://postgres/dendrite?sslmode=disable"
time="2021-05-19T16:08:26.523793077Z" level=debug msg="Setting DB connection limits" logrus_error=", can not add field \"ConnMaxLifetime\"" func=github.com/matrix-org/dendrite/internal/sqlutil.Open file="github.com/matrix-org/dendrite/internal/sqlutil/trace.go:132" dataSourceName="postgres://postgres/dendrite?sslmode=disable"
time="2021-05-19T16:08:26.527574540Z" level=panic msg="roomserver output log: write room event failure" func="github.com/matrix-org/dendrite/federationsender/consumers.(*OutputRoomEventConsumer).onMessage" file="github.com/matrix-org/dendrite/federationsender/consumers/roomserver.go:113" add="[$D9qdG_A4j8Sabk-ceXioc4PzjZIBWaycDy-h2IHJ-4Q]" del="[$OsLr5BDhAEEjn907uD9aRzaWUmh6yREjI_wpR7ZPo7Q]" error="pq: duplicate key value violates unique constraint \"federatonsender_joined_hosts_event_id_idx\"" event="{\"auth_events\":[\"$OsLr5BDhAEEjn907uD9aRzaWUmh6yREjI_wpR7ZPo7Q\",\"$1ZgLuJEdH_EqUF2LMLGm33v3zJs4Yo4yP28wM6OREw8\",\"$VCSywbpDMzmScq-_uXkLCh1i68Gn4xy1jH0oq9tDZXM\",\"$cL3H1pLIlwCIulZJM1UXDdGMGOgiASKuL66NNeu5mgc\"],\"content\":{\"avatar_url\":\"mxc://matrix.org/AmTHuRJCxeuFgddwCnvrHfII\",\"displayname\":\"Follpvosten\",\"membership\":\"join\"},\"depth\":12473,\"hashes\":{\"sha256\":\"BmveQ4pTUO4KyiD8v0S1Rqc54ZOma/86YItLCB61aco\"},\"origin\":\"matrix.org\",\"origin_server_ts\":1616838462617,\"prev_events\":[\"$m03wzUnhzaD36UBJPOzh_p1J_UJUoeRFGKgPKpWgJb8\"],\"prev_state\":[],\"room_id\":\"!gJYEKNllaubNlNkFIj:jupiterbroadcasting.com\",\"sender\":\"@Follpvosten:matrix.org\",\"signatures\":{\"matrix.org\":{\"ed25519:a_RXGa\":\"r1tHnOR/fmJHAKy8gi0SSA460oZuUktun83O42GMk8Y+1PC5B6pqjAdzu0NXPfESpJhOw2tjePRtM3s6N4QTDg\"}},\"state_key\":\"@Follpvosten:matrix.org\",\"type\":\"m.room.member\",\"unsigned\":{\"age_ts\":1616838462617,\"replaces_state\":\"$OsLr5BDhAEEjn907uD9aRzaWUmh6yREjI_wpR7ZPo7Q\"}}" event_id="$D9qdG_A4j8Sabk-ceXioc4PzjZIBWaycDy-h2IHJ-4Q"

Steps to reproduce

Not sure how to reproduce it, it suddenly happened.

kegsay commented 3 years ago

This was fixed in https://github.com/matrix-org/dendrite/pull/1824 which is unreleased due to issues with database migrations on master.