search

matrix-org / dendrite

Dendrite is a second-generation Matrix homeserver written in Go!
https://matrix-org.github.io/dendrite/
Apache License 2.0
5.66k stars 664 forks source link

Assertions needed to catch zero NIDs #955

Open neilalexander opened 4 years ago

neilalexander commented 4 years ago

All NIDs (event, event type, state block, state snapshot, event state key, room, etc) always start from 1, but there are sometimes cases where bugs cause us to request 0 NIDs. We should put some assertion to catch these bugs as they happen.

neilalexander commented 4 years ago

Whilst running the sytests for #954 against SQLite, I am noticing that the monolith logs are showing a lot of failed AddPrevEventsToEvents calls when building events.

It appears to be because QueryLatestEventsAndState calls DB.LatestEventIDs, which in turn calls selectLatestEventNIDs and then the selectLatestEventNIDsSQL query returns zero NIDs.

@Kegsay In SQLite these are wrapped in a transaction in LatestEventIDs—in Postgres they are not. Any thoughts as to what's going on here?

neilalexander commented 4 years ago

Actually it looks like this is also happening sometimes in Postgres mode, just in a different place, so that's fun.

Request panicked!
goroutine 6979 [running]:
runtime/debug.Stack(0xc000104d20, 0xc0005df590, 0xc000104e00)
    /goroot/src/runtime/debug/stack.go:24 +0x9d
github.com/matrix-org/util.Protect.func1.1(0xc00057c600, 0x102ce20, 0xc0003a8540)
    /gopath/pkg/mod/github.com/matrix-org/util@v0.0.0-20190711121626-527ce5ddefc7/json.go:98 +0x15d
panic(0xd64a40, 0x1007480)
    /goroot/src/runtime/panic.go:679 +0x1b2
github.com/matrix-org/dendrite/roomserver/query.(*RoomserverQueryAPI).QueryLatestEventsAndState(0xc000035f40, 0x10306e0, 0xc000445140, 0xc000210390, 0xc00018ce00, 0x0, 0x0)
    /src/roomserver/query/query.go:135 +0x6ee
github.com/matrix-org/dendrite/common.AddPrevEventsToEvent(0x10306e0, 0xc000445140, 0xc0005e0178, 0x103d580, 0xc000035f40, 0xc00018ce00, 0x15df3e0, 0xc000182180)
    /src/common/events.go:81 +0x2a1
github.com/matrix-org/dendrite/common.BuildEvent(0x10306e0, 0xc000445140, 0xc0005e0178, 0xc000180000, 0xbf9b47247a83982c, 0x3fe1d3ac7d, 0x15dfd60, 0x103d580, 0xc000035f40, 0xc00018ce00, ...)
    /src/common/events.go:48 +0x95
github.com/matrix-org/dendrite/clientapi/routing.joinRoomReq.joinRoomUsingServers(0xc00055f300, 0xbf9b47247a83982c, 0x3fe1d3ac7d, 0x15dfd60, 0xc0004451a0, 0xc0002271d0, 0x28, 0xc000180000, 0xc00018eee0, 0xc000208ae0, ...)
    /src/clientapi/routing/joinroom.go:243 +0x178
github.com/matrix-org/dendrite/clientapi/routing.joinRoomReq.joinRoomByID(0xc00055f300, 0xbf9b47247a83982c, 0x3fe1d3ac7d, 0x15dfd60, 0xc0004451a0, 0xc0002271d0, 0x28, 0xc000180000, 0xc00018eee0, 0xc000208ae0, ...)
    /src/clientapi/routing/joinroom.go:156 +0x91a
github.com/matrix-org/dendrite/clientapi/routing.JoinRoomByIDOrAlias(0xc00055f300, 0xc0002e6460, 0xc00032e41d, 0x20, 0xc000180000, 0xc00018eee0, 0xc000208ae0, 0x103d580, 0xc000035f40, 0x1032420, ...)
    /src/clientapi/routing/joinroom.go:85 +0xd2b
github.com/matrix-org/dendrite/clientapi/routing.Setup.func3(0xc00055f300, 0xc0002e6460, 0xd62e00, 0x1006a10, 0xeb7c60, 0xc0001b71f0)
    /src/clientapi/routing/routing.go:103 +0x261
github.com/matrix-org/dendrite/common.MakeAuthAPI.func1(0xc00057c700, 0x0, 0x0, 0x0, 0x0)
    /src/common/httpapi.go:33 +0x325
github.com/matrix-org/util.(*jsonRequestHandlerWrapper).OnIncomingRequest(0xc000010218, 0xc00057c700, 0xc00038da10, 0x30, 0x30, 0xe441e0)
    /gopath/pkg/mod/github.com/matrix-org/util@v0.0.0-20190711121626-527ce5ddefc7/json.go:79 +0x33
github.com/matrix-org/util.MakeJSONAPI.func1(0x102ce20, 0xc0003a8540, 0xc00057c700)
    /gopath/pkg/mod/github.com/matrix-org/util@v0.0.0-20190711121626-527ce5ddefc7/json.go:139 +0x97
github.com/matrix-org/util.Protect.func1(0x102ce20, 0xc0003a8540, 0xc00057c600)
    /gopath/pkg/mod/github.com/matrix-org/util@v0.0.0-20190711121626-527ce5ddefc7/json.go:103 +0xa1
net/http.HandlerFunc.ServeHTTP(...)
    /goroot/src/net/http/server.go:2007
github.com/matrix-org/dendrite/common.MakeExternalAPI.func1(0x102ce20, 0xc0003a8540, 0xc00057c500)
    /src/common/httpapi.go:46 +0x203
net/http.HandlerFunc.ServeHTTP(0xc000264d20, 0x102ce20, 0xc0003a8540, 0xc00057c500)
    /goroot/src/net/http/server.go:2007 +0x44
github.com/gorilla/mux.(*Router).ServeHTTP(0xc000178300, 0x102ce20, 0xc0003a8540, 0xc00057c200)
    /gopath/pkg/mod/github.com/gorilla/mux@v1.7.3/mux.go:212 +0xe2
github.com/matrix-org/dendrite/common.WrapHandlerInCORS.func1(0x102ce20, 0xc0003a8540, 0xc00057c200)
    /src/common/httpapi.go:147 +0x340
net/http.HandlerFunc.ServeHTTP(0xc0003d69c0, 0x102ce20, 0xc0003a8540, 0xc00057c200)
    /goroot/src/net/http/server.go:2007 +0x44
net/http.(*ServeMux).ServeHTTP(0x15dfc20, 0x102ce20, 0xc0003a8540, 0xc00057c200)
    /goroot/src/net/http/server.go:2387 +0x1bd
net/http.serverHandler.ServeHTTP(0xc00045e000, 0x102ce20, 0xc0003a8540, 0xc00057c200)
    /goroot/src/net/http/server.go:2802 +0xa4
net/http.(*conn).serve(0xc000253360, 0x1030620, 0xc0000767c0)
    /goroot/src/net/http/server.go:1890 +0x875
created by net/http.(*Server).Serve
    /goroot/src/net/http/server.go:2928 +0x384
" func=github.com/matrix-org/util.Protect.func1.1 file="/gopath/pkg/mod/github.com/matrix-org/util@v0.0.0-20190711121626-527ce5ddefc7/json.go:97
neilalexander commented 4 years ago

So roomserver_rooms in my local Dendrite SQLite database appears to have quite a few rows of zero NIDs in the last_event_sent_nid and state_snapshot_nid columns. I suspect they are not being updated properly as my Postgres one doesn't have this.