search

googleforgames / agones

Dedicated Game Server Hosting and Scaling for Multiplayer Games on Kubernetes
https://agones.dev
Apache License 2.0
6.09k stars 810 forks source link

Go unit test timeout after 10m #1672

Closed markmandel closed 4 years ago

markmandel commented 4 years ago

What happened:

Test times out after 10m. Looks like some kind of deadlock.

What you expected to happen:

Test to pass.

How to reproduce it (as minimally and precisely as possible):

Currently only happening occasionally in CI. Trying to work out how to reproduce it locally.

Anything else we need to know?:

I wonder if having SDKWatchSendOnExecute feature flag enabled will cause one of the previous either local or sdkserver tests to hang. That's my current theory. Seems to be focused on tests of either of those areas fro the stacktrackes though.

Environment:

Instances in tests: https://github.com/googleforgames/agones/pull/1663#issuecomment-652153122 https://github.com/googleforgames/agones/pull/1662#issuecomment-652041201

Another grep of Test from a failed test:

agones.dev/agones/pkg/sdkserver.TestLocalSDKServerPlayerCapacity
agones.dev/agones/pkg/sdkserver.TestLocalSDKServerPlayerCapacity.func1(0xc0008e6410, 0xc0000d4040, 0xc0004b8c60)
agones.dev/agones/pkg/sdkserver.TestLocalSDKServerPlayerConnectAndDisconnect.func2
agones.dev/agones/pkg/sdkserver.TestLocalSDKServerPlayerConnectAndDisconnect.func2
agones.dev/agones/pkg/sdkserver.TestLocalSDKServerPlayerConnectAndDisconnect.func2
agones.dev/agones/pkg/sdkserver.TestLocalSDKServerPlayerConnectAndDisconnect.func2
agones.dev/agones/pkg/sdkserver.TestLocalSDKServerPlayerConnectAndDisconnect.func2
agones.dev/agones/pkg/sdkserver.TestLocalSDKServerPlayerConnectAndDisconnect.func2
agones.dev/agones/pkg/sdkserver.TestLocalSDKServerPlayerConnectAndDisconnect.func2.1(0xc0003c9860, 0xc0000100f0, 0xc0003ee7e0)
agones.dev/agones/pkg/sdkserver.TestLocalSDKServerPlayerConnectAndDisconnect.func2.1(0xc0003eb790, 0xc0000100c0, 0xc0006df320)
agones.dev/agones/pkg/sdkserver.TestLocalSDKServerPlayerConnectAndDisconnect.func2.1(0xc0003ebd40, 0xc0000100d8, 0xc0006dfb00)
agones.dev/agones/pkg/sdkserver.TestLocalSDKServerPlayerConnectAndDisconnect.func2.1(0xc00097c680, 0xc00018e098, 0xc0001d0b40)
agones.dev/agones/pkg/sdkserver.TestLocalSDKServerPlayerConnectAndDisconnect.func2.1(0xc000120b60, 0xc000010070, 0xc000470000)
agones.dev/agones/pkg/sdkserver.TestLocalSDKServerPlayerConnectAndDisconnect.func2.1(0xc000367520, 0xc000302028, 0xc000410000)
agones.dev/agones/pkg/sdkserver.TestLocalSDKServerWatchGameServer
agones.dev/agones/pkg/sdkserver.TestLocalSDKServerWatchGameServer.func1(0xc0000c2000, 0xc0004b4700, 0xc0000d40c0, 0xc0004b8b40)
agones.dev/agones/pkg/sdkserver.TestSDKServerUpdateEventHandler(0xc000460360)
agones.dev/agones/pkg/sdkserver.TestSDKServerWatchGameServer(0xc000460000)
    /go/src/agones.dev/agones/pkg/sdkserver/localsdk_test.go:264 +0x2c6
    /go/src/agones.dev/agones/pkg/sdkserver/localsdk_test.go:265 +0x5d
    /go/src/agones.dev/agones/pkg/sdkserver/localsdk_test.go:308 +0x334
    /go/src/agones.dev/agones/pkg/sdkserver/localsdk_test.go:309 +0x92
    /go/src/agones.dev/agones/pkg/sdkserver/localsdk_test.go:418 +0x241
    /go/src/agones.dev/agones/pkg/sdkserver/localsdk_test.go:418 +0x241
    /go/src/agones.dev/agones/pkg/sdkserver/localsdk_test.go:418 +0x241
    /go/src/agones.dev/agones/pkg/sdkserver/localsdk_test.go:418 +0x241
    /go/src/agones.dev/agones/pkg/sdkserver/localsdk_test.go:418 +0x241
    /go/src/agones.dev/agones/pkg/sdkserver/localsdk_test.go:418 +0x241
    /go/src/agones.dev/agones/pkg/sdkserver/localsdk_test.go:419 +0x92
    /go/src/agones.dev/agones/pkg/sdkserver/localsdk_test.go:419 +0x92
    /go/src/agones.dev/agones/pkg/sdkserver/localsdk_test.go:419 +0x92
    /go/src/agones.dev/agones/pkg/sdkserver/localsdk_test.go:419 +0x92
    /go/src/agones.dev/agones/pkg/sdkserver/localsdk_test.go:419 +0x92
    /go/src/agones.dev/agones/pkg/sdkserver/localsdk_test.go:419 +0x92
    /go/src/agones.dev/agones/pkg/sdkserver/sdkserver_test.go:658 +0x278
    /go/src/agones.dev/agones/pkg/sdkserver/sdkserver_test.go:779 +0x59e
    /go/src/agones.dev/agones/pkg/sdkserver/sdkserver_test.go:1251 +0x86
    /go/src/agones.dev/agones/pkg/sdkserver/sdkserver_test.go:1251 +0x86
    /go/src/agones.dev/agones/pkg/sdkserver/sdkserver_test.go:1252 +0x78
    /go/src/agones.dev/agones/pkg/sdkserver/sdkserver_test.go:1252 +0x78
    /go/src/agones.dev/agones/pkg/sdkserver/sdkserver_test.go:1259 +0x6b
    /go/src/agones.dev/agones/pkg/sdkserver/sdkserver_test.go:1259 +0x6b
    /go/src/agones.dev/agones/pkg/sdkserver/sdkserver_test.go:1260 +0x93
    /go/src/agones.dev/agones/pkg/sdkserver/sdkserver_test.go:1260 +0x93
aLekSer commented 4 years ago

I assume that ParseFeatures() function could lead to an error state, if 2 test run in parallel without runtime.FeatureTestMutex.Lock() and one test enabled one feature and other enable another feature Gate. func ParseFeatures(queryString string)

aLekSer commented 4 years ago

I think I am ready with a fix. An easy way to reproduce this locally is on current master:

cd ./pkg/sdkserver
go test -run SDKServer

waitConnectedStreamCount() uses streamMutex as well as WatchGameServer() new feature.