search

justtrackio / gosoline

Gosoline is our framework which fuels all of our Golang applications
MIT License
93 stars 50 forks source link

Handle closed network connections by retrying #1166

Open ajscholl opened 1 week ago

ajscholl commented 1 week ago

When running integration tests, they sometimes fail with the following log:

12:32:37.802 handler info    looking for ddb table xxx
    testcase_application.go:79: 
            Error Trace:    /go/pkg/mod/github.com/justtrackio/gosoline@v0.26.2/pkg/test/suite/testcase_application.go:79
                                        /go/pkg/mod/github.com/justtrackio/gosoline@v0.26.2/pkg/test/suite/testcase_httpserver.go:107
                                        /go/pkg/mod/github.com/justtrackio/gosoline@v0.26.2/pkg/test/suite/run.go:166
            Error:          failed to create application under test
            Test:           TestGatewayTestSuite/TestXXX
            Messages:       can not build kernel factory: can not build module api: could not define routes: can not create xxx: can not create xxx: can not create xxx repo: can not create ddb repository: could not create ddb table xxx: can not check if the table already exists: can not describe table: operation error DynamoDB: DescribeTable, https response error StatusCode: 200, RequestID: af412002-2658-48a3-8915-506c3d64ec95, deserialization failed, failed to decode response body, read tcp 172.17.0.1:40560->172.17.0.1:32894: use of closed network connection

We should check for the "use of closed network connection" error and handle it like any other error which we just retry - it should be fine retrying here (or, if the container running dynamodb already exited, show a message about dynamodb being not (no longer) reachable).

Right now this sometimes randomly fails the automated tests in the phase loading the fixtures, i.e., no code under test is actually running yet, which makes for a frustrating developer experience.