search

sensu / sensu-go

Simple. Scalable. Multi-cloud monitoring.
https://sensu.io
MIT License
1.02k stars 175 forks source link

[Build stability] intermittent failure in TestSilencing e2e #1208

Closed nikkictl closed 5 years ago

nikkictl commented 6 years ago

Investigate intermittent failure in e2e TestSilencing:

--- FAIL: TestSilencing (18.19s)
    assertions.go:239: 

    Error Trace:    silencing_test.go:135

    Error:          Not equal: 

                    expected: 5

                    actual: 6
palourde commented 6 years ago

Just I had a similar failure:

--- FAIL: TestSilencing (16.21s)
    Error Trace:    silencing_test.go:135
    Error:          Not equal: 
                    expected: 3
                    actual: 4
palourde commented 6 years ago

Happened again here

{"component":"schedulerd","env":"default","level":"info","msg":"stopping scheduler","name":"check_silencing","org":"default","time":"2018-04-11T05:47:06Z"}
{"component":"backend","level":"info","msg":"shutting down eventd","time":"2018-04-11T05:47:06Z"}
{"component":"eventd","level":"info","msg":"shutting down eventd","time":"2018-04-11T05:47:06Z"}
{"component":"backend","level":"info","msg":"shutting down pipelined","time":"2018-04-11T05:47:06Z"}
{"component":"backend","level":"info","msg":"shutting down message bus","time":"2018-04-11T05:47:06Z"}
{"component":"etcd","level":"info","msg":"skipped leadership transfer for single member cluster","pkg":"etcdserver","time":"2018-04-11T05:47:06Z"}
WARNING: 2018/04/11 05:47:06 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:33463: connect: connection refused"; Reconnecting to {127.0.0.1:33463 0  <nil>}
WARNING: 2018/04/11 05:47:06 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:33463: connect: connection refused"; Reconnecting to {127.0.0.1:33463 0  <nil>}
WARNING: 2018/04/11 05:47:06 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:33463: connect: connection refused"; Reconnecting to {127.0.0.1:33463 0  <nil>}
--- FAIL: TestSilencing (16.23s)
    Error Trace:    silencing_test.go:135
    Error:          Not equal: 
                    expected: 3
                    actual: 4
palourde commented 6 years ago

And again: https://travis-ci.org/sensu/sensu-go/jobs/374926938#L985

{"component":"agentd","level":"warning","msg":"stopping session","recv error":"Connection closed: websocket: close 1001 (going away): bye","session":"67501225-2948-4072-95e8-938c396c9181","time":"2018-05-04T15:09:17Z"}
WARNING: 2018/05/04 15:09:17 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:42418: connect: connection refused"; Reconnecting to {127.0.0.1:42418 0  <nil>}
WARNING: 2018/05/04 15:09:17 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:42418: connect: connection refused"; Reconnecting to {127.0.0.1:42418 0  <nil>}
WARNING: 2018/05/04 15:09:17 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:42418: connect: connection refused"; Reconnecting to {127.0.0.1:42418 0  <nil>}
--- FAIL: TestSilencing (16.20s)
    Error Trace:    silencing_test.go:133
    Error:          Not equal: 
                    expected: 3
                    actual: 4
nikkictl commented 6 years ago

Hit this locally. We've had a large # of repros. I'm going to bring this to the top of the backlog.

--- FAIL: TestSilencing (16.35s)
        Error Trace:    silencing_test.go:133
    Error:          Not equal: 
                    expected: 3
                    actual: 4
annaplotkin commented 5 years ago

These tests have been removed.