search

gravitational / teleport

The easiest, and most secure way to access and protect all of your infrastructure.
https://goteleport.com
GNU Affero General Public License v3.0
17.46k stars 1.75k forks source link

`TestIntegrations/Disconnection/verify_that_lost_connections_to_auth_server_terminate_controlled_connections` flakiness #13699

Open ibeckermayer opened 2 years ago

ibeckermayer commented 2 years ago

Failure

Link(s) to logs

Relevant snippet

    integration_test.go:1325: 
            Error Trace:    integration_test.go:1325
                                        integration_test.go:1219
            Error:          Async error
            Test:           TestIntegrations/Disconnection/Test_6
            Messages:       expected EOF, ExitError, or nil, got ssh: could not start shell instead
zmb3 commented 2 years ago

Another: https://console.cloud.google.com/cloud-build/builds/d9dbce3d-5dcd-4b30-92c0-8523daff72af;step=0?project=ci-account

ibeckermayer commented 2 years ago

https://console.cloud.google.com/cloud-build/builds;region=us-west1/5d01dd34-3359-46b7-9613-418702a59808?project=ci-account

nklaassen commented 2 years ago

https://console.cloud.google.com/cloud-build/builds;region=us-west1/07738e83-7f6b-4470-bff8-f838a975b1c9;step=0?project=ci-account

ibeckermayer commented 2 years ago

https://console.cloud.google.com/cloud-build/builds/157bd0d4-7a7f-408f-bee9-c76c7495a732?project=ci-account

ibeckermayer commented 2 years ago

https://console.cloud.google.com/cloud-build/builds/681d892c-8109-4f12-92e3-14c01d419737?project=ci-account

ibeckermayer commented 2 years ago

https://console.cloud.google.com/cloud-build/builds/01e68768-2e6f-4144-a391-8c4297c8022e?project=ci-account

ibeckermayer commented 2 years ago

https://console.cloud.google.com/cloud-build/builds/5f7a3ba4-de6c-4704-aa45-a73513b97fd0?project=ci-account

ibeckermayer commented 2 years ago

https://console.cloud.google.com/cloud-build/builds/03be3195-a6f3-45c6-8554-98c891bcb2e1?project=ci-account

nklaassen commented 1 year ago

Slightly different error here:

{"caller":"srv/monitor.go:218","component":"proxy","level":"debug","message":"Disconnecting client: client is idle for 501.232556ms, exceeded idle timeout of 500ms","timestamp":"2022-11-10T20:11:03Z","trace.fields":{"id":25,"idle":500000000,"local":{"IP":"127.0.0.1","Port":34203,"Zone":""},"login":"ci","remote":{"IP":"127.0.0.1","Port":47902,"Zone":""},"teleportUser":"ci"}}
{"caller":"regular/sshserver.go:1542","component":"proxy","level":"debug","message":"Client 127.0.0.1:47902 disconnected.","timestamp":"2022-11-10T20:11:03Z","trace.fields":{"id":25,"idle":500000000,"local":{"IP":"127.0.0.1","Port":34203,"Zone":""},"login":"ci","remote":{"IP":"127.0.0.1","Port":47902,"Zone":""},"teleportUser":"ci"}}
    integration_test.go:1637: 
            Error Trace:    /workspace/integration/integration_test.go:1637
                                        /workspace/integration/integration_test.go:1531
            Error:          Async error
            Test:           TestIntegrations/Disconnection/Test_1
            Messages:       expected EOF, ExitError, or nil, got failed connecting to node localhost.  instead
GavinFrazar commented 1 year ago

Also got a test_1 failure: https://console.cloud.google.com/cloud-build/builds/a5bcc30a-a645-425c-ad99-38ec20b1ebf7?project=ci-account

integration_test.go:1634: 
            Error Trace:    /workspace/integration/integration_test.go:1634
                                        /workspace/integration/integration_test.go:1531
            Error:          timeout
            Test:           TestIntegrations/Disconnection/Test_1
            Messages:       Nov  9 23:01:14.850 timeout waiting for session to exit: {recordingMode:proxy options:{ForwardAgent:true MaxSessionTTL:0 PortForwarding:<nil> CertificateFormat: ClientIdleTimeout:500000000 DisconnectExpiredCert:false BPF:[] PermitX11Forwarding:false MaxConnections:0 MaxSessions:0 RequestAccess: RequestPrompt: RequireSessionMFA:false Lock: RecordSession:<nil> DesktopClipboard:<nil> CertExtensions:[] MaxKubernetesConnections:0 DesktopDirectorySharing:<nil> CreateHostUser:<nil> PinSourceIP:false SSHFileCopy:<nil> RequireMFAType:0 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} disconnectTimeout:1000000000 concurrentConns:1 sessCtlTimeout:0 postFunc:<nil> verifyError:<nil>}
nklaassen commented 1 year ago

another fail on branch/v10, this one is still around https://github.com/gravitational/teleport/actions/runs/4534349161/jobs/7991191739?pr=23649

zmb3 commented 1 year ago

Renamed the title to account for the changes in #23228

zmb3 commented 1 year ago

Another hit here: https://github.com/gravitational/teleport/actions/runs/6422082651/job/17437858787

    integration_test.go:2042: 
            Error Trace:    /__w/teleport/teleport/integration/integration_test.go:2042
                                        /__w/teleport/teleport/integration/integration_test.go:2155
                                        /__w/teleport/teleport/integration/integration_test.go:2057
            Error:          "[]" should have 1 item(s), but has 0
            Test:           TestIntegrations/Disconnection/verify_that_lost_connections_to_auth_server_terminate_controlled_connections
zmb3 commented 1 year ago

v14: https://github.com/gravitational/teleport/actions/runs/6509499545/job/17681133031?pr=33455

zmb3 commented 12 months ago

https://github.com/gravitational/teleport/actions/runs/6536661748/job/17748840016

    integration_test.go:2266: 
            Error Trace:    /__w/teleport/teleport/integration/integration_test.go:2266
                                        /__w/teleport/teleport/integration/integration_test.go:2392
                                        /__w/teleport/teleport/integration/integration_test.go:2281
            Error:          "[]" should have 1 item(s), but has 0
            Test:           TestIntegrations/Disconnection/verify_that_lost_connections_to_auth_server_terminate_controlled_connection
zmb3 commented 8 months ago

Sadly this has returned: https://github.com/gravitational/teleport/actions/runs/7776666786/job/21204093858?pr=37740

=== Failed
=== FAIL: integration TestIntegrations/Disconnection/verify_that_lost_connections_to_auth_server_terminate_controlled_connections (4.37s)
    integration_test.go:2180: 
            Error Trace:    /__w/teleport/teleport/integration/integration_test.go:2184
                                        /opt/go/src/runtime/asm_amd64.s:1650
            Error:          Received unexpected error:
                            rpc error: code = Canceled desc = context canceled
    integration_test.go:2180: 
            Error Trace:    /__w/teleport/teleport/integration/integration_test.go:2185
                                        /opt/go/src/runtime/asm_amd64.s:1650
            Error:          "[]" should have 1 item(s), but has 0
    integration_test.go:2180: 
            Error Trace:    /__w/teleport/teleport/integration/integration_test.go:2180
                                        /__w/teleport/teleport/integration/integration_test.go:2305
                                        /__w/teleport/teleport/integration/integration_test.go:2201
            Error:          Condition never satisfied
            Test:           TestIntegrations/Disconnection/verify_that_lost_connections_to_auth_server_terminate_controlled_connections
        --- FAIL: TestIntegrations/Disconnection/verify_that_lost_connections_to_auth_server_terminate_controlled_connections (4.37s)

=== FAIL: integration TestIntegrations/Disconnection (31.31s)
    --- FAIL: TestIntegrations/Disconnection (31.31s)

=== FAIL: integration TestIntegrations (741.71s)