search

gravitational / teleport

The easiest, and most secure way to access and protect all of your infrastructure.
https://goteleport.com
GNU Affero General Public License v3.0
17.43k stars 1.74k forks source link

`TestLockInForce` flakiness #13770

Open ibeckermayer opened 2 years ago

ibeckermayer commented 2 years ago

Failure

Link(s) to logs

Relevant snippet

    sshserver_test.go:392: 
            Error Trace:    sshserver_test.go:392
            Error:          Condition never satisfied
            Test:           TestLockInForce
            Messages:       Timed out waiting for session to finish
zmb3 commented 2 years ago

Another one: https://console.cloud.google.com/cloud-build/builds;region=us-west1/1dfa0b63-a331-4e6d-a118-78f8a15167b4;step=0?project=ci-account

zmb3 commented 2 years ago

More context:

=== CONT  TestLockInForce
=== CONT  TestLockInForce
    sshserver_test.go:412: 
            Error Trace:    sshserver_test.go:412
            Error:          Condition never satisfied
            Test:           TestLockInForce
            Messages:       Timed out waiting for session to finish
{"caller":"backend/buffer.go:312","component":"buffer","level":"debug","message":"Add Watcher(name=8cf7fcf7-76ef-4b7f-b330-803325b24bb6.localhost, prefixes=/locks, capacity=128, size=0).","timestamp":"2022-08-09T23:13:19Z"}
    sshserver_test.go:271: 
            Error Trace:    sshserver_test.go:271
                                        testing.go:1034
                                        testing.go:1203
                                        testing.go:1433
                                        panic.go:500
                                        testing.go:864
                                        sshserver_test.go:412
            Error:          An error is expected but got nil.
            Test:           TestLockInForce

Observed at https://console.cloud.google.com/cloud-build/builds/5075ecbf-b32a-48f0-9af1-108ea18ed8ec;step=0?project=ci-account

zmb3 commented 1 year ago

https://console.cloud.google.com/cloud-build/builds;region=global/32b2f991-c52d-4742-8526-60140b2bd71c;step=0?project=ci-account

zmb3 commented 1 year ago

This one is back: https://github.com/gravitational/teleport/actions/runs/5123805781/jobs/9214837070

@tigrato maybe?

tigrato commented 1 year ago

@zmb3 I don't think this is related to the StaleC changes I made previously.

This test actually creates the lock and does not rely on staleness to enforce it and so it shouldn't be affected by it.

Can it be the case that 1 second is just not enough to propagate the lock?

    // When I let the session idle (with the clock running at approx 10x speed)...
    sessionHasFinished := func() bool {
        f.clock.Advance(1 * time.Second)
        select {
        case <-endCh:
            return true
        default:
            return false
        }
    }
    require.Eventually(t, sessionHasFinished, 1*time.Second, 100*time.Millisecond,
        "Timed out waiting for session to finish")
zmb3 commented 1 year ago

New hit: https://github.com/gravitational/teleport/actions/runs/5588749832/jobs/10215996358

=== CONT  TestLockInForce
    sshserver_test.go:577: 
            Error Trace:    /__w/teleport/teleport/lib/srv/regular/sshserver_test.go:577
            Error:          Condition never satisfied
            Test:           TestLockInForce
            Messages:       Timed out waiting for session to finish
GavinFrazar commented 1 year ago

another: https://github.com/gravitational/teleport/actions/runs/5767199931/job/15636502323

=== Failed
=== FAIL: lib/srv/regular TestLockInForce (9.49s)
    sshserver_test.go:577: 
            Error Trace:    /__w/teleport/teleport/lib/srv/regular/sshserver_test.go:577
            Error:          Condition never satisfied
            Test:           TestLockInForce
            Messages:       Timed out waiting for session to finish
ravicious commented 6 months ago

https://github.com/gravitational/teleport/actions/runs/8263039746/job/22603625685#step:6:1487

ravicious commented 3 months ago

v15 https://github.com/gravitational/teleport/actions/runs/9853654550/job/27204662427#step:6:1583