`TestOneFailingBackend` flakiness

[zmb3]

Failure

Link(s) to logs

• https://github.com/gravitational/teleport/actions/runs/6890799958/job/18744564420 (branch/v14)

Relevant snippet

=== FAIL: lib/utils TestOneFailingBackend (0.11s)
    loadbalancer_test.go:123: 
            Error Trace:    /__w/teleport/teleport/lib/utils/loadbalancer_test.go:123
            Error:          Expected value not to be nil.
            Test:           TestOneFailingBackend
{"caller":"utils/loadbalancer.go:303","component":"loadbalancer","dest":{"IP":"127.0.0.1","Port":37797,"Zone":""},"level":"warning","message":"connection problem: readfrom tcp 127.0.0.1:37797-\u003e127.0.0.1:35100: use of closed network connection *net.OpError","source":{"IP":"127.0.0.1","Port":35100,"Zone":""},"timestamp":"2023-11-16T12:43:12Z","trace.fields":{"listen":"127.0.0.1:0"}}
{"caller":"utils/loadbalancer.go:240","component":"loadbalancer","level":"warning","message":"Failed to forward connection: readfrom tcp 127.0.0.1:37797-\u003e127.0.0.1:35100: use of closed network connection.","timestamp":"2023-11-16T12:43:12Z","trace.fields":{"listen":"127.0.0.1:0"}}

[ibeckermayer]

I got one here as

=== FAIL: lib/utils TestOneFailingBackend (0.11s)
    loadbalancer_test.go:125: 
            Error Trace:    /__w/teleport/teleport/lib/utils/loadbalancer_test.go:125
            Error:          An error is expected but got nil.
            Test:           TestOneFailingBackend

[ibeckermayer]

Odd one to fail... LoadBalancer hasn't been touched much in the last 7 years it's been around

https://github.com/gravitational/teleport/blame/master/lib/utils/loadbalancer.go#L108