`TestBenchmarkPostgres` flakiness

[ravicious]

Failure

Link(s) to logs

• https://github.com/gravitational/teleport/actions/runs/7017177125/job/19089939183#step:6:1487

Relevant snippet

=== FAIL: tool/tsh/common TestBenchmarkPostgres (5.58s)

* Requests originated: 1
* Requests failed: 1
* Last error: ERROR 1105 (HY000): lookup external-mysql on 127.0.0.11:53: server misbehaving

[zmb3]

Another failure here: https://github.com/gravitational/teleport/actions/runs/7619849290/job/20753624681

Looks slightly different:

✖  tool/tsh/common (5m35.847s) (coverage: 56.7% of statements) (-test.shuffle 1705974534692400753)

=== Failed
=== FAIL: tool/tsh/common TestBenchmarkPostgres (6.23s)
2024-01-23T01:53:27Z INFO [CLIENT]    No teleport login given. defaulting to root client/api.go:1086
    testing.go:1225: TempDir RemoveAll cleanup: unlinkat /tmp/TestBenchmarkPostgres1389941832/001/log/records/multi/300f6dd7-2803-4b6e-8ae8-b51cd70a5c58/47810f29-f8ab-40c8-a340-1bab67614611: directory not empty

Approval received, getting updated certificates...

> Profile URL:        https://127.0.0.1:24875/
  Logged in as:       alice@example.com
  Active requests:    4a0751f4-0746-41fd-bb9a-c241e77081bb
  Cluster:            leafcluster
  Roles:              access, requester
  Kubernetes:         disabled
  Valid until:        2024-01-23 02:53:31 +0000 UTC [valid for 1h0m0s]
  Extensions:         permit-agent-forwarding, permit-port-forwarding, permit-pty, private-key-policy

2024-01-23T01:53:31Z DEBU [KEYSTORE]  Adding known host localhost with proxy 127.0.0.1 client/trusted_certs_store.go:395
make: *** [Makefile:769: test-go-tsh] Error 1
make: *** Waiting for unfinished jobs....
> Profile URL:        https://127.0.0.1:24859/
  Logged in as:       alice@example.com
  Cluster:            localhost
  Roles:              access
  Kubernetes:         disabled
  Valid until:        2024-01-23 02:53:31 +0000 UTC [valid for 1h0m0s]
  Extensions:         permit-agent-forwarding, permit-port-forwarding, permit-pty, private-key-policy

Did you know? Teleport Connect offers the power of tsh in a desktop app.
Learn more at https://goteleport.com/docs/connect-your-client/teleport-connect/

--- PASS: TestBenchmarkPostgres/direct_connection (1.02s)
ERROR: must provide and database name and user

@gabrielcorado can you please take a look?

[zmb3]

https://github.com/gravitational/teleport/actions/runs/10151175274/job/28069898144

@gabrielcorado any updates on this one?

[zmb3]

v14 hit: https://github.com/gravitational/teleport/actions/runs/10710617570/job/29697706879

Looks like the tempdir cleanup problem

=== FAIL: tool/tsh/common TestBenchmarkPostgres (5.66s)
2024/09/04 22:50:30 WARN cert authority watch loop for client TLS config generator failed error="ca watcher exited with: event fanout system reset"
2024/09/04 22:50:30 WARN cert authority watch loop for client TLS config generator failed error="ca watcher exited with: event fanout system reset"
2024/09/04 22:50:30 WARN cert authority watch loop for client TLS config generator failed error="ca watcher exited with: event fanout system reset"
2024/09/04 22:50:30 WARN cert authority watch loop for client TLS config generator failed error="ca watcher exited with: event fanout system reset"
    testing.go:1231: TempDir RemoveAll cleanup: unlinkat /tmp/TestBenchmarkPostgres3401048672/001/log/records/multi/0d8afbb1-370a-4fe7-b167-a7abbda6304a/f1431edc-4c86-47f5-a330-49b5df28a34f: directory not empty