ccl/sqlproxyccl: TestPrivateEndpointsACL failed

[cockroach-teamcity]

ccl/sqlproxyccl.TestPrivateEndpointsACL failed with artifacts on release-23.2.3-rc @ 320e36d59a29f6267f95f7944799f7f995e32292:

=== RUN   TestPrivateEndpointsACL
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/a455c59caed0cf9d0a0298ad88bfe7a2/logTestPrivateEndpointsACL199661117
    test_log_scope.go:81: use -show-logs to present logs inline
    test_server_shim.go:159: automatically injected an external process virtual cluster under test; see comment at top of test_server_shim.go for details.
    proxy_handler_test.go:459: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/a455c59caed0cf9d0a0298ad88bfe7a2/logTestPrivateEndpointsACL199661117
--- FAIL: TestPrivateEndpointsACL (13.46s)
=== RUN   TestPrivateEndpointsACL/private_connection_allowed
    proxy_handler_test.go:388: 
            Error Trace:    github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/proxy_handler_test.go:2829
            Error:          Expected nil, but got: &sqlproxyccl.errWithCode{
                                code:  4,
                                cause: &withstack.withStack{
                                    cause: &errutil.withPrefix{
                                        cause:  &errors.errorString{s:"proxyproto: proxy protocol signature not present"},
                                        prefix: "while receiving startup message",
                                    },
                                    stack: &withstack.stack{0x38ff588, 0x390318a, 0x3909aed, 0x14933ba, 0x488401},
                                },
                            }
            Test:           TestPrivateEndpointsACL/private_connection_allowed
    --- FAIL: TestPrivateEndpointsACL/private_connection_allowed (0.13s)

Parameters:

• TAGS=bazel,gss,deadlock
• stress=true
  Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/sql-foundations @cockroachdb/server

This test on roachdash | Improve this report!

Jira issue: CRDB-36517

[jeffswenson]

Okay, I think I see why this is flaking. The client attempts to dial the sql server and waits for at most three seconds. If three seconds elapses, it closes the connection which would result in the client failing to send the proxy protocol header.

https://github.com/cockroachdb/cockroach/blob/06a3a51cf9a13dc1b8581cb462798b03e6d4724e/pkg/ccl/sqlproxyccl/proxy_handler_test.go#L188

Since this is a stress test failure, it's not particular surprising that the timeout would cause the test to fail, since the machine is intentionally overloaded.

[andy-kimball]

What's the next step on this issue?

[jeffswenson]

Closing as we haven't seen an instance of this in months.