search

kubernetes-sigs / apiserver-network-proxy

Apache License 2.0
371 stars 177 forks source link

Integration tests are flaky and sometimes time out #649

Open carreter opened 3 weeks ago

carreter commented 3 weeks ago

The tests under ./tests seem to take about as long as the configured test timeout, so they will flake pretty often.

Failures look like this:

--- FAIL: TestProxyDial_RequestCancelled_Concurrent_GRPC (61.89s)
    proxy_test.go:399: Timed out waiting for tunnel to close
    proxy_test.go:399: Timed out waiting for tunnel to close
    proxy_test.go:399: Timed out waiting for tunnel to close
    proxy_test.go:399: Timed out waiting for tunnel to close
    proxy_test.go:426: Agent connections leaked: 

        Diff:
        --- metric output does not match expectation; want
        +++ got:
        @@ -2,3 +2,3 @@
         # TYPE konnectivity_network_proxy_agent_open_endpoint_connections gauge
        -konnectivity_network_proxy_agent_open_endpoint_connections 0
        +konnectivity_network_proxy_agent_open_endpoint_connections 4

I think this is related to a behavior I've noticed where the agent will continue attempting to establish connections even after receiving a SIGTERM. I'm having trouble reproducing it right now, will elaborate further in a comment.

carreter commented 3 weeks ago

Here's the issue I was mentioning of connections being attempted even after a SIGTERM:

I0821 17:21:18.808036       1 clientset.go:196] "duplicate server" serverID="97b4725e-ead8-404e-987c-d01666f55d41" serverCount=1 clientsCount=1                                                                                                                                                                                                                                                                                                                                              
I0821 17:21:24.111147       1 client.go:214] "Connect to server" serverID="97b4725e-ead8-404e-987c-d01666f55d41"                                                                                                                                                                                                                                                                                                                                                                             
I0821 17:21:24.111373       1 clientset.go:196] "duplicate server" serverID="97b4725e-ead8-404e-987c-d01666f55d41" serverCount=1 clientsCount=1                                                                                                                                                                                                                                                                                                                                              
I0821 17:21:31.262424       1 server.go:124] "Received first signal" signal="terminated"                                                                                                                                                                                                                                                                                                                                                                                                     
I0821 17:21:31.719962       1 client.go:214] "Connect to server" serverID="97b4725e-ead8-404e-987c-d01666f55d41"                                                                                                                                                                                                                                                                                                                                                                             
I0821 17:21:31.720167       1 clientset.go:196] "duplicate server" serverID="97b4725e-ead8-404e-987c-d01666f55d41" serverCount=1 clientsCount=1                                                                                                                                                                                                                                                                                                                                              
I0821 17:21:42.039194       1 client.go:214] "Connect to server" serverID="97b4725e-ead8-404e-987c-d01666f55d41"                                                                                                                                                                                                                                                                                                                                                                             
I0821 17:21:42.039478       1 clientset.go:196] "duplicate server" serverID="97b4725e-ead8-404e-987c-d01666f55d41" serverCount=1 clientsCount=1                                                                                                                                                                                                                                                                                                                                              
I0821 17:21:52.514095       1 client.go:214] "Connect to server" serverID="97b4725e-ead8-404e-987c-d01666f55d41"                                                                                                                                                                                                                                                                                                                                                                             
I0821 17:21:52.514299       1 clientset.go:196] "duplicate server" serverID="97b4725e-ead8-404e-987c-d01666f55d41" serverCount=1 clientsCount=1                                                                                                                                                                                                                                                                                                                                              
Stream closed EOF for kube-system/konnectivity-agent-668fc5b9dd-ztzzp (konnectivity-agent-container)