search

ovn-org / ovn-kubernetes

A robust Kubernetes networking platform
https://ovn-kubernetes.io/
Apache License 2.0
824 stars 347 forks source link

CI flake: Admin net pol `TestNetworkPolicyV2Conformance` test attempting to use port already in-use #4348

Open martinkennelly opened 5 months ago

martinkennelly commented 5 months ago 
2024-05-09T17:45:20.1404158Z === RUN   TestNetworkPolicyV2Conformance
2024-05-09T17:45:20.1407058Z     network_policy_v2_test.go:31: Configuring environment for network policy V2 API conformance tests
2024-05-09T17:45:20.1421212Z     network_policy_v2_test.go:53: Starting the network policy V2 conformance test suite
2024-05-09T17:45:20.1422439Z     suite.go:125: Test Setup: Applying base manifests
2024-05-09T17:45:20.2616014Z     apply.go:124: Creating network-policy-conformance-gryffindor Namespace
2024-05-09T17:45:20.2686256Z     apply.go:124: Creating network-policy-conformance-slytherin Namespace
2024-05-09T17:45:20.2850881Z     apply.go:124: Creating network-policy-conformance-hufflepuff Namespace
2024-05-09T17:45:20.2958762Z     apply.go:124: Creating network-policy-conformance-ravenclaw Namespace
2024-05-09T17:45:20.3033351Z     apply.go:124: Creating network-policy-conformance-forbidden-forrest Namespace
2024-05-09T17:45:20.3185458Z     apply.go:124: Creating harry-potter StatefulSet
2024-05-09T17:45:20.3249033Z     apply.go:124: Creating draco-malfoy StatefulSet
2024-05-09T17:45:20.3312093Z     apply.go:124: Creating cedric-diggory StatefulSet
2024-05-09T17:45:20.3584068Z     apply.go:124: Creating luna-lovegood StatefulSet
2024-05-09T17:45:20.4050777Z     apply.go:124: Creating centaur StatefulSet
2024-05-09T17:45:20.4482467Z     suite.go:128: Test Setup: Ensuring Namespaces and Pods from base manifests are ready
2024-05-09T17:45:20.4551943Z     helper.go:123: StatefulSet replicas in namespace network-policy-conformance-gryffindor not rolled out yet. 0/2 replicas are available.
2024-05-09T17:45:21.4627860Z     helper.go:123: StatefulSet replicas in namespace network-policy-conformance-gryffindor not rolled out yet. 0/2 replicas are available.
....
....
2024-05-09T17:50:19.4577962Z     helper.go:123: StatefulSet replicas in namespace network-policy-conformance-forbidden-forrest not rolled out yet. 1/2 replicas are available.
2024-05-09T17:50:20.4499365Z     helper.go:120: Error retrieving StatefulSet harry-potter from namespace network-policy-conformance-gryffindor: client rate limiter Wait returned an error: context deadline exceeded
2024-05-09T17:50:20.4501830Z     helper.go:123: StatefulSet replicas in namespace network-policy-conformance-gryffindor not rolled out yet. 0/2 replicas are available.
2024-05-09T17:50:20.4502682Z     suite.go:143: 
2024-05-09T17:50:20.4503713Z            Error Trace:    /home/runner/go/pkg/mod/sigs.k8s.io/network-policy-api@v0.1.5/conformance/utils/kubernetes/helper.go:130
2024-05-09T17:50:20.4505387Z                                        /home/runner/go/pkg/mod/sigs.k8s.io/network-policy-api@v0.1.5/conformance/utils/suite/suite.go:143
2024-05-09T17:50:20.4507231Z                                        /home/runner/go/pkg/mod/sigs.k8s.io/network-policy-api@v0.1.5/conformance/utils/suite/conformance_suite.go:132
2024-05-09T17:50:20.4509264Z                                        /home/runner/work/ovn-kubernetes/ovn-kubernetes/test/conformance/network_policy_v2_test.go:89
2024-05-09T17:50:20.4509992Z            Error:          Received unexpected error:
2024-05-09T17:50:20.4510860Z                            context deadline exceeded
2024-05-09T17:50:20.4511779Z            Test:           TestNetworkPolicyV2Conformance
2024-05-09T17:50:20.4514561Z            Messages:       error waiting for network-policy-conformance-gryffindor, network-policy-conformance-slytherin, network-policy-conformance-hufflepuff, network-policy-conformance-ravenclaw, network-policy-conformance-forbidden-forrest namespaces to be ready

I checked the logs for why one of the replicas pods isnt coming up and it complained about trying to bind to a port already in-use:

$cat 5.log 
2024-05-09T17:48:22.496335577Z stderr F I0509 17:48:22.496230       1 log.go:194] Started HTTP server on port 34358
2024-05-09T17:48:22.496458999Z stderr F I0509 17:48:22.496401       1 log.go:194] Started UDP server on port 172.18.0.3 34345
2024-05-09T17:48:22.4965566Z stderr F I0509 17:48:22.496512       1 log.go:194] listen tcp :34358: bind: address already in use

Seen in https://github.com/ovn-org/ovn-kubernetes/pull/4344 (https://github.com/ovn-org/ovn-kubernetes/actions/runs/9020680306/job/24814796912?pr=4344)

First time seeing it.

martinkennelly commented 5 months ago

cc @tssurya

tssurya commented 5 months ago

thanks @martinkennelly for catching this /assign @tssurya

kyrtapz commented 2 months ago

Another one: https://github.com/ovn-org/ovn-kubernetes/actions/runs/9957563177/job/27511942727?pr=4499:

cat ovn-control-plane/pods/network-policy-conformance-forbidden-forrest_centaur-0_12bd720c-0ef3-4ec8-b7d8-8a866e80ea03/centaur-36364-tcp/5.log                                                                                                                                                                        
2024-07-16T14:17:49.609823679Z stderr F I0716 14:17:49.609731       1 log.go:194] Error from net.Listen(): listen tcp :36364: bind: address already in use