search

Kong / kubernetes-ingress-controller

:gorilla: Kong for Kubernetes: The official Ingress Controller for Kubernetes.
https://docs.konghq.com/kubernetes-ingress-controller/
Apache License 2.0
2.21k stars 590 forks source link

Integration test `TestTCPRouteEssentials` fails #5766

Open pmalek opened 6 months ago

pmalek commented 6 months ago

Problem statement

Recently changed TestTCPRouteEssentials failed on an unrelated PR CI run:

=== RUN   TestTCPRouteEssentials/essentials/verifying_behavior_when_Gateway_is_deleted_and_recreated
    tcproute_test.go:240: deleting the GatewayClass
    tcproute_test.go:243: verifying that the Gateway gets unlinked from the route via status
    tcproute_test.go:247: verifying that the data-plane configuration from the TCPRoute gets dropped with the GatewayClass now removed
    tcproute_test.go:250: putting the GatewayClass back
    tcproute_test.go:254: verifying that the Gateway gets linked to the route via status
    tcproute_test.go:258: verifying that creating the GatewayClass again triggers reconciliation of TCPRoutes and the route becomes available again
I0328 11:40:43.461860   15404 utils.go:29] "TCPRoute has been configured on the data-plane" logger="controllers.TCPRoute" GatewayV1Alpha2TCPRoute="ns-cff-ff3/b27dc3c2-311e-4f18-8d74-63d1f497402e" namespace="ns-cff-ff3" name="b27dc3c2-311e-4f18-8d74-63d1f497402e"
I0328 11:40:45.486142   15404 utils.go:29] "TCPRoute has been configured on the data-plane" logger="controllers.TCPRoute" GatewayV1Alpha2TCPRoute="ns-cff-ff3/b27dc3c2-311e-4f18-8d74-63d1f497402e" namespace="ns-cff-ff3" name="b27dc3c2-311e-4f18-8d74-63d1f497402e"
    tcproute_test.go:259: 
            Error Trace:    /home/runner/work/kubernetes-ingress-controller/kubernetes-ingress-controller/test/integration/isolated/assertion_helpers.go:43
                                        /opt/hostedtoolcache/go/1.21.6/x64/src/runtime/asm_amd64.s:1650
            Error:          Received unexpected error:
                            read tcp 172.18.0.1:36566->172.18.128.3:8888: read: connection reset by peer
    tcproute_test.go:259: 
            Error Trace:    /home/runner/work/kubernetes-ingress-controller/kubernetes-ingress-controller/test/integration/isolated/assertion_helpers.go:42
                                        /home/runner/work/kubernetes-ingress-controller/kubernetes-ingress-controller/test/integration/isolated/tcproute_test.go:259
                                        /home/runner/go/pkg/mod/sigs.k8s.io/e2e-framework@v0.3.1-0.20231113122213-262cac32d35e/pkg/env/env.go:435
                                        /home/runner/go/pkg/mod/sigs.k8s.io/e2e-framework@v0.3.1-0.20231113122213-262cac32d35e/pkg/env/env.go:468
            Error:          Condition never satisfied
            Test:           TestTCPRouteEssentials/essentials/verifying_behavior_when_Gateway_is_deleted_and_recreated
    tcproute_test.go:261: deleting the Gateway
    tcproute_test.go:264: verifying that the Gateway gets unlinked from the route via status
    tcproute_test.go:268: verifying that the data-plane configuration from the TCPRoute gets dropped with the Gateway now removed
    tcproute_test.go:271: putting the Gateway back
    tcproute_test.go:282: verifying that the Gateway gets linked to the route via status
I0328 11:43:43.978469   15404 utils.go:29] "Marking gateway as accepted" logger="controllers.Gateway" GatewayV1Gateway="ns-cff-ff3/8f76848a-cb1c-4b2e-b85c-10f2138d8da9" namespace="ns-cff-ff3" name="8f76848a-cb1c-4b2e-b85c-10f2138d8da9"
I0328 11:43:44.037657   15404 utils.go:29] "TCPRoute has been configured on the data-plane" logger="controllers.TCPRoute" GatewayV1Alpha2TCPRoute="ns-cff-ff3/b27dc3c2-311e-4f18-8d74-63d1f497402e" namespace="ns-cff-ff3" name="b27dc3c2-311e-4f18-8d74-63d1f497402e"
I0328 11:43:44.081030   15404 utils.go:29] "TCPRoute has been configured on the data-plane" logger="controllers.TCPRoute" GatewayV1Alpha2TCPRoute="ns-cff-ff3/b27dc3c2-311e-4f18-8d74-63d1f497402e" namespace="ns-cff-ff3" name="b27dc3c2-311e-4f18-8d74-63d1f497402e"
    tcproute_test.go:286: verifying that creating the Gateway again triggers reconciliation of TCPRoutes and the route becomes available again
    tcproute_test.go:287: 
            Error Trace:    /home/runner/work/kubernetes-ingress-controller/kubernetes-ingress-controller/test/integration/isolated/assertion_helpers.go:43
                                        /opt/hostedtoolcache/go/1.21.6/x64/src/runtime/asm_amd64.s:1650
            Error:          Received unexpected error:
                            read tcp 172.18.0.1:38584->172.18.128.3:8888: read: connection reset by peer
    tcproute_test.go:287: 
            Error Trace:    /home/runner/work/kubernetes-ingress-controller/kubernetes-ingress-controller/test/integration/isolated/assertion_helpers.go:42
                                        /home/runner/work/kubernetes-ingress-controller/kubernetes-ingress-controller/test/integration/isolated/tcproute_test.go:287
                                        /home/runner/go/pkg/mod/sigs.k8s.io/e2e-framework@v0.3.1-0.20231113122213-262cac32d35e/pkg/env/env.go:435
                                        /home/runner/go/pkg/mod/sigs.k8s.io/e2e-framework@v0.3.1-0.20231113122213-262cac32d35e/pkg/env/env.go:468
            Error:          Condition never satisfied
            Test:           TestTCPRouteEssentials/essentials/verifying_behavior_when_Gateway_is_deleted_and_recreated
    tcproute_test.go:289: deleting both GatewayClass and Gateway rapidly
    tcproute_test.go:293: verifying that the Gateway gets unlinked from the route via status
E0328 11:46:44.240285   15404 gateway_controller.go:197] "Could not retrieve gatewayclass" err="GatewayClass.gateway.networking.k8s.io \"a904a064-64fd-4369-bcf1-61c16c63920b\" not found" logger="controllers.Gateway" gatewayclass="a904a064-64fd-4369-bcf1-61c16c63920b"
    tcproute_test.go:297: verifying that the data-plane configuration from the TCPRoute does not get orphaned with the GatewayClass and Gateway gone
    tcproute_test.go:300: putting the GatewayClass back
    tcproute_test.go:304: putting the Gateway back
    tcproute_test.go:315: verifying that the Gateway gets linked to the route via status
I0328 11:46:44.772329   15404 utils.go:29] "Marking gateway as accepted" logger="controllers.Gateway" GatewayV1Gateway="ns-cff-ff3/8f76848a-cb1c-4b2e-b85c-10f2138d8da9" namespace="ns-cff-ff3" name="8f76848a-cb1c-4b2e-b85c-10f2138d8da9"
I0328 11:46:44.833257   15404 utils.go:29] "TCPRoute has been configured on the data-plane" logger="controllers.TCPRoute" GatewayV1Alpha2TCPRoute="ns-cff-ff3/b27dc3c2-311e-4f18-8d74-63d1f497402e" namespace="ns-cff-ff3" name="b27dc3c2-311e-4f18-8d74-63d1f497402e"
I0328 11:46:44.949636   15404 utils.go:29] "TCPRoute has been configured on the data-plane" logger="controllers.TCPRoute" GatewayV1Alpha2TCPRoute="ns-cff-ff3/b27dc3c2-311e-4f18-8d74-63d1f497402e" namespace="ns-cff-ff3" name="b27dc3c2-311e-4f18-8d74-63d1f497402e"
    tcproute_test.go:319: verifying that creating the Gateway again triggers reconciliation of TCPRoutes and the route becomes available again
    tcproute_test.go:320: 
            Error Trace:    /home/runner/work/kubernetes-ingress-controller/kubernetes-ingress-controller/test/integration/isolated/assertion_helpers.go:43
                                        /opt/hostedtoolcache/go/1.21.6/x64/src/runtime/asm_amd64.s:1650
            Error:          Received unexpected error:
                            read tcp 172.18.0.1:58578->172.18.128.3:8888: read: connection reset by peer
    tcproute_test.go:320: 
            Error Trace:    /home/runner/work/kubernetes-ingress-controller/kubernetes-ingress-controller/test/integration/isolated/assertion_helpers.go:42
                                        /home/runner/work/kubernetes-ingress-controller/kubernetes-ingress-controller/test/integration/isolated/tcproute_test.go:320
                                        /home/runner/go/pkg/mod/sigs.k8s.io/e2e-framework@v0.3.1-0.20231113122213-262cac32d35e/pkg/env/env.go:435
                                        /home/runner/go/pkg/mod/sigs.k8s.io/e2e-framework@v0.3.1-0.20231113122213-262cac32d35e/pkg/env/env.go:468
            Error:          Condition never satisfied
            Test:           TestTCPRouteEssentials/essentials/verifying_behavior_when_Gateway_is_deleted_and_recreated

https://github.com/Kong/kubernetes-ingress-controller/actions/runs/8466614681/job/23195804013?pr=5765

programmer04 commented 5 months ago

The above seems like a very random GH Action hiccup. But there is a more common flake for this test case 

=== RUN   TestTCPRouteEssentials/essentials/verifying_behavior_when_TCPRoute_is_modified
    tcproute_test.go:187: removing the parentrefs from the TCPRoute
    tcproute_test.go:202: verifying that the Gateway gets unlinked from the route via status
    tcproute_test.go:206: verifying that the tcpecho is no longer responding
    tcproute_test.go:216: putting the parentRefs back
    tcproute_test.go:225: verifying that the Gateway gets linked to the route via status
    tcproute_test.go:227: 
            Error Trace:    /home/runner/work/kubernetes-ingress-controller/kubernetes-ingress-controller/test/integration/isolated/tcproute_test.go:227
                                        /home/runner/go/pkg/mod/sigs.k8s.io/e2e-framework@v0.3.1-0.20231113122213-262cac32d35e/pkg/env/env.go:435
                                        /home/runner/go/pkg/mod/sigs.k8s.io/e2e-framework@v0.3.1-0.20231113122213-262cac32d35e/pkg/env/env.go:468
            Error:          Condition never satisfied
            Test:           TestTCPRouteEssentials/essentials/verifying_behavior_when_TCPRoute_is_modified
    tcproute_test.go:229: verifying that putting the parentRefs back results in the routes becoming available again
    tcproute_test.go:210: no longer responding check failure state: eof=false, reset=false, err=<nil>

example run