search

ovn-org / ovn-kubernetes

A robust Kubernetes networking platform
https://ovn-kubernetes.io/
Apache License 2.0
809 stars 338 forks source link

e2e test flakiness: e2e multiple external gateway validation #2514

Closed tssurya closed 1 month ago

tssurya commented 2 years ago

Sample Failures:

https://github.com/ovn-org/ovn-kubernetes/pull/2510/checks?check_run_id=3718190674#step:11:229 https://github.com/ovn-org/ovn-kubernetes/pull/2513/checks?check_run_id=3704130543#step:11:806

Needs more investigation. Also fails when running locally on a KIND cluster apart from CI, so should be relatively easy to reproduce.

• Failure [101.866 seconds]
BFD
/home/runner/work/ovn-kubernetes/ovn-kubernetes/test/e2e/external_gateways.go:380
  e2e multiple external gateway validation
  /home/runner/work/ovn-kubernetes/ovn-kubernetes/test/e2e/external_gateways.go:591
    Should validate TCP/UDP connectivity to multiple external gateways for a UDP / TCP scenario
    /home/runner/go/pkg/mod/github.com/onsi/ginkgo@v1.12.0/extensions/table/table.go:92
      IPV4 tcp [It]
      /home/runner/go/pkg/mod/github.com/onsi/ginkgo@v1.12.0/extensions/table/table_entry.go:43

      Sep 27 09:03:06.723: failed to reach 10.249.10.1 (tcp)
      Unexpected error:
          <exec.CodeExitError>: {
              Err: {
                  s: "error running /usr/local/bin/kubectl --server=https://10.1.0.238:11337 --kubeconfig=/home/runner/admin.conf --namespace=novxlan-externalgw-ecmp-694 exec e2e-exgw-src-pod -- bash -c echo | nc -w 1 10.249.10.1 80:\nCommand stdout:\n\nstderr:\ncommand terminated with exit code 1\n\nerror:\nexit status 1",
              },
              Code: 1,
          }
          error running /usr/local/bin/kubectl --server=https://10.1.0.238:11337 --kubeconfig=/home/runner/admin.conf --namespace=novxlan-externalgw-ecmp-694 exec e2e-exgw-src-pod -- bash -c echo | nc -w 1 10.249.10.1 80:
          Command stdout:

          stderr:
          command terminated with exit code 1

          error:
          exit status 1
      occurred

      /home/runner/work/ovn-kubernetes/ovn-kubernetes/test/e2e/external_gateways.go:982
tssurya commented 2 years ago

/assign @tssurya

tssurya commented 2 years ago

On the contrary, locally on KIND cluster, multiple external gateway validation passes, its e2e non-vxlan external gateway through a gateway pod Should validate TCP/UDP connectivity to an external gateway's loopback address via a pod with external gateway annotations enabled that failed.

• [SLOW TEST:24.461 seconds]
e2e multiple external gateway validation
/home/surya/go/src/github.com/ovn-org/ovn-kubernetes/test/e2e/external_gateways.go:223
  Should validate TCP/UDP connectivity to multiple external gateways for a UDP / TCP scenario
  /home/surya/go/pkg/mod/github.com/onsi/ginkgo@v1.12.0/extensions/table/table.go:92
    IPV4 tcp
    /home/surya/go/pkg/mod/github.com/onsi/ginkgo@v1.12.0/extensions/table/table_entry.go:41
------------------------------
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
Ran 1 of 60 Specs in 24.461 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 59 Skipped
e2e non-vxlan external gateway through a gateway pod Should validate TCP/UDP connectivity to an external gateway's loopback address via a pod with external gateway annotations enabled 
  TCP ipv4
  /home/surya/go/pkg/mod/github.com/onsi/ginkgo@v1.12.0/extensions/table/table_entry.go:41
[BeforeEach] e2e non-vxlan external gateway through a gateway pod
  /home/surya/go/pkg/mod/k8s.io/kubernetes@v1.20.2/test/e2e/framework/framework.go:174
STEP: Creating a kubernetes client
Sep 27 11:50:01.657: INFO: >>> kubeConfig: /home/surya/admin.conf
STEP: Building a namespace api object, basename externalgw-pod-novxlan
Sep 27 11:50:01.702: INFO: No PodSecurityPolicies found; assuming PodSecurityPolicy is disabled.
STEP: Waiting for a default service account to be provisioned in namespace
[BeforeEach] e2e non-vxlan external gateway through a gateway pod
  /home/surya/go/src/github.com/ovn-org/ovn-kubernetes/test/e2e/external_gateways.go:76
STEP: Creating the gateway containers for the icmp test
Sep 27 11:50:02.780: INFO: target ips are [10.249.10.1 10.249.10.2 10.249.10.3 10.249.10.4 10.249.10.5 10.249.10.6 10.249.10.7 10.249.10.8 10.249.10.9 10.249.10.10 10.249.10.11 10.249.10.12 10.249.10.13 10.249.10.14 10.249.10.15 10.249.10.16 10.249.10.17 10.249.10.18 10.249.10.19 10.249.10.20]
Sep 27 11:50:02.780: INFO: target ipsv6 are [fc00:f853:ccd:e794::1 fc00:f853:ccd:e794::2 fc00:f853:ccd:e794::3 fc00:f853:ccd:e794::4 fc00:f853:ccd:e794::5 fc00:f853:ccd:e794::6 fc00:f853:ccd:e794::7 fc00:f853:ccd:e794::8 fc00:f853:ccd:e794::9 fc00:f853:ccd:e794::10 fc00:f853:ccd:e794::11 fc00:f853:ccd:e794::12 fc00:f853:ccd:e794::13 fc00:f853:ccd:e794::14 fc00:f853:ccd:e794::15 fc00:f853:ccd:e794::16 fc00:f853:ccd:e794::17 fc00:f853:ccd:e794::18 fc00:f853:ccd:e794::19 fc00:f853:ccd:e794::20]
Sep 27 11:50:02.874: INFO: the pod side node is ovn-control-plane and the source node ip is 172.18.0.3 - fc00:f853:ccd:e793::3
STEP: Creating the source pod to reach the destination ips from
Sep 27 11:50:06.893: INFO: the pod source pod ip(s) are 10.244.1.4 - 
STEP: Setting up the destination ips to ex-gw-container1
STEP: Adding a route from ex-gw-container1 to the src pod
STEP: Setting up the listeners on the gateway
STEP: Setting up the destination ips to ex-gw-container2
STEP: Adding a route from ex-gw-container2 to the src pod
STEP: Setting up the listeners on the gateway
Sep 27 11:50:41.752: INFO: Annotating the external gateway pod with annotation [annotate pods e2e-gateway-pod1 k8s.v1.cni.cncf.io/network-status=[{"name":"foo","interface":"net1","ips":["172.18.0.5"],"mac":"01:23:45:67:89:10"}] k8s.ovn.org/routing-namespaces=externalgw-pod-novxlan-8081 k8s.ovn.org/routing-network=foo]
Sep 27 11:50:41.752: INFO: Running '/usr/local/bin/kubectl --server=https://192.168.18.13:11337 --kubeconfig=/home/surya/admin.conf --namespace=default annotate pods e2e-gateway-pod1 k8s.v1.cni.cncf.io/network-status=[{"name":"foo","interface":"net1","ips":["172.18.0.5"],"mac":"01:23:45:67:89:10"}] k8s.ovn.org/routing-namespaces=externalgw-pod-novxlan-8081 k8s.ovn.org/routing-network=foo'
Sep 27 11:50:41.819: INFO: stderr: ""
Sep 27 11:50:41.819: INFO: stdout: "pod/e2e-gateway-pod1 annotated\n"
Sep 27 11:50:41.819: INFO: Annotating the external gateway pod with annotation [annotate pods e2e-gateway-pod2 k8s.v1.cni.cncf.io/network-status=[{"name":"foo","interface":"net1","ips":["172.18.0.6"],"mac":"01:23:45:67:89:10"}] k8s.ovn.org/routing-namespaces=externalgw-pod-novxlan-8081 k8s.ovn.org/routing-network=foo]
Sep 27 11:50:41.819: INFO: Running '/usr/local/bin/kubectl --server=https://192.168.18.13:11337 --kubeconfig=/home/surya/admin.conf --namespace=default annotate pods e2e-gateway-pod2 k8s.v1.cni.cncf.io/network-status=[{"name":"foo","interface":"net1","ips":["172.18.0.6"],"mac":"01:23:45:67:89:10"}] k8s.ovn.org/routing-namespaces=externalgw-pod-novxlan-8081 k8s.ovn.org/routing-network=foo'
Sep 27 11:50:41.890: INFO: stderr: ""
Sep 27 11:50:41.890: INFO: stdout: "pod/e2e-gateway-pod2 annotated\n"
[It] TCP ipv4
  /home/surya/go/pkg/mod/github.com/onsi/ginkgo@v1.12.0/extensions/table/table_entry.go:41
STEP: Checking that ex-gw-container1 can reach the pod
STEP: Checking that ex-gw-container2 can reach the pod
Sep 27 11:50:42.222: INFO: Hostname for ex-gw-container1 is 60660b8f0210
Sep 27 11:50:42.315: INFO: Hostname for ex-gw-container2 is 70ac86dfa7e4
Sep 27 11:50:42.315: INFO: Expected hostnames are map[60660b8f0210:{} 70ac86dfa7e4:{}]
STEP: Checking that external ips are reachable with both gateways
Sep 27 11:50:42.315: INFO: Running '/usr/local/bin/kubectl --server=https://192.168.18.13:11337 --kubeconfig=/home/surya/admin.conf --namespace=externalgw-pod-novxlan-8081 exec e2e-exgw-src-ping-pod -- bash -c echo | nc -w 1 10.249.10.1 80'
Sep 27 11:50:43.433: INFO: rc: 1
Sep 27 11:50:43.433: FAIL: failed to reach 10.249.10.1 (tcp)
Unexpected error:
    <exec.CodeExitError>: {
        Err: {
            s: "error running /usr/local/bin/kubectl --server=https://192.168.18.13:11337 --kubeconfig=/home/surya/admin.conf --namespace=externalgw-pod-novxlan-8081 exec e2e-exgw-src-ping-pod -- bash -c echo | nc -w 1 10.249.10.1 80:\nCommand stdout:\n\nstderr:\ncommand terminated with exit code 1\n\nerror:\nexit status 1",
        },
        Code: 1,
    }
    error running /usr/local/bin/kubectl --server=https://192.168.18.13:11337 --kubeconfig=/home/surya/admin.conf --namespace=externalgw-pod-novxlan-8081 exec e2e-exgw-src-ping-pod -- bash -c echo | nc -w 1 10.249.10.1 80:
    Command stdout:

    stderr:
    command terminated with exit code 1

    error:
    exit status 1
occurred

Full Stack Trace
tssurya commented 2 years ago

Test is Flaky.

[surya@hidden-temple e2e]$ /usr/bin/go test  -run ^TestE2e$ github.com/ovn-org/ovn-kubernetes/test/e2e
I0927 13:55:27.830401 3650058 test_context.go:457] Tolerating taints "node-role.kubernetes.io/master" when considering if nodes are ready
Sep 27 13:55:27.830: INFO: The --provider flag is not set. Continuing as if --provider=skeleton had been used.
Running Suite: E2e Suite
========================
Random Seed: 1632743727 - Will randomize all specs
Will run 1 of 60 specs

SSSSSSSSSSSSSSSSSS
------------------------------
e2e multiple external gateway validation Should validate TCP/UDP connectivity to multiple external gateways for a UDP / TCP scenario 
  IPV4 tcp
  /home/surya/go/pkg/mod/github.com/onsi/ginkgo@v1.12.0/extensions/table/table_entry.go:41
[BeforeEach] e2e multiple external gateway validation
  /home/surya/go/pkg/mod/k8s.io/kubernetes@v1.20.2/test/e2e/framework/framework.go:174
STEP: Creating a kubernetes client
Sep 27 13:55:27.830: INFO: >>> kubeConfig: /home/surya/admin.conf
STEP: Building a namespace api object, basename novxlan-externalgw-ecmp
Sep 27 13:55:27.861: INFO: No PodSecurityPolicies found; assuming PodSecurityPolicy is disabled.
STEP: Waiting for a default service account to be provisioned in namespace
[BeforeEach] e2e multiple external gateway validation
  /home/surya/go/src/github.com/ovn-org/ovn-kubernetes/test/e2e/external_gateways.go:239
STEP: Creating the gateway containers for the icmp test
Sep 27 13:55:28.825: INFO: target ips are [10.249.10.1 10.249.10.2 10.249.10.3 10.249.10.4 10.249.10.5 10.249.10.6 10.249.10.7 10.249.10.8 10.249.10.9 10.249.10.10 10.249.10.11 10.249.10.12 10.249.10.13 10.249.10.14 10.249.10.15 10.249.10.16 10.249.10.17 10.249.10.18 10.249.10.19 10.249.10.20]
Sep 27 13:55:28.825: INFO: target ipsv6 are [fc00:f853:ccd:e794::1 fc00:f853:ccd:e794::2 fc00:f853:ccd:e794::3 fc00:f853:ccd:e794::4 fc00:f853:ccd:e794::5 fc00:f853:ccd:e794::6 fc00:f853:ccd:e794::7 fc00:f853:ccd:e794::8 fc00:f853:ccd:e794::9 fc00:f853:ccd:e794::10 fc00:f853:ccd:e794::11 fc00:f853:ccd:e794::12 fc00:f853:ccd:e794::13 fc00:f853:ccd:e794::14 fc00:f853:ccd:e794::15 fc00:f853:ccd:e794::16 fc00:f853:ccd:e794::17 fc00:f853:ccd:e794::18 fc00:f853:ccd:e794::19 fc00:f853:ccd:e794::20]
Sep 27 13:55:28.888: INFO: the pod side node is ovn-control-plane and the source node ip is 172.18.0.4 - fc00:f853:ccd:e793::4
STEP: Creating the source pod to reach the destination ips from
Sep 27 13:55:42.909: INFO: the pod source pod ip(s) are 10.244.2.3 - 
STEP: Setting up the destination ips to gw-test-container1
STEP: Adding a route from gw-test-container1 to the src pod
STEP: Setting up the listeners on the gateway
STEP: Setting up the destination ips to gw-test-container2
STEP: Adding a route from gw-test-container2 to the src pod
STEP: Setting up the listeners on the gateway
STEP: Resetting the gw annotation
Sep 27 13:55:59.197: INFO: Running '/usr/local/bin/kubectl --server=https://192.168.18.13:11337 --kubeconfig=/home/surya/admin.conf --namespace=novxlan-externalgw-ecmp-8081 annotate namespace novxlan-externalgw-ecmp-8081 k8s.ovn.org/routing-external-gws-'
Sep 27 13:55:59.260: INFO: stderr: ""
Sep 27 13:55:59.260: INFO: stdout: "namespace/novxlan-externalgw-ecmp-8081 annotated\n"
[It] IPV4 tcp
  /home/surya/go/pkg/mod/github.com/onsi/ginkgo@v1.12.0/extensions/table/table_entry.go:41
Sep 27 13:55:59.260: INFO: Annotating the external gateway test namespace to container gateways: 172.18.0.5,172.18.0.6
Sep 27 13:55:59.260: INFO: Running '/usr/local/bin/kubectl --server=https://192.168.18.13:11337 --kubeconfig=/home/surya/admin.conf --namespace=novxlan-externalgw-ecmp-8081 annotate namespace novxlan-externalgw-ecmp-8081 k8s.ovn.org/routing-external-gws=172.18.0.5,172.18.0.6'
Sep 27 13:55:59.320: INFO: stderr: ""
Sep 27 13:55:59.320: INFO: stdout: "namespace/novxlan-externalgw-ecmp-8081 annotated\n"
STEP: Checking that gw-test-container1 can reach the pod
STEP: Checking that gw-test-container2 can reach the pod
Sep 27 13:56:01.319: INFO: Hostname for gw-test-container1 is b2e884990a1e

Sep 27 13:56:01.428: INFO: Hostname for gw-test-container2 is 72b39532da5c

Sep 27 13:56:01.428: INFO: Expected hostnames are map[72b39532da5c:{} b2e884990a1e:{}]
Sep 27 13:56:01.428: INFO: Running '/usr/local/bin/kubectl --server=https://192.168.18.13:11337 --kubeconfig=/home/surya/admin.conf --namespace=novxlan-externalgw-ecmp-8081 exec e2e-exgw-src-pod -- bash -c echo | nc -w 1 10.249.10.1 80'
Sep 27 13:56:02.547: INFO: rc: 1
Sep 27 13:56:02.547: FAIL: failed to reach 10.249.10.1 (tcp)
Unexpected error:
    <exec.CodeExitError>: {
        Err: {
            s: "error running /usr/local/bin/kubectl --server=https://192.168.18.13:11337 --kubeconfig=/home/surya/admin.conf --namespace=novxlan-externalgw-ecmp-8081 exec e2e-exgw-src-pod -- bash -c echo | nc -w 1 10.249.10.1 80:\nCommand stdout:\n\nstderr:\ncommand terminated with exit code 1\n\nerror:\nexit status 1",
        },
        Code: 1,
    }
    error running /usr/local/bin/kubectl --server=https://192.168.18.13:11337 --kubeconfig=/home/surya/admin.conf --namespace=novxlan-externalgw-ecmp-8081 exec e2e-exgw-src-pod -- bash -c echo | nc -w 1 10.249.10.1 80:
    Command stdout:

    stderr:
    command terminated with exit code 1

    error:
    exit status 1
occurred

Full Stack Trace
github.com/ovn-org/ovn-kubernetes/test/e2e.pokeHostnameViaNC(0x1a66c63, 0x10, 0xc0008dc060, 0x1c, 0x1a5638b, 0x3, 0xc0004c6a90, 0xb, 0x50, 0x3, ...)
    /home/surya/go/src/github.com/ovn-org/ovn-kubernetes/test/e2e/external_gateways.go:982 +0x369
github.com/ovn-org/ovn-kubernetes/test/e2e.glob..func14.4(0xc000339260, 0x1a5638b, 0x3, 0x50, 0x50)
    /home/surya/go/src/github.com/ovn-org/ovn-kubernetes/test/e2e/external_gateways.go:355 +0x47e
reflect.Value.call(0x182c520, 0xc0002e0f60, 0x13, 0x1a566a2, 0x4, 0xc00031a660, 0x4, 0x4, 0x2ae, 0x0, ...)
    /usr/local/go/src/reflect/value.go:476 +0x8e7
reflect.Value.Call(0x182c520, 0xc0002e0f60, 0x13, 0xc00031a660, 0x4, 0x4, 0x0, 0x0, 0x0)
    /usr/local/go/src/reflect/value.go:337 +0xb9
github.com/onsi/ginkgo/extensions/table.TableEntry.generateIt.func1()
    /home/surya/go/pkg/mod/github.com/onsi/ginkgo@v1.12.0/extensions/table/table_entry.go:37 +0x57
github.com/onsi/ginkgo/internal/leafnodes.(*runner).runSync(0xc00031a720, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /home/surya/go/pkg/mod/github.com/onsi/ginkgo@v1.12.0/internal/leafnodes/runner.go:113 +0xa3
github.com/onsi/ginkgo/internal/leafnodes.(*runner).run(0xc00031a720, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /home/surya/go/pkg/mod/github.com/onsi/ginkgo@v1.12.0/internal/leafnodes/runner.go:64 +0x15c
github.com/onsi/ginkgo/internal/leafnodes.(*ItNode).Run(0xc000498520, 0x1c61bc0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /home/surya/go/pkg/mod/github.com/onsi/ginkgo@v1.12.0/internal/leafnodes/it_node.go:26 +0x87
github.com/onsi/ginkgo/internal/spec.(*Spec).runSample(0xc00029a000, 0x0, 0x1c61bc0, 0xc000056880)
    /home/surya/go/pkg/mod/github.com/onsi/ginkgo@v1.12.0/internal/spec/spec.go:215 +0x72f
github.com/onsi/ginkgo/internal/spec.(*Spec).Run(0xc00029a000, 0x1c61bc0, 0xc000056880)
    /home/surya/go/pkg/mod/github.com/onsi/ginkgo@v1.12.0/internal/spec/spec.go:138 +0xf2
github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).runSpec(0xc0002a7540, 0xc00029a000, 0x0)
    /home/surya/go/pkg/mod/github.com/onsi/ginkgo@v1.12.0/internal/specrunner/spec_runner.go:200 +0x111
github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).runSpecs(0xc0002a7540, 0x1)
    /home/surya/go/pkg/mod/github.com/onsi/ginkgo@v1.12.0/internal/specrunner/spec_runner.go:170 +0x147
github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).Run(0xc0002a7540, 0xc00070a140)
    /home/surya/go/pkg/mod/github.com/onsi/ginkgo@v1.12.0/internal/specrunner/spec_runner.go:66 +0x117
github.com/onsi/ginkgo/internal/suite.(*Suite).Run(0xc00003c320, 0x7f8928354230, 0xc000482f00, 0x1a5cb51, 0x9, 0xc00012c580, 0x1, 0x1, 0x1c9fd78, 0xc000056880, ...)
    /home/surya/go/pkg/mod/github.com/onsi/ginkgo@v1.12.0/internal/suite/suite.go:62 +0x426
github.com/onsi/ginkgo.RunSpecsWithCustomReporters(0x1c63360, 0xc000482f00, 0x1a5cb51, 0x9, 0xc00012c560, 0x1, 0x1, 0x1)
    /home/surya/go/pkg/mod/github.com/onsi/ginkgo@v1.12.0/ginkgo_dsl.go:226 +0x218
github.com/onsi/ginkgo.RunSpecsWithDefaultAndCustomReporters(0x1c63360, 0xc000482f00, 0x1a5cb51, 0x9, 0x0, 0x0, 0x0, 0x272aa40)
    /home/surya/go/pkg/mod/github.com/onsi/ginkgo@v1.12.0/ginkgo_dsl.go:214 +0xad
github.com/ovn-org/ovn-kubernetes/test/e2e.TestE2e(0xc000482f00)
    /home/surya/go/src/github.com/ovn-org/ovn-kubernetes/test/e2e/e2e_suite_test.go:77 +0xf5
testing.tRunner(0xc000482f00, 0x1b205d8)
    /usr/local/go/src/testing/testing.go:1193 +0xef
created by testing.(*T).Run
    /usr/local/go/src/testing/testing.go:1238 +0x2b3
[AfterEach] e2e multiple external gateway validation
  /home/surya/go/pkg/mod/k8s.io/kubernetes@v1.20.2/test/e2e/framework/framework.go:175
STEP: Collecting events from namespace "novxlan-externalgw-ecmp-8081".
STEP: Found 4 events.
Sep 27 13:56:02.550: INFO: At 2021-09-27 13:55:30 +0200 CEST - event for e2e-exgw-src-pod: {kubelet ovn-control-plane} Pulling: Pulling image "k8s.gcr.io/e2e-test-images/agnhost:2.26"
Sep 27 13:56:02.550: INFO: At 2021-09-27 13:55:41 +0200 CEST - event for e2e-exgw-src-pod: {kubelet ovn-control-plane} Pulled: Successfully pulled image "k8s.gcr.io/e2e-test-images/agnhost:2.26" in 10.285038427s
Sep 27 13:56:02.550: INFO: At 2021-09-27 13:55:41 +0200 CEST - event for e2e-exgw-src-pod: {kubelet ovn-control-plane} Created: Created container e2e-exgw-src-pod-container
Sep 27 13:56:02.550: INFO: At 2021-09-27 13:55:41 +0200 CEST - event for e2e-exgw-src-pod: {kubelet ovn-control-plane} Started: Started container e2e-exgw-src-pod-container
Sep 27 13:56:02.552: INFO: POD               NODE               PHASE    GRACE  CONDITIONS
Sep 27 13:56:02.552: INFO: e2e-exgw-src-pod  ovn-control-plane  Running         [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2021-09-27 13:55:28 +0200 CEST  } {Ready True 0001-01-01 00:00:00 +0000 UTC 2021-09-27 13:55:42 +0200 CEST  } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2021-09-27 13:55:42 +0200 CEST  } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2021-09-27 13:55:28 +0200 CEST  }]
Sep 27 13:56:02.552: INFO: 
Sep 27 13:56:02.553: INFO: 
Logging node info for node ovn-control-plane
Sep 27 13:56:02.554: INFO: Node Info: &Node{ObjectMeta:{ovn-control-plane    658e25d5-f217-4ec7-8d50-4cb4e1947e2a 1203 0 2021-09-27 13:51:14 +0200 CEST <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/os:linux ingress-ready:true k8s.ovn.org/ovnkube-db:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ovn-control-plane kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node-role.kubernetes.io/master:] map[k8s.ovn.org/host-addresses:["172.18.0.4","fc00:f853:ccd:e793::4"] k8s.ovn.org/l3-gateway-config:{"default":{"mode":"shared","interface-id":"breth0_ovn-control-plane","mac-address":"02:42:ac:12:00:04","ip-addresses":["172.18.0.4/16"],"ip-address":"172.18.0.4/16","next-hops":["172.18.0.1"],"next-hop":"172.18.0.1","node-port-enable":"true","vlan-id":"0"}} k8s.ovn.org/node-chassis-id:296a55f9-e8cb-41e1-a208-355b39ac0ea6 k8s.ovn.org/node-mgmt-port-mac-address:ea:73:a2:e7:63:3d k8s.ovn.org/node-primary-ifaddr:{"ipv4":"172.18.0.4/16"} k8s.ovn.org/node-subnets:{"default":"10.244.2.0/24"} k8s.ovn.org/topology-version:4 kubeadm.alpha.kubernetes.io/cri-socket:unix:///run/containerd/containerd.sock node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] []  [{kubelet Update v1 2021-09-27 13:51:14 +0200 CEST FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:ingress-ready":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}},"f:spec":{"f:providerID":{}},"f:status":{"f:addresses":{".":{},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}},"f:allocatable":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}}}}} {kubeadm Update v1 2021-09-27 13:51:17 +0200 CEST FieldsV1 {"f:metadata":{"f:annotations":{"f:kubeadm.alpha.kubernetes.io/cri-socket":{}},"f:labels":{"f:node-role.kubernetes.io/control-plane":{},"f:node-role.kubernetes.io/master":{}}}}} {kubectl-label Update v1 2021-09-27 13:52:31 +0200 CEST FieldsV1 {"f:metadata":{"f:labels":{"f:k8s.ovn.org/ovnkube-db":{}}}}} {ovn-control-plane Update v1 2021-09-27 13:52:41 +0200 CEST FieldsV1 {"f:metadata":{"f:annotations":{"f:k8s.ovn.org/host-addresses":{},"f:k8s.ovn.org/l3-gateway-config":{},"f:k8s.ovn.org/node-chassis-id":{},"f:k8s.ovn.org/node-mgmt-port-mac-address":{},"f:k8s.ovn.org/node-primary-ifaddr":{},"f:k8s.ovn.org/node-subnets":{},"f:k8s.ovn.org/topology-version":{}}}}} {kube-controller-manager Update v1 2021-09-27 13:52:43 +0200 CEST FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"10.244.0.0/24\"":{}}}}}]},Spec:NodeSpec{PodCIDR:10.244.0.0/24,DoNotUseExternalID:,ProviderID:kind://docker/ovn/ovn-control-plane,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{8 0} {<nil>} 8 DecimalSI},ephemeral-storage: {{73445531648 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{33436639232 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{8 0} {<nil>} 8 DecimalSI},ephemeral-storage: {{73445531648 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{33436639232 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2021-09-27 13:55:53 +0200 CEST,LastTransitionTime:2021-09-27 13:51:11 +0200 CEST,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2021-09-27 13:55:53 +0200 CEST,LastTransitionTime:2021-09-27 13:51:11 +0200 CEST,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2021-09-27 13:55:53 +0200 CEST,LastTransitionTime:2021-09-27 13:51:11 +0200 CEST,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2021-09-27 13:55:53 +0200 CEST,LastTransitionTime:2021-09-27 13:52:43 +0200 CEST,Reason:KubeletReady,Message:kubelet is posting ready status,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.18.0.4,},NodeAddress{Type:Hostname,Address:ovn-control-plane,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:bba66493d8104feead134b49800b77d2,SystemUUID:3c5f7e98-dd60-4035-9ae5-86baf0119360,BootID:aa6ee10a-cbfe-4d03-8e3c-d6197944d244,KernelVersion:5.11.22-100.fc32.x86_64,OSImage:Ubuntu Groovy Gorilla (development branch),ContainerRuntimeVersion:containerd://1.4.0,KubeletVersion:v1.20.0,KubeProxyVersion:v1.20.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[localhost/ovn-daemonset-f:dev],SizeBytes:562147486,},ContainerImage{Names:[k8s.gcr.io/etcd:3.4.13-0],SizeBytes:254659261,},ContainerImage{Names:[k8s.gcr.io/kube-proxy:v1.20.0],SizeBytes:136866161,},ContainerImage{Names:[docker.io/kindest/kindnetd:v20200725-4d6bea59],SizeBytes:118720874,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver:v1.20.0],SizeBytes:95511851,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager:v1.20.0],SizeBytes:88147263,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler:v1.20.0],SizeBytes:66088749,},ContainerImage{Names:[k8s.gcr.io/build-image/debian-base:v2.1.0],SizeBytes:53876619,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:a3f7549ea04c419276e8b84e90a515bbce5bc8a057be2ed974ec45492eca346e k8s.gcr.io/e2e-test-images/agnhost:2.26],SizeBytes:49216572,},ContainerImage{Names:[k8s.gcr.io/coredns:1.7.0],SizeBytes:45355487,},ContainerImage{Names:[docker.io/rancher/local-path-provisioner:v0.0.14],SizeBytes:41982521,},ContainerImage{Names:[k8s.gcr.io/pause:3.3],SizeBytes:685708,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Sep 27 13:56:02.554: INFO: 
Logging kubelet events for node ovn-control-plane
Sep 27 13:56:02.556: INFO: 
Logging pods the kubelet thinks is on node ovn-control-plane
Sep 27 13:56:02.573: INFO: e2e-exgw-src-pod started at 2021-09-27 13:55:28 +0200 CEST (0+1 container statuses recorded)
Sep 27 13:56:02.573: INFO:  Container e2e-exgw-src-pod-container ready: true, restart count 0
Sep 27 13:56:02.573: INFO: etcd-ovn-control-plane started at 2021-09-27 13:51:23 +0200 CEST (0+1 container statuses recorded)
Sep 27 13:56:02.573: INFO:  Container etcd ready: true, restart count 0
Sep 27 13:56:02.573: INFO: kube-apiserver-ovn-control-plane started at 2021-09-27 13:51:23 +0200 CEST (0+1 container statuses recorded)
Sep 27 13:56:02.573: INFO:  Container kube-apiserver ready: true, restart count 0
Sep 27 13:56:02.573: INFO: ovnkube-db-7455b798b7-hlbp4 started at 2021-09-27 13:52:31 +0200 CEST (0+2 container statuses recorded)
Sep 27 13:56:02.573: INFO:  Container nb-ovsdb ready: true, restart count 0
Sep 27 13:56:02.573: INFO:  Container sb-ovsdb ready: true, restart count 0
Sep 27 13:56:02.573: INFO: ovs-node-xrwnj started at 2021-09-27 13:52:31 +0200 CEST (0+1 container statuses recorded)
Sep 27 13:56:02.573: INFO:  Container ovs-daemons ready: true, restart count 0
Sep 27 13:56:02.573: INFO: ovnkube-node-t57rp started at 2021-09-27 13:52:32 +0200 CEST (0+3 container statuses recorded)
Sep 27 13:56:02.573: INFO:  Container ovn-controller ready: true, restart count 0
Sep 27 13:56:02.573: INFO:  Container ovnkube-node ready: true, restart count 0
Sep 27 13:56:02.573: INFO:  Container ovs-metrics-exporter ready: true, restart count 0
Sep 27 13:56:02.573: INFO: kube-controller-manager-ovn-control-plane started at 2021-09-27 13:51:23 +0200 CEST (0+1 container statuses recorded)
Sep 27 13:56:02.573: INFO:  Container kube-controller-manager ready: true, restart count 0
Sep 27 13:56:02.573: INFO: kube-scheduler-ovn-control-plane started at 2021-09-27 13:51:23 +0200 CEST (0+1 container statuses recorded)
Sep 27 13:56:02.573: INFO:  Container kube-scheduler ready: true, restart count 0
Sep 27 13:56:02.573: INFO: ovnkube-master-85567b87f7-dmsxw started at 2021-09-27 13:52:32 +0200 CEST (0+3 container statuses recorded)
Sep 27 13:56:02.573: INFO:  Container nbctl-daemon ready: true, restart count 0
Sep 27 13:56:02.573: INFO:  Container ovn-northd ready: true, restart count 0
Sep 27 13:56:02.573: INFO:  Container ovnkube-master ready: true, restart count 0
W0927 13:56:02.576167 3650058 metrics_grabber.go:105] Did not receive an external client interface. Grabbing metrics from ClusterAutoscaler is disabled.
Sep 27 13:56:02.643: INFO: 
Latency metrics for node ovn-control-plane
Sep 27 13:56:02.643: INFO: 
Logging node info for node ovn-worker
Sep 27 13:56:02.645: INFO: Node Info: &Node{ObjectMeta:{ovn-worker    36ce0006-7b73-4a8c-a5f1-7ba2ca5e3aab 785 0 2021-09-27 13:51:42 +0200 CEST <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/os:linux kubernetes.io/arch:amd64 kubernetes.io/hostname:ovn-worker kubernetes.io/os:linux] map[k8s.ovn.org/host-addresses:["172.18.0.2","fc00:f853:ccd:e793::2"] k8s.ovn.org/l3-gateway-config:{"default":{"mode":"shared","interface-id":"breth0_ovn-worker","mac-address":"02:42:ac:12:00:02","ip-addresses":["172.18.0.2/16"],"ip-address":"172.18.0.2/16","next-hops":["172.18.0.1"],"next-hop":"172.18.0.1","node-port-enable":"true","vlan-id":"0"}} k8s.ovn.org/node-chassis-id:05303806-b991-4cfc-97de-64bf05286adb k8s.ovn.org/node-mgmt-port-mac-address:6e:6e:2c:21:58:06 k8s.ovn.org/node-primary-ifaddr:{"ipv4":"172.18.0.2/16"} k8s.ovn.org/node-subnets:{"default":"10.244.0.0/24"} kubeadm.alpha.kubernetes.io/cri-socket:unix:///run/containerd/containerd.sock node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] []  [{kube-controller-manager Update v1 2021-09-27 13:51:42 +0200 CEST FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"10.244.1.0/24\"":{}}}}} {kubeadm Update v1 2021-09-27 13:51:42 +0200 CEST FieldsV1 {"f:metadata":{"f:annotations":{"f:kubeadm.alpha.kubernetes.io/cri-socket":{}}}}} {kubelet Update v1 2021-09-27 13:51:42 +0200 CEST FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}},"f:spec":{"f:providerID":{}},"f:status":{"f:addresses":{".":{},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}},"f:allocatable":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}}}}} {ovn-control-plane Update v1 2021-09-27 13:52:36 +0200 CEST FieldsV1 {"f:metadata":{"f:annotations":{"f:k8s.ovn.org/node-subnets":{}}}}} {ovn-worker Update v1 2021-09-27 13:52:41 +0200 CEST FieldsV1 {"f:metadata":{"f:annotations":{"f:k8s.ovn.org/host-addresses":{},"f:k8s.ovn.org/l3-gateway-config":{},"f:k8s.ovn.org/node-chassis-id":{},"f:k8s.ovn.org/node-mgmt-port-mac-address":{},"f:k8s.ovn.org/node-primary-ifaddr":{}}}}}]},Spec:NodeSpec{PodCIDR:10.244.1.0/24,DoNotUseExternalID:,ProviderID:kind://docker/ovn/ovn-worker,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{8 0} {<nil>} 8 DecimalSI},ephemeral-storage: {{73445531648 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{33436639232 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{8 0} {<nil>} 8 DecimalSI},ephemeral-storage: {{73445531648 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{33436639232 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2021-09-27 13:52:52 +0200 CEST,LastTransitionTime:2021-09-27 13:51:42 +0200 CEST,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2021-09-27 13:52:52 +0200 CEST,LastTransitionTime:2021-09-27 13:51:42 +0200 CEST,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2021-09-27 13:52:52 +0200 CEST,LastTransitionTime:2021-09-27 13:51:42 +0200 CEST,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2021-09-27 13:52:52 +0200 CEST,LastTransitionTime:2021-09-27 13:52:52 +0200 CEST,Reason:KubeletReady,Message:kubelet is posting ready status,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.18.0.2,},NodeAddress{Type:Hostname,Address:ovn-worker,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:b66a8175e6a24ae1b6576667740677f9,SystemUUID:51ecdb50-8abb-4de1-9b1e-70bf08edee3e,BootID:aa6ee10a-cbfe-4d03-8e3c-d6197944d244,KernelVersion:5.11.22-100.fc32.x86_64,OSImage:Ubuntu Groovy Gorilla (development branch),ContainerRuntimeVersion:containerd://1.4.0,KubeletVersion:v1.20.0,KubeProxyVersion:v1.20.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[localhost/ovn-daemonset-f:dev],SizeBytes:562147486,},ContainerImage{Names:[k8s.gcr.io/etcd:3.4.13-0],SizeBytes:254659261,},ContainerImage{Names:[k8s.gcr.io/kube-proxy:v1.20.0],SizeBytes:136866161,},ContainerImage{Names:[docker.io/kindest/kindnetd:v20200725-4d6bea59],SizeBytes:118720874,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver:v1.20.0],SizeBytes:95511851,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager:v1.20.0],SizeBytes:88147263,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler:v1.20.0],SizeBytes:66088749,},ContainerImage{Names:[k8s.gcr.io/build-image/debian-base:v2.1.0],SizeBytes:53876619,},ContainerImage{Names:[k8s.gcr.io/coredns:1.7.0],SizeBytes:45355487,},ContainerImage{Names:[docker.io/rancher/local-path-provisioner:v0.0.14],SizeBytes:41982521,},ContainerImage{Names:[k8s.gcr.io/pause:3.3],SizeBytes:685708,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Sep 27 13:56:02.645: INFO: 
Logging kubelet events for node ovn-worker
Sep 27 13:56:02.647: INFO: 
Logging pods the kubelet thinks is on node ovn-worker
Sep 27 13:56:02.658: INFO: ovnkube-node-5pfh7 started at 2021-09-27 13:52:32 +0200 CEST (0+3 container statuses recorded)
Sep 27 13:56:02.658: INFO:  Container ovn-controller ready: true, restart count 0
Sep 27 13:56:02.658: INFO:  Container ovnkube-node ready: true, restart count 0
Sep 27 13:56:02.658: INFO:  Container ovs-metrics-exporter ready: true, restart count 0
Sep 27 13:56:02.658: INFO: ovs-node-clnzw started at 2021-09-27 13:52:31 +0200 CEST (0+1 container statuses recorded)
Sep 27 13:56:02.658: INFO:  Container ovs-daemons ready: true, restart count 0
W0927 13:56:02.661040 3650058 metrics_grabber.go:105] Did not receive an external client interface. Grabbing metrics from ClusterAutoscaler is disabled.
Sep 27 13:56:02.693: INFO: 
Latency metrics for node ovn-worker
Sep 27 13:56:02.693: INFO: 
Logging node info for node ovn-worker2
Sep 27 13:56:02.695: INFO: Node Info: &Node{ObjectMeta:{ovn-worker2    b5aec24e-742f-44b2-8ecc-68a31110c664 752 0 2021-09-27 13:51:42 +0200 CEST <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/os:linux kubernetes.io/arch:amd64 kubernetes.io/hostname:ovn-worker2 kubernetes.io/os:linux] map[k8s.ovn.org/host-addresses:["172.18.0.3","fc00:f853:ccd:e793::3"] k8s.ovn.org/l3-gateway-config:{"default":{"mode":"shared","interface-id":"breth0_ovn-worker2","mac-address":"02:42:ac:12:00:03","ip-addresses":["172.18.0.3/16"],"ip-address":"172.18.0.3/16","next-hops":["172.18.0.1"],"next-hop":"172.18.0.1","node-port-enable":"true","vlan-id":"0"}} k8s.ovn.org/node-chassis-id:53f22323-66c3-42c8-9f03-b82dced25501 k8s.ovn.org/node-mgmt-port-mac-address:26:b5:06:eb:30:21 k8s.ovn.org/node-primary-ifaddr:{"ipv4":"172.18.0.3/16"} k8s.ovn.org/node-subnets:{"default":"10.244.1.0/24"} kubeadm.alpha.kubernetes.io/cri-socket:unix:///run/containerd/containerd.sock node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] []  [{kube-controller-manager Update v1 2021-09-27 13:51:42 +0200 CEST FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"10.244.2.0/24\"":{}}}}} {kubelet Update v1 2021-09-27 13:51:42 +0200 CEST FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}},"f:spec":{"f:providerID":{}},"f:status":{"f:addresses":{".":{},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}},"f:allocatable":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}}}}} {kubeadm Update v1 2021-09-27 13:51:43 +0200 CEST FieldsV1 {"f:metadata":{"f:annotations":{"f:kubeadm.alpha.kubernetes.io/cri-socket":{}}}}} {ovn-control-plane Update v1 2021-09-27 13:52:36 +0200 CEST FieldsV1 {"f:metadata":{"f:annotations":{"f:k8s.ovn.org/node-subnets":{}}}}} {ovn-worker2 Update v1 2021-09-27 13:52:41 +0200 CEST FieldsV1 {"f:metadata":{"f:annotations":{"f:k8s.ovn.org/host-addresses":{},"f:k8s.ovn.org/l3-gateway-config":{},"f:k8s.ovn.org/node-chassis-id":{},"f:k8s.ovn.org/node-mgmt-port-mac-address":{},"f:k8s.ovn.org/node-primary-ifaddr":{}}}}}]},Spec:NodeSpec{PodCIDR:10.244.2.0/24,DoNotUseExternalID:,ProviderID:kind://docker/ovn/ovn-worker2,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{8 0} {<nil>} 8 DecimalSI},ephemeral-storage: {{73445531648 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{33436639232 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{8 0} {<nil>} 8 DecimalSI},ephemeral-storage: {{73445531648 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{33436639232 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2021-09-27 13:52:42 +0200 CEST,LastTransitionTime:2021-09-27 13:51:42 +0200 CEST,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2021-09-27 13:52:42 +0200 CEST,LastTransitionTime:2021-09-27 13:51:42 +0200 CEST,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2021-09-27 13:52:42 +0200 CEST,LastTransitionTime:2021-09-27 13:51:42 +0200 CEST,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2021-09-27 13:52:42 +0200 CEST,LastTransitionTime:2021-09-27 13:52:42 +0200 CEST,Reason:KubeletReady,Message:kubelet is posting ready status,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.18.0.3,},NodeAddress{Type:Hostname,Address:ovn-worker2,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:baa21ea8817548ff887d547b9e6040b7,SystemUUID:20cd7b72-1e66-489b-99c4-e9fef27af2a0,BootID:aa6ee10a-cbfe-4d03-8e3c-d6197944d244,KernelVersion:5.11.22-100.fc32.x86_64,OSImage:Ubuntu Groovy Gorilla (development branch),ContainerRuntimeVersion:containerd://1.4.0,KubeletVersion:v1.20.0,KubeProxyVersion:v1.20.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[localhost/ovn-daemonset-f:dev],SizeBytes:562147486,},ContainerImage{Names:[k8s.gcr.io/etcd:3.4.13-0],SizeBytes:254659261,},ContainerImage{Names:[k8s.gcr.io/kube-proxy:v1.20.0],SizeBytes:136866161,},ContainerImage{Names:[docker.io/kindest/kindnetd:v20200725-4d6bea59],SizeBytes:118720874,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver:v1.20.0],SizeBytes:95511851,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager:v1.20.0],SizeBytes:88147263,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler:v1.20.0],SizeBytes:66088749,},ContainerImage{Names:[k8s.gcr.io/build-image/debian-base:v2.1.0],SizeBytes:53876619,},ContainerImage{Names:[k8s.gcr.io/coredns:1.7.0],SizeBytes:45355487,},ContainerImage{Names:[docker.io/rancher/local-path-provisioner:v0.0.14],SizeBytes:41982521,},ContainerImage{Names:[k8s.gcr.io/pause:3.3],SizeBytes:685708,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Sep 27 13:56:02.695: INFO: 
Logging kubelet events for node ovn-worker2
Sep 27 13:56:02.697: INFO: 
Logging pods the kubelet thinks is on node ovn-worker2
Sep 27 13:56:02.710: INFO: coredns-74ff55c5b-mt6g4 started at 2021-09-27 13:52:51 +0200 CEST (0+1 container statuses recorded)
Sep 27 13:56:02.710: INFO:  Container coredns ready: true, restart count 0
Sep 27 13:56:02.710: INFO: coredns-74ff55c5b-z5rrw started at 2021-09-27 13:52:51 +0200 CEST (0+1 container statuses recorded)
Sep 27 13:56:02.710: INFO:  Container coredns ready: true, restart count 0
Sep 27 13:56:02.710: INFO: ovs-node-kp798 started at 2021-09-27 13:52:31 +0200 CEST (0+1 container statuses recorded)
Sep 27 13:56:02.710: INFO:  Container ovs-daemons ready: true, restart count 0
Sep 27 13:56:02.710: INFO: ovnkube-node-nsj6g started at 2021-09-27 13:52:32 +0200 CEST (0+3 container statuses recorded)
Sep 27 13:56:02.710: INFO:  Container ovn-controller ready: true, restart count 0
Sep 27 13:56:02.710: INFO:  Container ovnkube-node ready: true, restart count 0
Sep 27 13:56:02.710: INFO:  Container ovs-metrics-exporter ready: true, restart count 0
Sep 27 13:56:02.710: INFO: local-path-provisioner-78776bfc44-gcrwn started at 2021-09-27 13:52:51 +0200 CEST (0+1 container statuses recorded)
Sep 27 13:56:02.710: INFO:  Container local-path-provisioner ready: true, restart count 0
W0927 13:56:02.712893 3650058 metrics_grabber.go:105] Did not receive an external client interface. Grabbing metrics from ClusterAutoscaler is disabled.
Sep 27 13:56:02.756: INFO: 
Latency metrics for node ovn-worker2
Sep 27 13:56:02.756: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
STEP: Destroying namespace "novxlan-externalgw-ecmp-8081" for this suite.
[AfterEach] e2e multiple external gateway validation
  /home/surya/go/src/github.com/ovn-org/ovn-kubernetes/test/e2e/external_gateways.go:261

• Failure [35.425 seconds]
e2e multiple external gateway validation
/home/surya/go/src/github.com/ovn-org/ovn-kubernetes/test/e2e/external_gateways.go:223
  Should validate TCP/UDP connectivity to multiple external gateways for a UDP / TCP scenario
  /home/surya/go/pkg/mod/github.com/onsi/ginkgo@v1.12.0/extensions/table/table.go:92
    IPV4 tcp [It]
    /home/surya/go/pkg/mod/github.com/onsi/ginkgo@v1.12.0/extensions/table/table_entry.go:41

    Sep 27 13:56:02.547: failed to reach 10.249.10.1 (tcp)
    Unexpected error:
        <exec.CodeExitError>: {
            Err: {
                s: "error running /usr/local/bin/kubectl --server=https://192.168.18.13:11337 --kubeconfig=/home/surya/admin.conf --namespace=novxlan-externalgw-ecmp-8081 exec e2e-exgw-src-pod -- bash -c echo | nc -w 1 10.249.10.1 80:\nCommand stdout:\n\nstderr:\ncommand terminated with exit code 1\n\nerror:\nexit status 1",
            },
            Code: 1,
        }
        error running /usr/local/bin/kubectl --server=https://192.168.18.13:11337 --kubeconfig=/home/surya/admin.conf --namespace=novxlan-externalgw-ecmp-8081 exec e2e-exgw-src-pod -- bash -c echo | nc -w 1 10.249.10.1 80:
        Command stdout:

        stderr:
        command terminated with exit code 1

        error:
        exit status 1
    occurred

    /home/surya/go/src/github.com/ovn-org/ovn-kubernetes/test/e2e/external_gateways.go:982
------------------------------
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS

Summarizing 1 Failure:

[Fail] e2e multiple external gateway validation Should validate TCP/UDP connectivity to multiple external gateways for a UDP / TCP scenario [It] IPV4 tcp 
/home/surya/go/src/github.com/ovn-org/ovn-kubernetes/test/e2e/external_gateways.go:982

Ran 1 of 60 Specs in 35.426 seconds
FAIL! -- 0 Passed | 1 Failed | 0 Pending | 59 Skipped
--- FAIL: TestE2e (35.43s)
FAIL
FAIL    github.com/ovn-org/ovn-kubernetes/test/e2e  35.454s
FAIL
[surya@hidden-temple e2e]$ 
[surya@hidden-temple e2e]$ 
[surya@hidden-temple e2e]$ 
[surya@hidden-temple e2e]$ 
[surya@hidden-temple e2e]$ /usr/bin/go test  -run ^TestE2e$ github.com/ovn-org/ovn-kubernetes/test/e2e
I0927 14:07:10.361055 3668373 test_context.go:457] Tolerating taints "node-role.kubernetes.io/master" when considering if nodes are ready
Sep 27 14:07:10.361: INFO: The --provider flag is not set. Continuing as if --provider=skeleton had been used.
Running Suite: E2e Suite
========================
Random Seed: 1632744430 - Will randomize all specs
Will run 1 of 60 specs

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
e2e multiple external gateway validation Should validate TCP/UDP connectivity to multiple external gateways for a UDP / TCP scenario 
  IPV4 tcp
  /home/surya/go/pkg/mod/github.com/onsi/ginkgo@v1.12.0/extensions/table/table_entry.go:41
[BeforeEach] e2e multiple external gateway validation
  /home/surya/go/pkg/mod/k8s.io/kubernetes@v1.20.2/test/e2e/framework/framework.go:174
STEP: Creating a kubernetes client
Sep 27 14:07:10.361: INFO: >>> kubeConfig: /home/surya/admin.conf
STEP: Building a namespace api object, basename novxlan-externalgw-ecmp
Sep 27 14:07:10.396: INFO: No PodSecurityPolicies found; assuming PodSecurityPolicy is disabled.
STEP: Waiting for a default service account to be provisioned in namespace
[BeforeEach] e2e multiple external gateway validation
  /home/surya/go/src/github.com/ovn-org/ovn-kubernetes/test/e2e/external_gateways.go:239
STEP: Creating the gateway containers for the icmp test
Sep 27 14:07:11.473: INFO: target ips are [10.249.10.1 10.249.10.2 10.249.10.3 10.249.10.4 10.249.10.5 10.249.10.6 10.249.10.7 10.249.10.8 10.249.10.9 10.249.10.10 10.249.10.11 10.249.10.12 10.249.10.13 10.249.10.14 10.249.10.15 10.249.10.16 10.249.10.17 10.249.10.18 10.249.10.19 10.249.10.20]
Sep 27 14:07:11.473: INFO: target ipsv6 are [fc00:f853:ccd:e794::1 fc00:f853:ccd:e794::2 fc00:f853:ccd:e794::3 fc00:f853:ccd:e794::4 fc00:f853:ccd:e794::5 fc00:f853:ccd:e794::6 fc00:f853:ccd:e794::7 fc00:f853:ccd:e794::8 fc00:f853:ccd:e794::9 fc00:f853:ccd:e794::10 fc00:f853:ccd:e794::11 fc00:f853:ccd:e794::12 fc00:f853:ccd:e794::13 fc00:f853:ccd:e794::14 fc00:f853:ccd:e794::15 fc00:f853:ccd:e794::16 fc00:f853:ccd:e794::17 fc00:f853:ccd:e794::18 fc00:f853:ccd:e794::19 fc00:f853:ccd:e794::20]
Sep 27 14:07:11.554: INFO: the pod side node is ovn-control-plane and the source node ip is 172.18.0.4 - fc00:f853:ccd:e793::4
STEP: Creating the source pod to reach the destination ips from
Sep 27 14:07:15.570: INFO: the pod source pod ip(s) are 10.244.2.4 - 
STEP: Setting up the destination ips to gw-test-container1
STEP: Adding a route from gw-test-container1 to the src pod
STEP: Setting up the listeners on the gateway
STEP: Setting up the destination ips to gw-test-container2
STEP: Adding a route from gw-test-container2 to the src pod
STEP: Setting up the listeners on the gateway
STEP: Resetting the gw annotation
Sep 27 14:07:31.996: INFO: Running '/usr/local/bin/kubectl --server=https://192.168.18.13:11337 --kubeconfig=/home/surya/admin.conf --namespace=novxlan-externalgw-ecmp-8081 annotate namespace novxlan-externalgw-ecmp-8081 k8s.ovn.org/routing-external-gws-'
Sep 27 14:07:32.063: INFO: stderr: ""
Sep 27 14:07:32.063: INFO: stdout: "namespace/novxlan-externalgw-ecmp-8081 annotated\n"
[It] IPV4 tcp
  /home/surya/go/pkg/mod/github.com/onsi/ginkgo@v1.12.0/extensions/table/table_entry.go:41
Sep 27 14:07:32.063: INFO: Annotating the external gateway test namespace to container gateways: 172.18.0.5,172.18.0.6
Sep 27 14:07:32.063: INFO: Running '/usr/local/bin/kubectl --server=https://192.168.18.13:11337 --kubeconfig=/home/surya/admin.conf --namespace=novxlan-externalgw-ecmp-8081 annotate namespace novxlan-externalgw-ecmp-8081 k8s.ovn.org/routing-external-gws=172.18.0.5,172.18.0.6'
Sep 27 14:07:32.119: INFO: stderr: ""
Sep 27 14:07:32.119: INFO: stdout: "namespace/novxlan-externalgw-ecmp-8081 annotated\n"
STEP: Checking that gw-test-container1 can reach the pod
STEP: Checking that gw-test-container2 can reach the pod
Sep 27 14:07:34.111: INFO: Hostname for gw-test-container1 is e5d83a1d4b02

Sep 27 14:07:34.226: INFO: Hostname for gw-test-container2 is 643faeac3a1a

Sep 27 14:07:34.226: INFO: Expected hostnames are map[643faeac3a1a:{} e5d83a1d4b02:{}]
Sep 27 14:07:34.226: INFO: Running '/usr/local/bin/kubectl --server=https://192.168.18.13:11337 --kubeconfig=/home/surya/admin.conf --namespace=novxlan-externalgw-ecmp-8081 exec e2e-exgw-src-pod -- bash -c echo | nc -w 1 10.249.10.1 80'
Sep 27 14:07:34.352: INFO: stderr: ""
Sep 27 14:07:34.352: INFO: stdout: "e5d83a1d4b02\n"
Sep 27 14:07:34.352: INFO: Running '/usr/local/bin/kubectl --server=https://192.168.18.13:11337 --kubeconfig=/home/surya/admin.conf --namespace=novxlan-externalgw-ecmp-8081 exec e2e-exgw-src-pod -- bash -c echo | nc -w 1 10.249.10.1 80'
Sep 27 14:07:34.494: INFO: stderr: ""
Sep 27 14:07:34.494: INFO: stdout: "643faeac3a1a\n"
Sep 27 14:07:34.494: INFO: Received hostnames for protocol tcp are map[643faeac3a1a:{} e5d83a1d4b02:{}] 
[AfterEach] e2e multiple external gateway validation
  /home/surya/go/pkg/mod/k8s.io/kubernetes@v1.20.2/test/e2e/framework/framework.go:175
Sep 27 14:07:34.494: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
STEP: Destroying namespace "novxlan-externalgw-ecmp-8081" for this suite.
[AfterEach] e2e multiple external gateway validation
  /home/surya/go/src/github.com/ovn-org/ovn-kubernetes/test/e2e/external_gateways.go:261

• [SLOW TEST:24.587 seconds]
e2e multiple external gateway validation
/home/surya/go/src/github.com/ovn-org/ovn-kubernetes/test/e2e/external_gateways.go:223
  Should validate TCP/UDP connectivity to multiple external gateways for a UDP / TCP scenario
  /home/surya/go/pkg/mod/github.com/onsi/ginkgo@v1.12.0/extensions/table/table.go:92
    IPV4 tcp
    /home/surya/go/pkg/mod/github.com/onsi/ginkgo@v1.12.0/extensions/table/table_entry.go:41
------------------------------
SSSSSSSSSSSSSSSS
Ran 1 of 60 Specs in 24.588 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 59 Skipped
PASS | FOCUSED
FAIL    github.com/ovn-org/ovn-kubernetes/test/e2e  24.615s
FAIL
[surya@hidden-temple e2e]$
github-actions[bot] commented 1 month ago

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 5 days.

github-actions[bot] commented 1 month ago

This issue was closed because it has been stalled for 5 days with no activity.

oshoval commented 1 month ago

happened here one time https://github.com/ovn-org/ovn-kubernetes/actions/runs/10301096279/job/28513027518?pr=4590 assuming not related to the PR