kubeovn / kube-ovn

A Bridge between SDN and Cloud Native (Project under CNCF)
https://kubeovn.github.io/docs/stable/en/
Apache License 2.0
1.97k stars 450 forks source link

subnet e2e fails occasionally #2392

Closed zhangzujian closed 1 year ago

zhangzujian commented 1 year ago
• [FAILED] [8.089 seconds]
[CNI:Kube-OVN] [group:subnet]
/home/runner/work/kube-ovn/kube-ovn/test/e2e/framework/framework.go:139
  [It] should support subnet AvailableIPRange and UsingIPRange is correct when restart deployment [Conformance]
  /home/runner/work/kube-ovn/kube-ovn/test/e2e/kube-ovn/subnet/subnet.go:727

  Begin Captured StdOut/StdErr Output >>
    delete deployment deployment-8733 failed with err <nil>
  << End Captured StdOut/StdErr Output

  Begin Captured GinkgoWriter Output >>
    [BeforeEach] [CNI:Kube-OVN] [group:subnet]
      set up framework | framework.go:178
    STEP: Creating a kubernetes client 02/28/23 06:53:59.865
    Feb 28 06:53:59.865: INFO: >>> kubeConfig: /home/runner/.kube/config
    STEP: Building a namespace api object, basename subnet 02/28/23 06:53:59.866
    STEP: Waiting for a default service account to be provisioned in namespace 02/28/23 06:53:59.893
    STEP: Waiting for kube-root-ca.crt to be provisioned in namespace 02/28/23 06:53:59.896
    [BeforeEach] [CNI:Kube-OVN] [group:subnet]
      /home/runner/work/kube-ovn/kube-ovn/test/e2e/framework/framework.go:58
    STEP: Setting kubernetes context 02/28/23 06:53:59.898
    STEP: Creating a Kube-OVN client 02/28/23 06:53:59.898
    Feb 28 06:53:59.898: INFO: >>> kubeConfig: /home/runner/.kube/config
    [BeforeEach] [CNI:Kube-OVN] [group:subnet]
      /home/runner/work/kube-ovn/kube-ovn/test/e2e/kube-ovn/subnet/subnet.go:43
    [It] should support subnet AvailableIPRange and UsingIPRange is correct when restart deployment [Conformance]
      /home/runner/work/kube-ovn/kube-ovn/test/e2e/kube-ovn/subnet/subnet.go:727
    STEP: Creating subnet subnet-7759 02/28/23 06:53:59.899
    Feb 28 06:53:59.904: INFO: Waiting up to 2m0s for subnet subnet-7759 condition Ready to be true
    Feb 28 06:53:59.907: INFO: Couldn't find condition Ready on subnet subnet-7759
    Feb 28 06:54:01.919: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:0, Replicas:0, UpdatedReplicas:0, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:0, Conditions:[]v1.DeploymentCondition(nil), CollisionCount:(*int32)(nil)}
    Feb 28 06:54:03.922: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:5, UpdatedReplicas:5, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:5, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2023, time.February, 28, 6, 54, 1, 0, time.Local), LastTransitionTime:time.Date(2023, time.February, 28, 6, 54, 1, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2023, time.February, 28, 6, 54, 2, 0, time.Local), LastTransitionTime:time.Date(2023, time.February, 28, 6, 54, 1, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"deployment-8733-77bb6f7494\" is progressing."}}, CollisionCount:(*int32)(nil)}
    Feb 28 06:54:05.9[26](https://github.com/zhangzujian/kube-ovn/actions/runs/4290297851/jobs/7474222179#step:17:27): FAIL: Expected
        <string>: 
    to equal
        <string>: 10.99.177.2-10.99.177.6

    Full Stack Trace
    github.com/kubeovn/kube-ovn/test/e2e/kube-ovn/subnet.glob..func1.13.1({0x0, 0x0}, {0xc00065dea0, 0x19}, {0xc000992dd0, 0xb}, {0xc000992d80, 0xd}, 0x5)
        /home/runner/work/kube-ovn/kube-ovn/test/e2e/kube-ovn/subnet/subnet.go:755 +0x305
    github.com/kubeovn/kube-ovn/test/e2e/kube-ovn/subnet.glob..func1.13()
        /home/runner/work/kube-ovn/kube-ovn/test/e2e/kube-ovn/subnet/subnet.go:761 +0x943
    github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func3({0xa84a5e, 0xc000970780})
        /home/runner/go/pkg/mod/github.com/onsi/ginkgo/v2@v2.4.0/internal/node.go:449 +0x1b
    github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func2()
        /home/runner/go/pkg/mod/github.com/onsi/ginkgo/v2@v2.4.0/internal/suite.go:750 +0x98
    created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode
        /home/runner/go/pkg/mod/github.com/onsi/ginkgo/v2@v2.4.0/internal/suite.go:738 +0xe3d
    [AfterEach] [CNI:Kube-OVN] [group:subnet]
      /home/runner/work/kube-ovn/kube-ovn/test/e2e/kube-ovn/subnet/subnet.go:72
    STEP: Deleting deployment  02/[28](https://github.com/zhangzujian/kube-ovn/actions/runs/4290297851/jobs/7474222179#step:17:29)/23 06:54:05.926
    STEP: Deleting pods  02/28/23 06:54:05.9[29](https://github.com/zhangzujian/kube-ovn/actions/runs/4290297851/jobs/7474222179#step:17:30)
    STEP: Deleting subnet subnet-7759 02/28/23 06:54:05.929
    Feb 28 06:54:05.9[34](https://github.com/zhangzujian/kube-ovn/actions/runs/4290297851/jobs/7474222179#step:17:35): INFO: Waiting for subnet subnet-7759 to disappear
    Feb 28 06:54:05.947: INFO: Subnet subnet-7759 still exists
    Feb 28 06:54:07.948: INFO: Waiting for subnet subnet-7759 to disappear
    Feb 28 06:54:07.951: INFO: Subnet subnet-7759 no longer exists
    [DeferCleanup (Each)] [CNI:Kube-OVN] [group:subnet]
      dump namespaces | framework.go:196
    STEP: dump namespace information after failure 02/28/23 06:54:07.951
    [DeferCleanup (Each)] [CNI:Kube-OVN] [group:subnet]
      tear down framework | framework.go:193
    STEP: Destroying namespace "subnet-2582" for this suite. 02/28/23 06:54:07.9[51](https://github.com/zhangzujian/kube-ovn/actions/runs/4290297851/jobs/7474222179#step:17:52)
  << End Captured GinkgoWriter Output

  Feb 28 06:[54](https://github.com/zhangzujian/kube-ovn/actions/runs/4290297851/jobs/7474222179#step:17:55):05.926: Expected
      <string>: 
  to equal
      <string>: 10.99.177.2-10.99.177.6
  In [It] at: /home/runner/work/kube-ovn/kube-ovn/test/e2e/kube-ovn/subnet/subnet.go:7[55](https://github.com/zhangzujian/kube-ovn/actions/runs/4290297851/jobs/7474222179#step:17:56)
zhangzujian commented 1 year ago
• [FAILED] [425.256 seconds]
[CNI:Kube-OVN] [group:underlay]
/home/runner/work/kube-ovn/kube-ovn/test/e2e/framework/framework.go:143
  [It] should support underlay to overlay subnet interconnection  [Conformance]
  /home/runner/work/kube-ovn/kube-ovn/test/e2e/kube-ovn/underlay/underlay.go:438

  Begin Captured GinkgoWriter Output >>
    [BeforeEach] [CNI:Kube-OVN] [group:underlay]
      set up framework | framework.go:178
    STEP: Creating a kubernetes client 03/01/23 08:04:26.542
    Mar  1 08:04:26.542: INFO: >>> kubeConfig: /home/runner/.kube/config
    STEP: Building a namespace api object, basename underlay 03/01/23 08:04:26.543
    STEP: Waiting for a default service account to be provisioned in namespace 03/01/23 08:04:26.563
    STEP: Waiting for kube-root-ca.crt to be provisioned in namespace 03/01/23 08:04:26.567
    [BeforeEach] [CNI:Kube-OVN] [group:underlay]
      /home/runner/work/kube-ovn/kube-ovn/test/e2e/framework/framework.go:58
    STEP: Setting kubernetes context 03/01/23 08:04:26.57
    [BeforeEach] [CNI:Kube-OVN] [group:underlay]
      /home/runner/work/kube-ovn/kube-ovn/test/e2e/kube-ovn/underlay/underlay.go:66
    STEP: Getting kind nodes 03/01/23 08:04:26.57
    STEP: Connecting nodes to the docker network 03/01/23 08:04:26.572
    STEP: Getting node links that belong to the docker network 03/01/23 08:04:27.07
    Mar  1 08:04:27.072: INFO: Executing command "ip -d -j address show" in container 5b68e665248b8b8473e3b5508effa679bcb6b9b0ae1d6f41426b50585cf4a820
    Mar  1 08:04:27.202: INFO: stdout: [{"ifindex":1,"ifname":"lo","flags":["LOOPBACK","UP","LOWER_UP"],"mtu":65536,"qdisc":"noqueue","operstate":"UNKNOWN","group":"default","txqlen":1000,"link_type":"loopback","address":"00:00:00:00:00:00","broadcast":"00:00:00:00:00:00","promiscuity":0,"min_mtu":0,"max_mtu":0,"num_tx_queues":1,"num_rx_queues":1,"gso_max_size":65536,"gso_max_segs":65535,"addr_info":[{"family":"inet","local":"127.0.0.1","prefixlen":8,"scope":"host","label":"lo","valid_life_time":4294967295,"preferred_life_time":4294967295},{"family":"inet6","local":"::1","prefixlen":128,"scope":"host","valid_life_time":4294967295,"preferred_life_time":4294967295}]},{"ifindex":2,"ifname":"kube-ipvs0","flags":["BROADCAST","NOARP"],"mtu":1500,"qdisc":"noop","operstate":"DOWN","group":"default","link_type":"ether","address":"c6:e5:55:46:89:5f","broadcast":"ff:ff:ff:ff:ff:ff","promiscuity":0,"min_mtu":0,"max_mtu":0,"linkinfo":{"info_kind":"dummy"},"num_tx_queues":1,"num_rx_queues":1,"gso_max_size":65536,"gso_max_segs":65535,"addr_info":[{"family":"inet6","local":"fd00:10:96::bd57","prefixlen":128,"scope":"global","valid_life_time":4294967295,"preferred_life_time":4294967295},{"family":"inet6","local":"fd00:10:96::e647","prefixlen":128,"scope":"global","valid_life_time":4294967295,"preferred_life_time":4294967295},{"family":"inet6","local":"fd00:10:96::6e3","prefixlen":128,"scope":"global","valid_life_time":4294967295,"preferred_life_time":4294967295},{"family":"inet6","local":"fd00:10:96::46e","prefixlen":128,"scope":"global","valid_life_time":4294967295,"preferred_life_time":4294967295},{"family":"inet6","local":"fd00:10:96::13a6","prefixlen":128,"scope":"global","valid_life_time":4294967295,"preferred_life_time":4294967295},{"family":"inet6","local":"fd00:10:96::9c47","prefixlen":128,"scope":"global","valid_life_time":4294967295,"preferred_life_time":4294967295},{"family":"inet6","local":"fd00:10:96::89c","prefixlen":128,"scope":"global","valid_life_time":4294967295,"preferred_life_time":4294967295},{"family":"inet6","local":"fd00:10:96::a","prefixlen":128,"scope":"global","valid_life_time":4294967295,"preferred_life_time":4294967295},{"family":"inet6","local":"fd00:10:96::1","prefixlen":128,"scope":"global","valid_life_time":4294967295,"preferred_life_time":4294967295}]},{"ifindex":3,"ifname":"ovs-system","flags":["BROADCAST","MULTICAST"],"mtu":1500,"qdisc":"noop","operstate":"DOWN","group":"default","txqlen":1000,"link_type":"ether","address":"b2:e2:3f:cf:1e:ce","broadcast":"ff:ff:ff:ff:ff:ff","promiscuity":1,"min_mtu":68,"max_mtu":65535,"linkinfo":{"info_kind":"openvswitch"},"num_tx_queues":1,"num_rx_queues":1,"gso_max_size":65536,"gso_max_segs":65535,"addr_info":[]},{"ifindex":4,"ifname":"br-int","flags":["BROADCAST","MULTICAST"],"mtu":1380,"qdisc":"noop","operstate":"DOWN","group":"default","txqlen":1000,"link_type":"ether","address":"2a:5f:42:3a:f8:f8","broadcast":"ff:ff:ff:ff:ff:ff","promiscuity":1,"min_mtu":68,"max_mtu":65535,"linkinfo":{"info_kind":"openvswitch"},"num_tx_queues":1,"num_rx_queues":1,"gso_max_size":65536,"gso_max_segs":65535,"addr_info":[]},{"ifindex":5,"ifname":"genev_sys_6081","flags":["BROADCAST","MULTICAST","UP","LOWER_UP"],"mtu":65000,"qdisc":"noqueue","master":"ovs-system","operstate":"UNKNOWN","group":"default","txqlen":1000,"link_type":"ether","address":"de:97:70:9b:6d:83","broadcast":"ff:ff:ff:ff:ff:ff","promiscuity":1,"min_mtu":68,"max_mtu":65465,"linkinfo":{"info_kind":"geneve","info_data":{"external":true,"id":0,"ttl":0,"df":"unset","port":6081,"udp_zero_csum6_rx":true},"info_slave_kind":"openvswitch"},"num_tx_queues":1,"num_rx_queues":1,"gso_max_size":65536,"gso_max_segs":65535,"addr_info":[{"family":"inet6","local":"fe80::dc97:70ff:fe9b:6d83","prefixlen":64,"scope":"link","valid_life_time":4294967295,"preferred_life_time":4294967295}]},{"ifindex":6,"ifname":"mirror0","flags":["BROADCAST","MULTICAST","UP","LOWER_UP"],"mtu":1380,"qdisc":"noqueue","operstate":"UNKNOWN","group":"default","txqlen":1000,"link_type":"ether","address":"66:fa:b4:7b:60:b0","broadcast":"ff:ff:ff:ff:ff:ff","promiscuity":1,"min_mtu":68,"max_mtu":65535,"linkinfo":{"info_kind":"openvswitch"},"num_tx_queues":1,"num_rx_queues":1,"gso_max_size":65536,"gso_max_segs":65535,"addr_info":[{"family":"inet6","local":"fe80::64fa:b4ff:fe7b:60b0","prefixlen":64,"scope":"link","valid_life_time":4294967295,"preferred_life_time":4294967295}]},{"ifindex":7,"link_index":8,"ifname":"eth0","flags":["BROADCAST","MULTICAST","UP","LOWER_UP"],"mtu":1500,"qdisc":"noqueue","operstate":"UP","group":"default","link_type":"ether","address":"02:42:ac:12:00:03","broadcast":"ff:ff:ff:ff:ff:ff","link_netnsid":0,"promiscuity":0,"min_mtu":68,"max_mtu":65535,"linkinfo":{"info_kind":"veth"},"num_tx_queues":2,"num_rx_queues":2,"gso_max_size":65536,"gso_max_segs":65535,"addr_info":[{"family":"inet","local":"172.18.0.3","prefixlen":16,"broadcast":"172.18.255.255","scope":"global","label":"eth0","valid_life_time":4294967295,"preferred_life_time":4294967295},{"family":"inet6","local":"fc00:f853:ccd:e793::3","prefixlen":64,"scope":"global","nodad":true,"valid_life_time":4294967295,"preferred_life_time":4294967295},{"family":"inet6","local":"fe80::42:acff:fe12:3","prefixlen":64,"scope":"link","valid_life_time":4294967295,"preferred_life_time":4294967295}]},{"ifindex":8,"ifname":"ovn0","flags":["BROADCAST","MULTICAST","UP","LOWER_UP"],"mtu":1380,"qdisc":"noqueue","operstate":"UNKNOWN","group":"default","txqlen":1000,"link_type":"ether","address":"00:00:00:23:b0:85","broadcast":"ff:ff:ff:ff:ff:ff","promiscuity":1,"min_mtu":68,"max_mtu":65535,"linkinfo":{"info_kind":"openvswitch"},"num_tx_queues":1,"num_rx_queues":1,"gso_max_size":65536,"gso_max_segs":65535,"addr_info":[{"family":"inet6","local":"fd00:100:64::3","prefixlen":64,"scope":"global","valid_life_time":4294967295,"preferred_life_time":4294967295},{"family":"inet6","local":"fe80::200:ff:fe23:b085","prefixlen":64,"scope":"link","valid_life_time":4294967295,"preferred_life_time":4294967295}]},{"ifindex":12,"link_index":11,"ifname":"281af2aa6a64_h","flags":["BROADCAST","MULTICAST","UP","LOWER_UP"],"mtu":1380,"qdisc":"noqueue","master":"ovs-system","operstate":"UP","group":"default","txqlen":1000,"link_type":"ether","address":"02:e9:55:33:1c:7c","broadcast":"ff:ff:ff:ff:ff:ff","link_netnsid":2,"promiscuity":1,"min_mtu":68,"max_mtu":65535,"linkinfo":{"info_kind":"veth","info_slave_kind":"openvswitch"},"num_tx_queues":2,"num_rx_queues":2,"gso_max_size":65536,"gso_max_segs":65535,"addr_info":[{"family":"inet6","local":"fe80::e9:55ff:fe33:1c7c","prefixlen":64,"scope":"link","valid_life_time":4294967295,"preferred_life_time":4294967295}]},{"ifindex":14,"link_index":13,"ifname":"83b14b6f3a5a_h","flags":["BROADCAST","MULTICAST","UP","LOWER_UP"],"mtu":1380,"qdisc":"noqueue","master":"ovs-system","operstate":"UP","group":"default","txqlen":1000,"link_type":"ether","address":"9a:d2:ec:26:e4:db","broadcast":"ff:ff:ff:ff:ff:ff","link_netnsid":3,"promiscuity":1,"min_mtu":68,"max_mtu":65535,"linkinfo":{"info_kind":"veth","info_slave_kind":"openvswitch"},"num_tx_queues":2,"num_rx_queues":2,"gso_max_size":65536,"gso_max_segs":65535,"addr_info":[{"family":"inet6","local":"fe80::98d2:ecff:fe26:e4db","prefixlen":64,"scope":"link","valid_life_time":4294967295,"preferred_life_time":4294967295}]},{"ifindex":16,"link_index":15,"ifname":"ca4c76eab877_h","flags":["BROADCAST","MULTICAST","UP","LOWER_UP"],"mtu":1380,"qdisc":"noqueue","master":"ovs-system","operstate":"UP","group":"default","txqlen":1000,"link_type":"ether","address":"82:e5:fc:a9:de:dd","broadcast":"ff:ff:ff:ff:ff:ff","link_netnsid":1,"promiscuity":1,"min_mtu":68,"max_mtu":65535,"linkinfo":{"info_kind":"veth","info_slave_kind":"openvswitch"},"num_tx_queues":2,"num_rx_queues":2,"gso_max_size":65536,"gso_max_segs":65535,"addr_info":[{"family":"inet6","local":"fe80::80e5:fcff:fea9:dedd","prefixlen":64,"scope":"link","valid_life_time":4294967295,"preferred_life_time":4294967295}]},{"ifindex":18,"link_index":19,"ifname":"eth3","flags":["BROADCAST","MULTICAST","UP","LOWER_UP"],"mtu":1500,"qdisc":"noqueue","operstate":"UP","group":"default","link_type":"ether","address":"02:42:ac:13:00:02","broadcast":"ff:ff:ff:ff:ff:ff","link_netnsid":0,"promiscuity":0,"min_mtu":68,"max_mtu":65535,"linkinfo":{"info_kind":"veth"},"num_tx_queues":2,"num_rx_queues":2,"gso_max_size":65536,"gso_max_segs":65535,"addr_info":[{"family":"inet","local":"172.19.0.2","prefixlen":16,"broadcast":"172.19.255.255","scope":"global","label":"eth3","valid_life_time":4294967295,"preferred_life_time":4294967295},{"family":"inet6","local":"fc00:adb1:b29b:608d::2","prefixlen":64,"scope":"global","nodad":true,"valid_life_time":4294967295,"preferred_life_time":4294967295},{"family":"inet6","local":"fe80::42:acff:fe13:2","prefixlen":64,"scope":"link","tentative":true,"valid_life_time":4294967295,"preferred_life_time":4294967295}]}]

    Mar  1 08:04:27.202: INFO: stderr: 
    Mar  1 08:04:27.203: INFO: Executing command "ip -d -j route show" in container 5b68e665248b8b8473e3b5508effa679bcb6b9b0ae1d6f41426b50585cf4a820
    Mar  1 08:04:27.300: INFO: stdout: [{"type":"unicast","dst":"default","gateway":"172.18.0.1","dev":"eth0","protocol":"boot","scope":"global","flags":[]},{"type":"unicast","dst":"172.18.0.0/16","dev":"eth0","protocol":"kernel","scope":"link","prefsrc":"172.18.0.3","flags":[]},{"type":"unicast","dst":"172.19.0.0/16","dev":"eth3","protocol":"kernel","scope":"link","prefsrc":"172.19.0.2","flags":[]}]

    Mar  1 08:04:27.300: INFO: stderr: 
    Mar  1 08:04:27.300: INFO: Executing command "ip -d -j -6 route show" in container 5b68e665248b8b8473e3b5508effa679bcb6b9b0ae1d6f41426b50585cf4a820
    Mar  1 08:04:27.417: INFO: stdout: [{"type":"unicast","dst":"fc00:adb1:b29b:608d::/64","dev":"eth3","protocol":"kernel","scope":"global","metric":256,"flags":[],"pref":"medium"},{"type":"unicast","dst":"fc00:f853:ccd:e793::/64","dev":"eth0","protocol":"kernel","scope":"global","metric":256,"flags":[],"pref":"medium"},{"type":"unicast","dst":"fd00:10:16::/64","gateway":"fd00:100:64::1","dev":"ovn0","protocol":"boot","scope":"global","prefsrc":"fc00:f853:ccd:e793::3","metric":1024,"flags":[],"pref":"medium"},{"type":"unicast","dst":"fd00:100:64::/64","dev":"ovn0","protocol":"kernel","scope":"global","metric":256,"flags":[],"pref":"medium"},{"type":"unicast","dst":"fe80::/64","dev":"eth0","protocol":"kernel","scope":"global","metric":256,"flags":[],"pref":"medium"},{"type":"unicast","dst":"fe80::/64","dev":"genev_sys_6081","protocol":"kernel","scope":"global","metric":256,"flags":[],"pref":"medium"},{"type":"unicast","dst":"fe80::/64","dev":"mirror0","protocol":"kernel","scope":"global","metric":256,"flags":[],"pref":"medium"},{"type":"unicast","dst":"fe80::/64","dev":"ovn0","protocol":"kernel","scope":"global","metric":256,"flags":[],"pref":"medium"},{"type":"unicast","dst":"fe80::/64","dev":"281af2aa6a64_h","protocol":"kernel","scope":"global","metric":256,"flags":[],"pref":"medium"},{"type":"unicast","dst":"fe80::/64","dev":"83b14b6f3a5a_h","protocol":"kernel","scope":"global","metric":256,"flags":[],"pref":"medium"},{"type":"unicast","dst":"fe80::/64","dev":"ca4c76eab877_h","protocol":"kernel","scope":"global","metric":256,"flags":[],"pref":"medium"},{"type":"unicast","dst":"fe80::/64","dev":"eth3","protocol":"kernel","scope":"global","metric":256,"flags":[],"pref":"medium"},{"type":"unicast","dst":"default","gateway":"fc00:f853:ccd:e793::1","dev":"eth0","protocol":"boot","scope":"global","metric":1024,"flags":[],"pref":"medium"}]

    Mar  1 08:04:27.417: INFO: stderr: 
    Mar  1 08:04:27.417: INFO: Executing command "ip -d -j address show" in container ca81da5e5f711f0fe5ee792cfb1a7c7f2c651bf0692ad141919ea7e7dcafefc6
    Mar  1 08:04:27.497: INFO: stdout: [{"ifindex":1,"ifname":"lo","flags":["LOOPBACK","UP","LOWER_UP"],"mtu":65536,"qdisc":"noqueue","operstate":"UNKNOWN","group":"default","txqlen":1000,"link_type":"loopback","address":"00:00:00:00:00:00","broadcast":"00:00:00:00:00:00","promiscuity":0,"min_mtu":0,"max_mtu":0,"num_tx_queues":1,"num_rx_queues":1,"gso_max_size":65536,"gso_max_segs":65535,"addr_info":[{"family":"inet","local":"127.0.0.1","prefixlen":8,"scope":"host","label":"lo","valid_life_time":4294967295,"preferred_life_time":4294967295},{"family":"inet6","local":"::1","prefixlen":128,"scope":"host","valid_life_time":4294967295,"preferred_life_time":4294967295}]},{"ifindex":2,"ifname":"kube-ipvs0","flags":["BROADCAST","NOARP"],"mtu":1500,"qdisc":"noop","operstate":"DOWN","group":"default","link_type":"ether","address":"c6:f3:b9:01:b2:6e","broadcast":"ff:ff:ff:ff:ff:ff","promiscuity":0,"min_mtu":0,"max_mtu":0,"linkinfo":{"info_kind":"dummy"},"num_tx_queues":1,"num_rx_queues":1,"gso_max_size":65536,"gso_max_segs":65535,"addr_info":[{"family":"inet6","local":"fd00:10:96::bd57","prefixlen":128,"scope":"global","valid_life_time":4294967295,"preferred_life_time":4294967295},{"family":"inet6","local":"fd00:10:96::e647","prefixlen":128,"scope":"global","valid_life_time":4294967295,"preferred_life_time":4294967295},{"family":"inet6","local":"fd00:10:96::6e3","prefixlen":128,"scope":"global","valid_life_time":4294967295,"preferred_life_time":4294967295},{"family":"inet6","local":"fd00:10:96::46e","prefixlen":128,"scope":"global","valid_life_time":4294967295,"preferred_life_time":4294967295},{"family":"inet6","local":"fd00:10:96::13a6","prefixlen":128,"scope":"global","valid_life_time":4294967295,"preferred_life_time":4294967295},{"family":"inet6","local":"fd00:10:96::9c47","prefixlen":128,"scope":"global","valid_life_time":4294967295,"preferred_life_time":4294967295},{"family":"inet6","local":"fd00:10:96::89c","prefixlen":128,"scope":"global","valid_life_time":4294967295,"preferred_life_time":4294967295},{"family":"inet6","local":"fd00:10:96::1","prefixlen":128,"scope":"global","valid_life_time":4294967295,"preferred_life_time":4294967295},{"family":"inet6","local":"fd00:10:96::a","prefixlen":128,"scope":"global","valid_life_time":4294967295,"preferred_life_time":4294967295}]},{"ifindex":3,"ifname":"ovs-system","flags":["BROADCAST","MULTICAST"],"mtu":1500,"qdisc":"noop","operstate":"DOWN","group":"default","txqlen":1000,"link_type":"ether","address":"a6:1c:c7:81:e1:52","broadcast":"ff:ff:ff:ff:ff:ff","promiscuity":1,"min_mtu":68,"max_mtu":65535,"linkinfo":{"info_kind":"openvswitch"},"num_tx_queues":1,"num_rx_queues":1,"gso_max_size":65536,"gso_max_segs":65535,"addr_info":[]},{"ifindex":4,"ifname":"br-int","flags":["BROADCAST","MULTICAST"],"mtu":1380,"qdisc":"noop","operstate":"DOWN","group":"default","txqlen":1000,"link_type":"ether","address":"be:0c:25:d8:e4:9c","broadcast":"ff:ff:ff:ff:ff:ff","promiscuity":1,"min_mtu":68,"max_mtu":65535,"linkinfo":{"info_kind":"openvswitch"},"num_tx_queues":1,"num_rx_queues":1,"gso_max_size":65536,"gso_max_segs":65535,"addr_info":[]},{"ifindex":5,"link_index":6,"ifname":"eth0","flags":["BROADCAST","MULTICAST","UP","LOWER_UP"],"mtu":1500,"qdisc":"noqueue","operstate":"UP","group":"default","link_type":"ether","address":"02:42:ac:12:00:02","broadcast":"ff:ff:ff:ff:ff:ff","link_netnsid":0,"promiscuity":0,"min_mtu":68,"max_mtu":65535,"linkinfo":{"info_kind":"veth"},"num_tx_queues":2,"num_rx_queues":2,"gso_max_size":65536,"gso_max_segs":65535,"addr_info":[{"family":"inet","local":"172.18.0.2","prefixlen":16,"broadcast":"172.18.255.255","scope":"global","label":"eth0","valid_life_time":4294967295,"preferred_life_time":4294967295},{"family":"inet6","local":"fc00:f853:ccd:e793::2","prefixlen":64,"scope":"global","nodad":true,"valid_life_time":4294967295,"preferred_life_time":4294967295},{"family":"inet6","local":"fe80::42:acff:fe12:2","prefixlen":64,"scope":"link","valid_life_time":4294967295,"preferred_life_time":4294967295}]},{"ifindex":6,"ifname":"mirror0","flags":["BROADCAST","MULTICAST","UP","LOWER_UP"],"mtu":1380,"qdisc":"noqueue","operstate":"UNKNOWN","group":"default","txqlen":1000,"link_type":"ether","address":"a2:91:00:eb:a3:a7","broadcast":"ff:ff:ff:ff:ff:ff","promiscuity":1,"min_mtu":68,"max_mtu":65535,"linkinfo":{"info_kind":"openvswitch"},"num_tx_queues":1,"num_rx_queues":1,"gso_max_size":65536,"gso_max_segs":65535,"addr_info":[{"family":"inet6","local":"fe80::a091:ff:feeb:a3a7","prefixlen":64,"scope":"link","valid_life_time":4294967295,"preferred_life_time":4294967295}]},{"ifindex":7,"ifname":"genev_sys_6081","flags":["BROADCAST","MULTICAST","UP","LOWER_UP"],"mtu":65000,"qdisc":"noqueue","master":"ovs-system","operstate":"UNKNOWN","group":"default","txqlen":1000,"link_type":"ether","address":"7a:98:c6:f9:eb:8b","broadcast":"ff:ff:ff:ff:ff:ff","promiscuity":1,"min_mtu":68,"max_mtu":65465,"linkinfo":{"info_kind":"geneve","info_data":{"external":true,"id":0,"ttl":0,"df":"unset","port":6081,"udp_zero_csum6_rx":true},"info_slave_kind":"openvswitch"},"num_tx_queues":1,"num_rx_queues":1,"gso_max_size":65536,"gso_max_segs":65535,"addr_info":[{"family":"inet6","local":"fe80::7898:c6ff:fef9:eb8b","prefixlen":64,"scope":"link","valid_life_time":4294967295,"preferred_life_time":4294967295}]},{"ifindex":8,"ifname":"ovn0","flags":["BROADCAST","MULTICAST","UP","LOWER_UP"],"mtu":1380,"qdisc":"noqueue","operstate":"UNKNOWN","group":"default","txqlen":1000,"link_type":"ether","address":"00:00:00:25:1f:92","broadcast":"ff:ff:ff:ff:ff:ff","promiscuity":1,"min_mtu":68,"max_mtu":65535,"linkinfo":{"info_kind":"openvswitch"},"num_tx_queues":1,"num_rx_queues":1,"gso_max_size":65536,"gso_max_segs":65535,"addr_info":[{"family":"inet6","local":"fd00:100:64::2","prefixlen":64,"scope":"global","valid_life_time":4294967295,"preferred_life_time":4294967295},{"family":"inet6","local":"fe80::200:ff:fe25:1f92","prefixlen":64,"scope":"link","valid_life_time":4294967295,"preferred_life_time":4294967295}]},{"ifindex":16,"link_index":15,"ifname":"c7cb1ab258c2_h","flags":["BROADCAST","MULTICAST","UP","LOWER_UP"],"mtu":1380,"qdisc":"noqueue","master":"ovs-system","operstate":"UP","group":"default","txqlen":1000,"link_type":"ether","address":"ba:65:05:c3:84:8b","broadcast":"ff:ff:ff:ff:ff:ff","link_netnsid":2,"promiscuity":1,"min_mtu":68,"max_mtu":65535,"linkinfo":{"info_kind":"veth","info_slave_kind":"openvswitch"},"num_tx_queues":2,"num_rx_queues":2,"gso_max_size":65536,"gso_max_segs":65535,"addr_info":[{"family":"inet6","local":"fe80::b865:5ff:fec3:848b","prefixlen":64,"scope":"link","valid_life_time":4294967295,"preferred_life_time":4294967295}]},{"ifindex":20,"link_index":21,"ifname":"eth3","flags":["BROADCAST","MULTICAST","UP","LOWER_UP"],"mtu":1500,"qdisc":"noqueue","operstate":"UP","group":"default","link_type":"ether","address":"02:42:ac:13:00:03","broadcast":"ff:ff:ff:ff:ff:ff","link_netnsid":0,"promiscuity":0,"min_mtu":68,"max_mtu":65535,"linkinfo":{"info_kind":"veth"},"num_tx_queues":2,"num_rx_queues":2,"gso_max_size":65536,"gso_max_segs":65535,"addr_info":[{"family":"inet","local":"172.19.0.3","prefixlen":16,"broadcast":"172.19.255.255","scope":"global","label":"eth3","valid_life_time":4294967295,"preferred_life_time":4294967295},{"family":"inet6","local":"fc00:adb1:b29b:608d::3","prefixlen":64,"scope":"global","nodad":true,"valid_life_time":4294967295,"preferred_life_time":4294967295},{"family":"inet6","local":"fe80::42:acff:fe13:3","prefixlen":64,"scope":"link","tentative":true,"valid_life_time":4294967295,"preferred_life_time":4294967295}]}]

    Mar  1 08:04:27.497: INFO: stderr: 
    Mar  1 08:04:27.498: INFO: Executing command "ip -d -j route show" in container ca81da5e5f711f0fe5ee792cfb1a7c7f2c651bf0692ad141919ea7e7dcafefc6
    Mar  1 08:04:27.556: INFO: stdout: [{"type":"unicast","dst":"default","gateway":"172.18.0.1","dev":"eth0","protocol":"boot","scope":"global","flags":[]},{"type":"unicast","dst":"172.18.0.0/16","dev":"eth0","protocol":"kernel","scope":"link","prefsrc":"172.18.0.2","flags":[]},{"type":"unicast","dst":"172.19.0.0/16","dev":"eth3","protocol":"kernel","scope":"link","prefsrc":"172.19.0.3","flags":[]}]

    Mar  1 08:04:27.556: INFO: stderr: 
    Mar  1 08:04:27.556: INFO: Executing command "ip -d -j -6 route show" in container ca81da5e5f711f0fe5ee792cfb1a7c7f2c651bf0692ad141919ea7e7dcafefc6
    Mar  1 08:04:27.616: INFO: stdout: [{"type":"unicast","dst":"fc00:adb1:b29b:608d::/64","dev":"eth3","protocol":"kernel","scope":"global","metric":256,"flags":[],"pref":"medium"},{"type":"unicast","dst":"fc00:f853:ccd:e793::/64","dev":"eth0","protocol":"kernel","scope":"global","metric":256,"flags":[],"pref":"medium"},{"type":"unicast","dst":"fd00:10:16::/64","gateway":"fd00:100:64::1","dev":"ovn0","protocol":"boot","scope":"global","prefsrc":"fc00:f853:ccd:e793::2","metric":1024,"flags":[],"pref":"medium"},{"type":"unicast","dst":"fd00:100:64::/64","dev":"ovn0","protocol":"kernel","scope":"global","metric":256,"flags":[],"pref":"medium"},{"type":"unicast","dst":"fe80::/64","dev":"eth0","protocol":"kernel","scope":"global","metric":256,"flags":[],"pref":"medium"},{"type":"unicast","dst":"fe80::/64","dev":"mirror0","protocol":"kernel","scope":"global","metric":256,"flags":[],"pref":"medium"},{"type":"unicast","dst":"fe80::/64","dev":"genev_sys_6081","protocol":"kernel","scope":"global","metric":256,"flags":[],"pref":"medium"},{"type":"unicast","dst":"fe80::/64","dev":"ovn0","protocol":"kernel","scope":"global","metric":256,"flags":[],"pref":"medium"},{"type":"unicast","dst":"fe80::/64","dev":"c7cb1ab258c2_h","protocol":"kernel","scope":"global","metric":256,"flags":[],"pref":"medium"},{"type":"unicast","dst":"fe80::/64","dev":"eth3","protocol":"kernel","scope":"global","metric":256,"flags":[],"pref":"medium"},{"type":"unicast","dst":"default","gateway":"fc00:f853:ccd:e793::1","dev":"eth0","protocol":"boot","scope":"global","metric":1024,"flags":[],"pref":"medium"}]

    Mar  1 08:04:27.616: INFO: stderr: 
    [It] should support underlay to overlay subnet interconnection  [Conformance]
      /home/runner/work/kube-ovn/kube-ovn/test/e2e/kube-ovn/underlay/underlay.go:438
    STEP: Creating provider network 03/01/23 08:04:27.616
    STEP: Getting docker network kube-ovn-vlan 03/01/23 08:04:41.69
    STEP: Creating vlan vlan-12908 03/01/23 08:04:41.692
    STEP: Creating underlay subnet subnet-15715 03/01/23 08:04:41.707
    Mar  1 08:04:41.710: INFO: Waiting up to 2m0s for subnet subnet-15715 condition Ready to be true
    Mar  1 08:04:41.731: INFO: Couldn't find condition Ready on subnet subnet-15715
    STEP: Creating underlay subnet pod 03/01/23 08:04:43.738
    Mar  1 08:04:43.744: INFO: Waiting up to 5m0s for pod "pod-14919" in namespace "underlay-4966" to be "running and ready"
    Mar  1 08:04:43.753: INFO: Pod "pod-14919": Phase="Pending", Reason="", readiness=false. Elapsed: 9.119118ms
    Mar  1 08:04:43.753: INFO: The phase of Pod pod-14919 is Pending, waiting for it to be Running (with Ready = true)
    Mar  1 08:04:45.758: INFO: Pod "pod-14919": Phase="Pending", Reason="", readiness=false. Elapsed: 2.014028252s
    Mar  1 08:04:45.758: INFO: The phase of Pod pod-14919 is Pending, waiting for it to be Running (with Ready = true)
    Mar  1 08:04:47.758: INFO: Pod "pod-14919": Phase="Pending", Reason="", readiness=false. Elapsed: 4.014082531s
    Mar  1 08:04:47.758: INFO: The phase of Pod pod-14919 is Pending, waiting for it to be Running (with Ready = true)
    Mar  1 08:09:43.757: INFO: Pod "pod-14919": Phase="Pending", Reason="", readiness=false. Elapsed: 5m0.013165713s
    Mar  1 08:09:43.757: INFO: The phase of Pod pod-14919 is Pending, waiting for it to be Running (with Ready = true)
    Mar  1 08:09:43.759: INFO: Pod "pod-14919": Phase="Pending", Reason="", readiness=false. Elapsed: 5m0.015273223s
    Mar  1 08:09:43.759: INFO: The phase of Pod pod-14919 is Pending, waiting for it to be Running (with Ready = true)
    Mar  1 08:09:43.760: INFO: Unexpected error: 
        <*pod.timeoutError | 0xc0013f1e90>: {
            msg: "timed out while waiting for pod underlay-4966/pod-14919 to be running and ready",
            observedObjects: [
                <*v1.Pod | 0xc000ea4d80>{
                    TypeMeta: {Kind: "", APIVersion: ""},
                    ObjectMeta: {
                        Name: "pod-14919",
                        GenerateName: "",
                        Namespace: "underlay-4966",
                        SelfLink: "",
                        UID: "16e2aeee-cb39-4128-b203-b697c2961cba",
                        ResourceVersion: "3954",
                        Generation: 0,
                        CreationTimestamp: {
                            Time: {
                                wall: 0,
                                ext: 63813254683,
                                loc: {
                                    name: "Local",
                                    zone: [
                                        {name: "UTC", offset: 0, isDST: false},
                                    ],
                                    tx: [
                                        {
                                            when: -9223372036854775808,
                                            index: 0,
                                            isstd: false,
                                            isutc: false,
                                        },
                                    ],
                                    extend: "UTC0",
                                    cacheStart: 9223372036854775807,
                                    cacheEnd: 9223372036854775807,
                                    cacheZone: {name: "UTC", offset: 0, isDST: false},
                                },
                            },
                        },
                        DeletionTimestamp: nil,
                        DeletionGracePeriodSeconds: nil,
                        Labels: nil,
                        Annotations: {
                            "ovn.kubernetes.io/cidr": "fc00:adb1:b29b:608d::/64",
                            "ovn.kubernetes.io/ip_address": "fc00:adb1:b29b:608d::5",
                            "ovn.kubernetes.io/logical_router": "ovn-cluster",
                            "ovn.kubernetes.io/mac_address": "00:00:00:47:55:47",
                            "ovn.kubernetes.io/allocated": "true",
                            "ovn.kubernetes.io/gateway": "fc00:adb1:b29b:608d::1",
                            "ovn.kubernetes.io/logical_switch": "subnet-15715",
                            "ovn.kubernetes.io/pod_nic_type": "veth-pair",
                            "ovn.kubernetes.io/routed": "true",
                        },
                        OwnerReferences: nil,
                        Finalizers: nil,
                        ManagedFields: [
                            {
                                Manager: "kube-ovn-controller",
                                Operation: "Update",
                                APIVersion: "v1",
                                Time: {
                                    Time: {
                                        wall: 0,
                                        ext: 63813254683,
                                        loc: {
                                            name: "Local",
                                            zone: [...],
                                            tx: [...],
                                            extend: "UTC0",
                                            cacheStart: 9223372036854775807,
                                            cacheEnd: 9223372036854775807,
                                            cacheZone: {name: ..., offset: ..., isDST: ...},
                                        },
                                    },
                                },
                                FieldsType: "FieldsV1",
                                FieldsV1: {
                                    Raw: "{\"f:metadata\":{\"f:annotations\":{\"f:ovn.kubernetes.io/allocated\":{},\"f:ovn.kubernetes.io/cidr\":{},\"f:ovn.kubernetes.io/gateway\":{},\"f:ovn.kubernetes.io/ip_address\":{},\"f:ovn.kubernetes.io/logical_router\":{},\"f:ovn.kubernetes.io/mac_address\":{},\"f:ovn.kubernetes.io/pod_nic_type\":{},\"f:ovn.kubernetes.io/routed\":{}}}}",
                                },
                                Subresource: "",
                            },
                            {
                                Manager: "kube-ovn.test",
                                Operation: "Update",
                                APIVersion: "v1",
                                Time: {
                                    Time: {
                                        wall: 0,
                                        ext: 63813254683,
                                        loc: {
                                            name: "Local",
                                            zone: [...],
                                            tx: [...],
                                            extend: "UTC0",
                                            cacheStart: 9223372036854775807,
                                            cacheEnd: 9223372036854775807,
                                            cacheZone: {name: ..., offset: ..., isDST: ...},
                                        },
                                    },
                                },
                                FieldsType: "FieldsV1",
                                FieldsV1: {
                                    Raw: "{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:ovn.kubernetes.io/logical_switch\":{}}},\"f:spec\":{\"f:containers\":{\"k:{\\\"name\\\":\\\"container\\\"}\":{\".\":{},\"f:args\":{},\"f:image\":{},\"f:imagePullPolicy\":{},\"f:name\":{},\"f:resources\":{},\"f:terminationMessagePath\":{},\"f:terminationMessagePolicy\":{}}},\"f:dnsPolicy\":{},\"f:enableServiceLinks\":{},\"f:restartPolicy\":{},\"f:schedulerName\":{},\"f:securityContext\":{},\"f:terminationGracePeriodSeconds\":{}}}",
                                },
                                Subresource: "",
                            },
                            {
                                Manager: "kubelet",
                                Operation: "Update",
                                APIVersion: "v1",
                                Time: {
                                    Time: {
                                        wall: 0,
                                        ext: 63813254683,
                                        loc: {
                                            name: "Local",
                                            zone: [...],
                                            tx: [...],
                                            extend: "UTC0",
                                            cacheStart: 9223372036854775807,
                                            cacheEnd: 9223372036854775807,
                                            cacheZone: {name: ..., offset: ..., isDST: ...},
                                        },
                                    },
                                },
                                FieldsType: "FieldsV1",
                                FieldsV1: {
                                    Raw: "{\"f:status\":{\"f:conditions\":{\"k:{\\\"type\\\":\\\"ContainersReady\\\"}\":{\".\":{},\"f:lastProbeTime\":{},\"f:lastTransitionTime\":{},\"f:message\":{},\"f:reason\":{},\"f:status\":{},\"f:type\":{}},\"k:{\\\"type\\\":\\\"Initialized\\\"}\":{\".\":{},\"f:lastProbeTime\":{},\"f:lastTransitionTime\":{},\"f:status\":{},\"f:type\":{}},\"k:{\\\"type\\\":\\\"Ready\\\"}\":{\".\":{},\"f:lastProbeTime\":{},\"f:lastTransitionTime\":{},\"f:message\":{},\"f:reason\":{},\"f:status\":{},\"f:type\":{}}},\"f:containerStatuses\":{},\"f:hostIP\":{},\"f:startTime\":{}}}",
                                },
                                Subresource: "status",
                            },
                        ],
                    },
                    Spec: {
                        Volumes: [
                            {
                                Name: "kube-api-access-lr2gf",
                                VolumeSource: {
                                    HostPath: nil,
                                    EmptyDir: nil,
                                    GCEPersistentDisk: nil,
                                    AWSElasticBlockStore: nil,
                   ...

    Gomega truncated this representation as it exceeds 'format.MaxLength'.
    Consider having the object provide a custom 'GomegaStringer' representation
    or adjust the parameters in Gomega's 'format' package.

    Learn more here: https://onsi.github.io/gomega/#adjusting-output

    Mar  1 08:09:43.760: FAIL: timed out while waiting for pod underlay-4966/pod-14919 to be running and ready

    Full Stack Trace
    k8s.io/kubernetes/test/e2e/framework/pod.(*PodClient).CreateSync(0xc000e12cf0, 0xc0016a6000?)
        /home/runner/go/pkg/mod/k8s.io/kubernetes@v1.26.2/test/e2e/framework/pod/pod_client.go:107 +0x94
    github.com/kubeovn/kube-ovn/test/e2e/kube-ovn/underlay.glob..func1.7()
        /home/runner/work/kube-ovn/kube-ovn/test/e2e/kube-ovn/underlay/underlay.go:495 +0x10a5
    github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func3({0x0, 0x0})
        /home/runner/go/pkg/mod/github.com/onsi/ginkgo/v2@v2.4.0/internal/node.go:449 +0x1b
    github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func2()
        /home/runner/go/pkg/mod/github.com/onsi/ginkgo/v2@v2.4.0/internal/suite.go:750 +0x98
    created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode
        /home/runner/go/pkg/mod/github.com/onsi/ginkgo/v2@v2.4.0/internal/suite.go:738 +0xe3d
    [AfterEach] [CNI:Kube-OVN] [group:underlay]
      /home/runner/work/kube-ovn/kube-ovn/test/e2e/kube-ovn/underlay/underlay.go:260
    STEP: Deleting pod pod-17487 03/01/23 08:09:43.761
    Mar  1 08:09:43.764: INFO: Waiting for pod pod-17487 to disappear
    Mar  1 08:09:43.766: INFO: Pod pod-17487 no longer exists
    STEP: Deleting underlay pod pod-14919 03/01/23 08:09:43.766
    Mar  1 08:09:43.773: INFO: Waiting for pod pod-14919 to disappear
    Mar  1 08:09:43.775: INFO: Pod pod-14919 still exists
    Mar  1 08:11:31.777: INFO: Waiting for pod pod-14919 to disappear
    Mar  1 08:11:31.782: INFO: Pod pod-14919 no longer exists
    STEP: Deleting overlay pod  03/01/23 08:11:31.782
    Mar  1 08:11:31.782: FAIL: Failed to delete pod "": resource name may not be empty

    Full Stack Trace
    k8s.io/kubernetes/test/e2e/framework/pod.(*PodClient).DeleteSync(0xc000e12cf0, {0x0, 0x0}, {{{0x0, 0x0}, {0x0, 0x0}}, 0x0, 0x0, 0x0, ...}, ...)
        /home/runner/go/pkg/mod/k8s.io/kubernetes@v1.26.2/test/e2e/framework/pod/pod_client.go:183 +0x195
    github.com/kubeovn/kube-ovn/test/e2e/framework.(*PodClient).DeleteSync(...)
        /home/runner/work/kube-ovn/kube-ovn/test/e2e/framework/pod.go:25
    github.com/kubeovn/kube-ovn/test/e2e/kube-ovn/underlay.glob..func1.2()
        /home/runner/work/kube-ovn/kube-ovn/test/e2e/kube-ovn/underlay/underlay.go:275 +0x2c5
    github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func3({0xa84ece, 0xc0008bbc80})
        /home/runner/go/pkg/mod/github.com/onsi/ginkgo/v2@v2.4.0/internal/node.go:449 +0x1b
    github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func2()
        /home/runner/go/pkg/mod/github.com/onsi/ginkgo/v2@v2.4.0/internal/suite.go:750 +0x98
    created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode
        /home/runner/go/pkg/mod/github.com/onsi/ginkgo/v2@v2.4.0/internal/suite.go:738 +0xe3d
    [DeferCleanup (Each)] [CNI:Kube-OVN] [group:underlay]
      dump namespaces | framework.go:196
    STEP: dump namespace information after failure 03/01/23 08:11:31.783
    [DeferCleanup (Each)] [CNI:Kube-OVN] [group:underlay]
      tear down framework | framework.go:193
    STEP: Destroying namespace "underlay-4966" for this suite. 03/01/23 08:11:31.783
  << End Captured GinkgoWriter Output

  Mar  1 08:09:43.760: timed out while waiting for pod underlay-4966/pod-1[491](https://github.com/kubeovn/kube-ovn/actions/runs/4301158673/jobs/7498268970#step:15:492)9 to be running and ready
  In [It] at: /home/runner/go/pkg/mod/k8s.io/kubernetes@v1.26.2/test/e2e/framework/pod/pod_client.go:107

  There were additional failures detected after the initial failure.  Here's a summary - for full details run Ginkgo in verbose mode:
    [FAILED] in [AfterEach] at /home/runner/go/pkg/mod/k8s.io/kubernetes@v1.26.2/test/e2e/framework/pod/pod_client.go:183
    [PANICKED] in [DeferCleanup (Each)] at /opt/hostedtoolcache/go/1.20.1/x64/src/runtime/panic.go:260