containernetworking / plugins

Some reference and example networking plugins, maintained by the CNI team.
Apache License 2.0
2.23k stars 788 forks source link

Flaky VRF tests due to missing IPv6 route #1103

Closed LionelJouin closed 1 month ago

LionelJouin commented 1 month ago

In the tests:

IPv6 are added to an interface, and routes using that IPv6 is created. In order to create the route, we should wait for some IPv6 route to be created (created automatically by the kernel when adding the IPv6 to the interface). Without that route, the RouteAdd call in the tests will fail, for example:

• [FAILED] [0.011 seconds]
vrf plugin [It] filters the correct routes to import to new VRF
/home/runner/work/plugins/plugins/plugins/meta/vrf/vrf_test.go:278

  Timeline >>
  STEP: Setting custom routing for IF0Name @ 09/20/24 07:14:34.461
  STEP: Setting custom routing for IF1Name @ 09/20/24 07:14:34.462
  STEP: Adding if1 to the VRF @ 09/20/24 07:14:34.464
  STEP: Checking routes are moved correctly to VRF @ 09/20/24 07:14:34.467
  [FAILED] in [It] - /home/runner/work/plugins/plugins/plugins/meta/vrf/vrf_test.go:392 @ 09/20/24 07:14:34.469
  << Timeline

  [FAILED] Unexpected error:
      <syscall.Errno>: 
      invalid argument
      0x16
  occurred
  In [It] at: /home/runner/work/plugins/plugins/plugins/meta/vrf/vrf_test.go:392 @ 09/20/24 07:14:34.469

Here is some job that have failed. https://github.com/containernetworking/plugins/actions/runs/10954784634/job/30417500257?pr=1092 https://github.com/containernetworking/plugins/actions/runs/11109122762/job/30863918356?pr=1082

LionelJouin commented 1 month ago

Here is the second issue related to the VRF plugin:

Running tests
without coverage profile generation...
github.com/containernetworking/plugins/plugins/meta/vrf
=== RUN   TestVRF
Running Suite: plugins/meta/vrf - containernetworking/plugins/plugins/meta/vrf
=======================================================================================================
Random Seed: 1728311452

Will run 1 of 1 specs
------------------------------
• [FAILED] [0.630 seconds]
vrf plugin [It] adds the interface and custom routing to new VRF
containernetworking/plugins/plugins/meta/vrf/vrf_test.go:183

  Timeline >>
  STEP: Setting custom routing first @ 10/07/24 16:30:52.644
  [FAILED] in [It] - containernetworking/plugins/plugins/meta/vrf/vrf_test.go:269 @ 10/07/24 16:30:53.263
  << Timeline

  [FAILED] Unexpected error:
      <*errors.errorString | 0xc000228970>: 
      cmdAdd failed: could not add route '{Ifindex: 2 Dst: 1111:dddd::/80 Src: abcd:1234:ffff::cdde Gw: abcd:1234:ffff::1 Flags: [] Table: 1 Realm: 0}': invalid argument
      {
          s: "cmdAdd failed: could not add route '{Ifindex: 2 Dst: 1111:dddd::/80 Src: abcd:1234:ffff::cdde Gw: abcd:1234:ffff::1 Flags: [] Table: 1 Realm: 0}': invalid argument",
      }
  occurred
  In [It] at: containernetworking/plugins/plugins/meta/vrf/vrf_test.go:269 @ 10/07/24 16:30:53.263
------------------------------

Summarizing 1 Failure:
  [FAIL] vrf plugin [It] adds the interface and custom routing to new VRF
  containernetworking/plugins/plugins/meta/vrf/vrf_test.go:269

Ran 1 of 1 Specs in 0.633 seconds
FAIL! -- 0 Passed | 1 Failed | 0 Pending | 0 Skipped
--- FAIL: TestVRF (0.63s)
FAIL
FAIL    github.com/containernetworking/plugins/plugins/meta/vrf 0.649s
FAIL

I am still investigating the issue. Adding a 100ms sleep time after this (plugins/meta/vrf/vrf.go#L130-L144) solves the problem.