Validate Pods test fails returning index out of range

[bguzman-3pillar]

Sometimes running the version bumps jobs fails with this error, even locally happened.

• Happening on rke2 & k3s
• OSs tested: SLES, Ubuntu, Rhel9 (I usually used these ones)
• Have tested using 1 server
• Happens in all version bumps jobs but isn't always, https://jenkins.int.rancher.io/job/k3s-tests/view/k3s-version-bump/ & https://jenkins.int.rancher.io/job/rke2-tests/view/rke2-version-bump/

07:58:42.514  FAILED! VersionTemplate Upgrade: Validate Pods
07:58:42.514  
07:58:42.514  ------------------------------
07:58:42.514  • [PANICKED] [0.220 seconds]
07:58:42.514  VersionTemplate Upgrade: [It] Validate Pods
07:58:42.514  /go/src/github.com/rancher/distros-test-framework/entrypoint/versionbump/runc_test.go:29
07:58:42.514  
07:58:42.514    [PANICKED] Test Panicked
07:58:42.514    In [It] at: /go/pkg/mod/github.com/onsi/gomega@v1.27.6/internal/async_assertion.go:321 @ 09/27/23 13:58:42.483
07:58:42.514  
07:58:42.514    runtime error: index out of range [0] with length 0
07:58:42.514  
07:58:42.514    Full Stack Trace
07:58:42.514      github.com/onsi/gomega/internal.(*AsyncAssertion).buildActualPoller.func3.1()
07:58:42.514        /go/pkg/mod/github.com/onsi/gomega@v1.27.6/internal/async_assertion.go:321 +0x1c5
07:58:42.514      panic({0x112ce60?, 0xc0006e8a08?})
07:58:42.514        /usr/local/go/src/runtime/panic.go:914 +0x21f
07:58:42.514      github.com/rancher/distros-test-framework/shared.parsePods({0x0, 0x0})
07:58:42.514        /go/src/github.com/rancher/distros-test-framework/shared/cluster.go:345 +0x329
07:58:42.514      github.com/rancher/distros-test-framework/shared.GetPods(0x0)
07:58:42.514        /go/src/github.com/rancher/distros-test-framework/shared/cluster.go:328 +0xc5
07:58:42.514      github.com/rancher/distros-test-framework/pkg/testcase.TestPodStatus.func1({0x157f718, 0xc000602210})
07:58:42.514        /go/src/github.com/rancher/distros-test-framework/pkg/testcase/pod.go:23 +0x58
07:58:42.514      reflect.Value.call({0xfc94c0?, 0xc00057f2e0?, 0x3b9aca00?}, {0x11b15b7, 0x4}, {0xc0004be798, 0x1, 0x3?})
07:58:42.514        /usr/local/go/src/reflect/value.go:596 +0xce7
07:58:42.514      reflect.Value.Call({0xfc94c0?, 0xc00057f2e0?, 0xc000056000?}, {0xc0004be798?, 0x4ec6829a7e4065?, 0x11616a0?})
07:58:42.515        /usr/local/go/src/reflect/value.go:380 +0xb9
07:58:42.515      github.com/onsi/gomega/internal.(*AsyncAssertion).buildActualPoller.func3()
07:58:42.515        /go/pkg/mod/github.com/onsi/gomega@v1.27.6/internal/async_assertion.go:325 +0x11f
07:58:42.515      github.com/onsi/gomega/internal.(*AsyncAssertion).match(0xc0001ec000, {0x1575de0?, 0x1e34da0}, 0x1, {0x0, 0x0, 0x0})
07:58:42.515        /go/pkg/mod/github.com/onsi/gomega@v1.27.6/internal/async_assertion.go:398 +0x179
07:58:42.515      github.com/onsi/gomega/internal.(*AsyncAssertion).Should(0xc0001ec000, {0x1575de0, 0x1e34da0}, {0x0, 0x0, 0x0})
07:58:42.515        /go/pkg/mod/github.com/onsi/gomega@v1.27.6/internal/async_assertion.go:145 +0x86
07:58:42.515      github.com/rancher/distros-test-framework/pkg/testcase.TestPodStatus(0x12333c0, 0x12333b8, 0x12333c8)
07:58:42.515        /go/src/github.com/rancher/distros-test-framework/pkg/testcase/pod.go:29 +0xf3
07:58:42.515      github.com/rancher/distros-test-framework/entrypoint/versionbump.glob..func1.3()
07:58:42.515        /go/src/github.com/rancher/distros-test-framework/entrypoint/versionbump/runc_test.go:30 +0x28

Examples: https://jenkins.int.rancher.io/job/k3s-tests/view/k3s-version-bump/job/k3s-runc-version-bump/26/console https://jenkins.int.rancher.io/job/k3s-tests/view/k3s-version-bump/job/k3s-etcd-version-bump/21/console https://jenkins.int.rancher.io/job/k3s-tests/view/k3s-version-bump/job/k3s-cniplugin-version-bump/9/console

[fmoral2]

please provide:

• both products?
• different type of os.s ?
• only with config 1 server or else ?
• happens in all version bumps?

[fmoral2]

here you can see the fix applied on the current PR https://github.com/rancher/distros-test-framework/pull/32

• working 3 consecutive jobs ( the failing ones in the middle is the runs without the correct jenkins file, since u need to start one than replay )

k3s https://jenkins.int.rancher.io/job/k3s-tests/view/k3s-version-bump/job/k3s-runc-version-bump/ https://jenkins.int.rancher.io/job/k3s-tests/view/k3s-version-bump/job/k3s-cniplugin-version-bump/ https://jenkins.int.rancher.io/job/k3s-tests/view/k3s-version-bump/job/k3s-cniplugin-version-bump/ https://jenkins.int.rancher.io/job/k3s-tests/view/k3s-version-bump/job/k3s-all-version-bump-validation/

rke2 -WIP

[fmoral2]

fixed here https://github.com/rancher/distros-test-framework/pull/32