kubevirt / kubevirt

Kubernetes Virtualization API and runtime in order to define and manage virtual machines.

https://kubevirt.io

Apache License 2.0

5.67k stars 1.35k forks source link

flaky test: `[rfe_id:393][crit:high][vendor:cnv-qe@redhat.com][level:system][sig-compute] VM Live Migration Starting a VirtualMachineInstance [Serial] migration monitor [test_id:6980]Migration should fail if target pod fails during target preparation` #13143

Closed dhiller closed 1 week ago

dhiller commented 1 month ago

What happened

Flaky test detected: [rfe_id:393][crit:high][vendor:cnv-qe@redhat.com][level:system][sig-compute] VM Live Migration Starting a VirtualMachineInstance [Serial] migration monitor [test_id:6980]Migration should fail if target pod fails during target preparation 1

/kind flake /priority critical-urgent

/sig compute

Additional context

Happened to start failing on periodics around four days ago according to search ci 2, seems to fail only on k8s-1.31, where impact is yet below quarantine threshold 3

Flake Action Plan

As the assignee, thoroughly review the issue and put the resulting report as comment into this issue.

Then decide on one of the following actions:

The flake is a bug

Add label commenting /triage accepted
Create a PR to fix the bug
Reference this issue in the PR
Keep this issue open until the testcase does not fail anymore

The flake is non-critical or an issue that is hard to fix

Quarantine the test by creating a pull request assigning [QUARANTINE] tag to test name and Quarantine decorator to the test
Reference this issue on the PR

There was an infrastructure issue

An infra issue is anything "below" the testcase

Add label commenting /triage infra-issue
Close the issue, adding a comment with details about the infrastructure issue.

After the flake has been fixed, document the learning from it inside the fix PR

dhiller commented 1 month ago

/sig ci

dhiller commented 3 weeks ago

@fossedihelm please find someone to take a look at this, since it's starting to get more pressing, according to flake stats: https://storage.googleapis.com/kubevirt-prow/reports/flakefinder/kubevirt/kubevirt/flake-stats-14days-2024-11-06.html#%5brfe_id%3a393%5d%5bcrit%3ahigh%5d%5bvendor%3acnv-qe%40redhat.com%5d%5blevel%3asystem%5d%5bsig-compute%5d%20VM%20Live%20Migration%20Starting%20a%20VirtualMachineInstance%20%5bSerial%5d%20migration%20monitor%20%5btest_id%3a6980%5dMigration%20should%20fail%20if%20target%20pod%20fails%20during%20target%20preparation

brianmcarey commented 3 weeks ago

This flake first started to appear around the 24th / 25th of October https://testgrid.kubernetes.io/kubevirt-periodics#periodic-kubevirt-e2e-k8s-1.31-sig-compute-migrations&width=5

dhiller commented 1 week ago

@fossedihelm please chime in here whether there's a related issue you are working on

fossedihelm commented 1 week ago

@dhiller It was a different issue: https://github.com/kubevirt/kubevirt/pull/13302