Closed khushboo-rancher closed 3 months ago
We don't have those test cases listed in manual test plans
:
Failed cases marked as e2e_be
is legacy automation tests, which not actually equals to the test plan. (the test plan is based on UI operation)
To notice that we don't test migrating VM in the upgrade test. (which mentioned in https://github.com/harvester/harvester/issues/4758#issuecomment-1929180078)
Failed cases marked as e2e_be is legacy automation tests, which not actually equals to the test plan.
What's e2e_be? Did you mean these cases need not be added in the test plan for automation?
The failed scenarios due to bumping kubevirt to v1.1.0 https://github.com/harvester/harvester/issues/5020 and https://github.com/harvester/harvester/issues/5076 should be covered in the automation backend test cases.
Failed cases marked as e2e_be is legacy automation tests, which not actually equals to the test plan.
What's e2e_be? Did you mean these cases need not be added in the test plan for automation?
The failed scenarios due to bumping kubevirt to v1.1.0 harvester/harvester#5020 and harvester/harvester#5076 should be covered in the automation backend test cases.
The e2e_be
marked that it would be covered by legacy automation code ideally, but actually, most manual test plan marked with e2e_be
is not covered correctly.
For issue#5020, we were discussed in QA meeting that Albin hit this bug which caused by Longhorn^1 and already fixed in latest Harvester. In automation test, we only test that the volume could be created from image,^2 but NOT test whether it is able to create any VMs, the test case rewrote based on legacy code with same test steps.
For issue #5076, upgrade testing will not test VM migration during upgrading, that's the reason why automation didn't catch the bug.
Got it. To summarize we'll have to implement below two cases in automation.
please update the title and content to relevant topic, thanks.
close as the test case passed in several builds.
What's the test to develop? Please describe
According to https://github.com/harvester/harvester/issues/4758#issuecomment-1925613014, Vm basic operations failed but we didn't catch this on CI. We need to analyze why we didn't we catch this. If we don't have respective test cases, we need to implement this on priority.