search

harvester / tests

Harvester test cases
Apache License 2.0
11 stars 31 forks source link

[TEST] Add Test `Volume is created from image and create a VM using that volume` #1097

Closed khushboo-rancher closed 3 months ago

khushboo-rancher commented 8 months ago

What's the test to develop? Please describe

According to https://github.com/harvester/harvester/issues/4758#issuecomment-1925613014, Vm basic operations failed but we didn't catch this on CI. We need to analyze why we didn't we catch this. If we don't have respective test cases, we need to implement this on priority.

lanfon72 commented 8 months ago

We don't have those test cases listed in manual test plans: image

Failed cases marked as e2e_be is legacy automation tests, which not actually equals to the test plan. (the test plan is based on UI operation)

To notice that we don't test migrating VM in the upgrade test. (which mentioned in https://github.com/harvester/harvester/issues/4758#issuecomment-1929180078)

khushboo-rancher commented 8 months ago

Failed cases marked as e2e_be is legacy automation tests, which not actually equals to the test plan.

What's e2e_be? Did you mean these cases need not be added in the test plan for automation?

The failed scenarios due to bumping kubevirt to v1.1.0 https://github.com/harvester/harvester/issues/5020 and https://github.com/harvester/harvester/issues/5076 should be covered in the automation backend test cases.

lanfon72 commented 8 months ago

Failed cases marked as e2e_be is legacy automation tests, which not actually equals to the test plan.

What's e2e_be? Did you mean these cases need not be added in the test plan for automation?

The failed scenarios due to bumping kubevirt to v1.1.0 harvester/harvester#5020 and harvester/harvester#5076 should be covered in the automation backend test cases.

The e2e_be marked that it would be covered by legacy automation code ideally, but actually, most manual test plan marked with e2e_be is not covered correctly.

For issue#5020, we were discussed in QA meeting that Albin hit this bug which caused by Longhorn^1 and already fixed in latest Harvester. In automation test, we only test that the volume could be created from image,^2 but NOT test whether it is able to create any VMs, the test case rewrote based on legacy code with same test steps.

For issue #5076, upgrade testing will not test VM migration during upgrading, that's the reason why automation didn't catch the bug.

khushboo-rancher commented 8 months ago

Got it. To summarize we'll have to implement below two cases in automation.

  1. Test Volume is created from image and create a VM using that volume. (Let's track this scenario implementation with this ticket.)
  2. While upgrading, testing the VM is migrated successfully. (Let's create a ticket to track that.)
lanfon72 commented 8 months ago

please update the title and content to relevant topic, thanks.

lanfon72 commented 3 months ago

close as the test case passed in several builds.