(feat): Add (at least) one negative workflow for each OpenEBS experiment

[uditgaurav]

This issue is for the feature request in the litmus-e2e pipeline

what it is about?

• Adding at least one negative workflow for each Openebs experiment running in the pipeline.

why and how?

• Negative testing ensures that our experiments can gracefully handle invalid input or unexpected scenarios. The purpose of negative testing is to detect such situations and prevent experiments from crashing.
• So it is important to add at least one negative test case per experiment in the pipeline.

List of Experiments:

• openebs-pool-container-failure
• openebs-pool-pod-failure
• openebs-target-container-failure
• openebs-target-network-delay
• openebs-target-network-loss
• openebs-target-pod-failure

[uditgaurav]

Test cases which have been added for openebs pool container failure experiment

• Verdict Check
  • It involves a basis verdict check of the experiment. It is already done at the end of the experiment.
• Resource Version test
  • In the resource version test, it has been tested whether the resource version of the CSP pod has been changed after chaos injection. For successful chaos injection - the value of Resource Version should change.
• Container Restart Count test
  • In the container restart count test, we are testing whether the number of container restart count is getting changed or not after chaos injection. For successful chaos injection - the number of restart count should change.
• Container ID test
  • In the container ID test, the container ID before chaos and after the chaos has been matched and for the change in the ID, it is considered a successful test.

[uditgaurav]

Validations for OpenEBS Experiments (To be done)

Experiments	Validations (Test Cases)
	Before Chaos	After Chaos
Openebs Pool Container Failure	1. Resource Version changes (from old) 2. Restart count of cstor-pool-mgmt and cstor-pool containers.(from old) 3. Container ID changes (from old value)	1. Resource Version changes (to new value) 2. Restart count of cstor-pool-mgmt and cstor-pool containers.(to new value) 3. Container ID changes (to new value)
Openebs Pool Pod Failure	1. New cstor-sparse-pool pod is created in openebs ns - and deleting the old one. New pod can be tested by checking:- - Resource Version(old) - Name - podIP - startTime	1. The old cstor-sparse-pool pod Gets deleted and new one gets created with:- - Resource Version (new) - Name(different) - podIP(different) - startTime(different)
Openebs Target Container Failure	1. Resource Version is changed (from a random number). 2. RestartCount for cstor-istgt container changes (from 0). 3. ContainerID changed (status.containerStatuses[0].containerID). 4. In container cstor-istgt the value of startedAt changed from (random value). 5. lastState contains empty value for cstor-istgt container	1. Resource Version changed (to a random number). 2. ContainerID changed to new value. 3. RestartCount for cstor-istgt container changes (to 1). 4. In container cstor-istgt the value of startedAt changed from (random higher value) 5. lastState now contains value like containerID,exitCode,reason,finishedAt
Openebs Target Network Delay	kube-ip tables are used to check network issues.	It will give an error message if it finds any network issues.
Openebs Target Network Loss	kube-ip tables are used to check network issues.	It will give an error message if it finds any network issues.
Openebs Target Pod Failure	1. Restart count changes for target pod (some value) 2. Resource Version of the target pod changes (random value) 3. PodIP gets changed (say 10.32.1.12) 4. Container ID for cstor-volume-mgmt (from some value) 5. lastState contains empty value for cstor-volume-mgmt container	1. Restart count changes for target pod ( some value + 1) 2. Resource Version of the target pod changes(other random value) 3. PodIP gets changed (It will become 10.32.1.13) 4. Container Id for cstor-volume-mgmt (to new value) 5. lastState contains non empty value for cstor-volume-mgmt container

[uditgaurav]

The test cases which are added for openebs pool pod failure experiment:

• Verdict Check
  • It involves a basis verdict check of the experiment. It is already done at the end of the experiment.
• Resource Version test
  • In the resource version test, it has been tested whether the resource version of the CSP pod has been changed after chaos injection. For successful chaos injection - the value of Resource Version should change.
• Pod Name test
  • In openebs pool pod failure experiment, anyone replica of the csp pod is getting down and new replicas come up with a new pod name this change in the name of the pod has been tested in this test.
• PodIP test
  • In the PodIP test, the change in the PodIP is compared between the old replica and the newly formed replica of csp pod.
• HostIP
  • In the HostIP test, the change in the HostIP is compared between the old replica and the newly formed replica of csp pod.
• StartTime
  • In the StartTime test, the change in the start time of csp pod is tested before chaos and after the chaos is injected.

[uditgaurav]

Test cases which have been added for openebs target container failure

• Verdict Check
  • It involves a basis verdict check of the experiment. It is already done at the end of the experiment.
• Resource Version test
  • In the resource version test, it has been tested whether the resource version of the target pod has been changed after chaos injection. For successful chaos injection - the value of Resource Version should change.
• Container Restart Count test
  • In the container restart count test, we are testing whether the number of container restart count is getting changed or not after chaos injection. For successful chaos injection - the number of restart count of the target container should change.
• Container ID test
  • In the container ID test, the container ID before and after the chaos has been matched. One container ID should be different after chaos is the expected condition for a successful test run.
• StartedAt test
  • In startedAt test, the time at which the container started has been stored before chaos and matched with the restarted time after chaos. If the time doesn't match - this means the chaos has been injected successfully.

[uditgaurav]

This issue contains:

• The addition of Validations (Test cases) for openebs target pod failure

The tests added for openebs target pod failure are:

• Verdict Check
  • It involves a basis verdict check of the experiment. It is already done at the end of the experiment.
• Resource Version test
  • In the resource version test, it has been tested whether the resource version of the target pod has been changed after chaos injection. For successful chaos injection - the value of Resource Version should change.
• Container Restart Count test
  • In the container restart count test, we are testing whether the number of container restart count is getting changed or not after chaos injection. For successful chaos injection - the number of restart count of the target container should change.
• PodIP test
  • In the PodIP test, the change in the PodIP is compared between the old replica and the newly formed replica of target pod.
• Container ID test
  • In the container ID test, the container ID before and after the chaos has been matched. One container ID should be different after chaos is the expected condition for a successful test run.