Phased Testing has been created to address the issues related to Event Based Testing. Event Based Testing is a notion where tests adapt to external events, and allow you to simulate how your product reacts to an external event.
Examples of events are:
Phased Testing has been created to address the issues related to Event Based Testing. Event Based Testing is a notion where tests adapt to external events, and allow you to simulate how your product reacts to an external event. We identify two types of events:
This library was originally created to help validate system changes such as upgrades and migrations.
Interruptive events are cases such as system & application upgrades, system migrations and dependant service upgrades. Where the whole system requires a down-time in order to perform these events. This library allows you to define tests in such a way, so that they can be interrupted at any point awaiting an event, and to carry on where they left off. More specifically based on your design the Phased tests will ensure that a scenario will work on an upgraded system no matter where it is interrupted.
This process can be used for validating :
Phased Testing, when testing Non-Interruptive events breaks down and reexecutes the tests in the way shown below:
If we want to simulate all the use cases for a workflow of a user we will end up with too many duplicate code. This is why we came up with Phased Testing, which allows a scenario to cover all the possible steps in which a workflow can be interrupted.
Non-Interruptive events are cases such as system restarts, load injections and other unexpected events. These events do not require the whole system to restart.
A typical use case for non-interruptive event is chaos testing.
This version runs with the TestNG runner. You can use this library by including it in your project.
The following dependency needs to be added to your pom file:
<dependency>
<groupId>com.adobe.campaign.tests.phased</groupId>
<artifactId>phased-testing-testng</artifactId>
<version>8.11.1</version>
</dependency>
We have a standard demo that can be accessed through the Phased Test Demo.
Phases are directives at execution time, where we let the system know, in what way we want our tests to interact with an event.
We have four test phases:
We have three modes of execution of a Phased Test:
The steps of each scenario are executed one by one without interruption.
Single Execution Mode is used only when a workflow will always be interrupted at a given stage. This is particularly relevant when your scenario will expect a time concuming external process to finish. In this case we execute all steps till the Phase End marker. When in Consumer mode, we execute the rest of the steps.
The diagram above represents what will be executed by the following code:
@Test
@PhasedTest
public class ShuffledTest {
public void step1(String val) {
PhasedTestManager.produce("step1Val","A");
}
public void step2(String val) {
String l_fetchedValue = PhasedTestManager.consume("step1Val");
PhasedTestManager.produce("step2Val",l_fetchedValue + "B");
}
@PhaseEvent
public void step3(String val) {
String l_fetchedValue = PhasedTestManager.consume("step2Val");
assertEquals(l_fetchedValue, "AB");
}
}
When in Shuffled mode, we execute all the possible ordered combinations of the steps.
The code below will react differently depending on the PHASE/Execution mode it is subject to :
@Test
@PhasedTest
public class ShuffledTest {
public void step1(String val) {
PhasedTestManager.produce("A1");
}
public void step2(String val) {
String l_fetchedValue = PhasedTestManager.consume("A1");
PhasedTestManager.produce("B1",l_fetchedValue + "B");
}
public void step3(String val) {
String l_fetchedValue = PhasedTestManager.consume("B1");
assertEquals(l_fetchedValue, "AB");
}
}
When a shuffled test is executed in an interruptive mode (Phases PRODUCER and CONSUMER), we execute all the possible ordered combinations interruptions the scenario can be subject to. Example Given a test with three steps, in Producer State, we :
When in Consumer state we :
As of version 8, we are introducing the asynchronous phase mode. Asynchronous phases are destined for non-interruptive events. They allow you to inject an event during the execution of the steps of a scenario. An asynchronous execution of a shuffled test, will shuffle the test, but will for each phase group execute, in parallel, the given event for each step.
Example:
The Phased Testing is activated using two annotations:
Moreover, you need to :
Note : As of version 7.0.11, we now have the possibility to let the framework pick the order for us.
In order for a test scenario to be executed in shuffle mode you need to add the following annotation at the class level @PhasedTest
In order for a test scenario to be executed in shuffle mode you simply need to set the annotation @PhaseEvent
somewhere along its steps. The location of this annotation is where you expect the interruption to occur..
Optionally if you consider that the scenario can never be run as non-phased, you need also include: @PhasedTest(executeInactive = false)
. When executeInactive is false, the Single Run scenario will only run when in Phases.
Ideally you should set the default data provider on your tests. This allows you to execute the test locally without needing to force the Phased Test listener.
@Test( dataProvider = PhasedDataProvider.DEFAULT, dataProviderClass = PhasedDataProvider.class)
@PhasedTest
public class MyPhasedTest {
}
However, whenever the Phased Test Listener discovers a Phased Test, it will add the necessary data providers needed for running the test. But, ideally it is best to set the default providers in orrder to not lose the possibility of local execution.
In the case of non-interruptive events there are a few things to consider:
In order have some level of predictability for non-interruptive events, we have defined an api for non-interruptive events. For an event to be able to be used by the Phased Tests it needs to inherit from the abstract class com.adobe.campaign.tests.integro.phased.NonInterruptiveEvent
which extends Runnable
.
In the example before we have created an event NonInterruptiveEventExample
:
public class NonInterruptiveEventExample extends NonInterruptiveEvent {
@Override
public boolean startEvent() {
return false;
}
@Override
public boolean isFinished() {
return false;
}
@Override
public boolean waitTillFinished() {
return false;
}
}
As you can see we have to implement three methods:
startEvent
starts the event.isFinished
allow the system to see if the event we declared has finished.waitTillFinished
waits until the event has finished.In order to define these event you will need to implement these methods, as you who are defining the event have the best knowledge on how these event will work.
At times the simple execution of an event is not sufficient. We need to perform an event clean up action to reset the system to a stable state. For this we allow you to define a 'tearDownEven' actions for an event. This means that after an event has been finished, we perform an additional set of actions before the next step is executed. To make use of this you need to override the method tearDownEvent
in your event. The framework will then execute this action right before the next step is triggered.
@Override
public boolean tearDownEvent() {
// Perform actions
return true; //Return true if the actions were successful
}
In order for your scenario to interact with an event you will need to declare it. This can be done in three ways (in order of precedence) :
If you have the event declared in more than one level (for example on both the PhasedEvent and the PhasedTest annotation), it is the value with more precedence which is taken into account.
The mode is only applicable to Single Run execution modes.
In the case of single run scenarios, we can specify which phase event should be triggered on the annotation itself. This is by setting the eventClasses
attribute for the @PhaseEvent
annotation.
In the example below the PhaseEvent is always executed at the step2 of the scenario. Here we have specified that when we are in asynchroous mode only the event com.adobe.campaign.tests.integro.phased.data.events.MyNonInterruptiveEvent
should be executed.
@PhasedTest
@Test
public class SingleRunScenarioWithEvent {
public void step1(String val) {
PhasedTestManager.produceInStep("A");
}
@PhaseEvent(eventClasses = {"com.adobe.campaign.tests.integro.phased.data.events.MyNonInterruptiveEvent"})
public void step2(String val) {
String l_fetchedValue = PhasedTestManager.consumeFromStep("step1");
PhasedTestManager.produceInStep(l_fetchedValue + "B");
}
public void step3(String val) {
String l_fetchedValue = PhasedTestManager.consumeFromStep("step2");
assertEquals(l_fetchedValue, "AB");
}
}
In this case we expect us to specify if a scenario is only subject to the same event. This will be done at the @PhasedTest
annotation using the attribute eventClasses
. When set we only use the specified event.
@PhasedTest(eventClasses = {"com.adobe.campaign.tests.integro.phased.data.events.MyNonInterruptiveEvent"})
@Test
public class ShuffledScenarioWithEvent {
public void step1(String val) {
PhasedTestManager.produce("step1Value","A");
}
public void step2(String val) {
String l_fetchedValue = PhasedTestManager.consume("step1Value");
PhasedTestManager.produce("Step2Value", l_fetchedValue + "B");
}
public void step3(String val) {
String l_fetchedValue = PhasedTestManager.consume("Step2Value");
assertEquals(l_fetchedValue, "AB");
}
}
In this case, we state that all scenarios should be using the same Event. We can activate this mode by setting the environment variable PHASED.EVENTS.NONINTERRUPTIVE
to the event class.
This works for both Shuffled and Single-Run tests. If we want to run all tests with the event com.adobe.campaign.tests.integro.phased.data.events.MyNonInterruptiveEvent
, we enter:
mvn clean test -DPHASED.EVENTS.NONINTERRUPTIVE=com.adobe.campaign.tests.integro.phased.data.events.MyNonInterruptiveEvent
You can also add it as a property in your testng definition file.
As of version 8.11.2, we can inject an event to a specific step of a Phased Scenario. This is done by:
PHASED.EVENTS.NONINTERRUPTIVE
.PHASED.EVENTS.TARGET
.The step should point to a method. For method step1
in the class a.b.c.ScenarioA
you can set:
a.b.c.ScenarioA.step1
ScenarioA#step1
ScenarioA.step1
In the case of nested tests, for method step1
in the class a.b.c.ScenarioA
, and sub-class NestedClassB
you need to use the $
notation. It will look like:
a.b.c.ScenarioA$NestedClassB.step1
ScenarioA$NestedClassB#step1
ScenarioA$NestedClassB.step1
Here is an example of running a specific event for a specific test:
mvn clean test -DPHASED.EVENTS.NONINTERRUPTIVE=com.adobe.campaign.tests.integro.phased.data.events.MyNonInterruptiveEvent -DPHASED.EVENTS.TARGET=ScenarioA$NestedClassB#step1
We have introduced the possibility of defining Before and After Phases. This means that you can state if a method can be invoked before or after the phased tests are executed. These methods are only activated when we are in a Phase, and will not run when executed when we execute the scenarios in Non-Phased mode.
However, Before/After Phase methods are like any other Before/After method as, when invoked, they will affect all underlying tests, even if they are not Phased Tests.
To activate this functionality you add the annotations @BeforePhase
& @AfterPhase
to a TestNG configuration method such as: @BeforeSuite, @AfterSuite, @BeforeGroups, @AfterGroups, @BeforeTest and @AfterTest.
To your configuration method. Example:
@BeforePhase
@BeforeSuite
public void myBeforePhaseSuite() {
//Perform actions
}
In the example above the method myBeforePhaseSuite
will be invoked in the beginning of the suite. By default, the BeforePhase method is invoked when we are in a Phase I.e. Producer or Consumer.
You can configure this with the attribute appliesToPhases
, which accepts an array of Phases
. In the example below we are activating AfterPhase for the Consumer phase only.
@AfterPhase(appliesToPhases = {Phases.CONSUMER})
@AfterSuite
public void myAfterPhasedSuite() {
//Perform actions
}
As of version 7.0.9 of Phased Testing which is based on the 7.5 of TestNG, we can now define nested Phased tests. This allows you to regroup the phased tests under the same class. Thus, you will have Phased Tests that resemble method based tests.
Example:
public class PhasedTestSeries_NestedContainer {
@Test
@PhasedTest
public class PhasedScenario1 {
public void step1(String val) {
PhasedTestManager.produce("myValX","A");
}
public void step2(String val) {
String l_fetchedValue = PhasedTestManager.consume("myValX");
assertEquals(l_fetchedValue, "A");
}
}
@Test
@PhasedTest
public class PhasedScenario2 {
public void step1(String val) {
PhasedTestManager.produce("MyVal1","AB");
}
public void step2(String val) {
String l_fetchedValue = PhasedTestManager.consume("MyVal1");
assertEquals(l_fetchedValue, "AB");
}
}
}
We are able to run tests in phases since each step stores the information needed for the following steps. For now this is done at the discretion of the developer. This storage is important as it helps us keep track of the tests:
Managing this data is obviously essential to the Phased Tests. We will discuss this in more detail in the chapter on "Managing Phased Data".
We have the following system properties:
We have four phased states:
This property is passed whenever we want to specify a non-interruptive event at run time. By passing the full name of the non-interruptive event, we can tell the system around which event our tests should be wrapped.
This parameter allows you to tell the PhaseTestManager which DataBroker implementation you want to use. The is usually a full class path (package name + class name). More on this will be dealt with in the chapter on Phased Data Broker.
This is the path in which the Phased Data is stored, and fetched. If not set, the path /phased_output/phased_tests/phaseData.properties will be used.
By default, Phased Test data is stored under the directory phased_output. You can override this by setting this system property. If not set, the default directory phased_output will be used.
By default, we deactivate retry analyzer for the phased tests. However, if you really want to use your retry listener, we can stop the phase test listener from deactivating it.
By default, we do not modify reports. Each step in a scenario is reported as is. We have introduced a "Report By Phase Group" functionality, which is activated with this property.
As of version 7.0.11, we will be detecting the order based on the code. These rules are deduced by analyzing the test code. Since it is not easy to deduce, we require the user to set the root directory from whoch the sources can be found. This directory should point to le location from which the first package directory starts.
As of version 7.0.11, we will be detecting the order based on the code. In 7.0.11, whenever this system property is set (the value is not important in this version), we execute the steps of a scenario based on their position within the class.
For versions < 8.0.0 we had a bug where the default execution mode was executed in a phase group called "phased-data-provider-single". This was incorrect, and as of version 8.0.0 the default execution mode of a phased test is "phased-default". Due to backward compatibility, we allow users to keep the old mode if they chose to.
Usually when your test code is in the repository of the product being tested, you will be having a delta in tests between two versions N & N+1. In such cases you will want to only execute the tests that exist in both versions.
For this, as of version 7.0.9, we have introduced the functionality that allows you to automatically select the phased tests that were executed in a previous phase. This means that when activated in a CONSUMER Phase, the selection is made based on the tests that were executed in the PRODUCER Phase. This functionality is activated when you pass or include the test group PHASED_PRODUCED_TESTS
.
By default, the phased tests, being implemented in TestNG follow the same rules as that test framework. This means that up to version 7.0.10 (included), the execution of the steps in a scenario follows an alphabetical rule.
As of version 8 we have implemented code based order. Whenever the system property, PHASED.TESTS.DETECT.ORDER is set, the steps are executed in the order the way we declared in the code. By default, we expect the code to be in maven where the tests are in the directory src/test/java. However, this can be overriden by setting the execution property PHASED.TESTS.CODE.ROOT.
Nested class tests are usually quite tricky in Surefire because dollar sign '$' used for identifiying these object needs to be escaped. You can run a nested tests in the following way:
```mvn clean test -Dtest='PhasedTestSeries_NestedContainer$PhasedScenario1'```
or
mvn clean test -Dtest=PhasedTestSeries_NestedContainer\$PhasedScenario1
Although we try to keep the execution of a scenario like any other test scenario, we feel that it is useful to document how the state of a scenario works.
Whenever a scenario step fails the following steps are marked as SKIPPED.
If a phase is not executed, the steps in the next phase are also SKIPPED.
The way data is stored between two phases is in two ways:
At the end of the producer phase we store all the phase data in a properties file. By default it is stored under: