adobe / phased-testing

A TestNG plugin that allows tests to be reused in validating System changes such as upgrades or system changes that happen in chaos engineering.
MIT License
7 stars 3 forks source link

Consider Phase Event as a parallel non-interruptive event #87

Open baubakg opened 2 years ago

baubakg commented 2 years ago

We should consider the possibility of executing a non-inerruptive parallel event. For an even X and a scenario:

A-> B -> C

The Phase event X should happen: A(X) -> B -> C A -> B(X) -> C A -> B -> C(X)

baubakg commented 2 years ago

Introduce phase chaos

baubakg commented 2 years ago

We can have three modes of execution.

(in the cas of parallel events, when executing between steps, it works as serial)

The event to be executed is either on test level or suite level

All phased tests can be subject to non-interruptive events. (the events follow the rules for shuffled and non-shuffled)

Execution (two possibilities) :

The non-interruptive event can be specific to a scenario or be set globally.

A scenario subject to interruptive events can also be subject to non-interruptive events as well. This means that a scenario that is wrapped around a non-interruptive event at one go can also be wraped around an interruptive event as well. However this may not be simultaneous; I.e. a scenario cannot be subject to both interruptive and non-interruptive events.

This means that we may have two separate executions. One for interruptive events and one for non-interruptive ones. However, can a scenario be executed as both non-phased and interruptive? Answer: NO

A parallel event needs to start before a step and finish after (before starting the next). Otherwise we cannot map an action/step to an event and a problem.

We need link a list of events to a step-pg.

In the end we need to repeat the scenario = nr of steps.

baubakg commented 2 years ago

Examples

  1. A scenario that allows for a specific Non-interruptive class / method. If set it overrides the passed non-interruptive events:
Class InterruptiveEvents {
  //restartProductServer
  public void event1() {
  }

  //restartDatabaseServer
  public void event2() {
  }
}

@PhasedTest(canShuffle = true,  interruptedBy=InterruptiveEvents.class, interruptionModel="during")
@Test
public void class MyPhasedTest {

      public void step1(String val) {  
      }

      public void step2(String val) {  
      }

      public void step3(String val) {  
      }
}

When run the standard phase groups are multiplie by the number of event types. In the example above, we will normally have 4 phase groups (0-3, 1-2, 2-1, 3-0), however with the events, we get:

Note to self when we are in a "during" event maybe the 0-3,3-0 may be irrelevant.

Scenario level configurations, override the global configurations

baubakg commented 1 year ago

Ok we have the following cases:

NR Phase Execution Mode Split Shuffle
1 PRODUCER/CONSUMER SHUFFLED true true
2 PRODUCER/CONSUMER Single true false
3 NON_PHASED N/A false false
4 ASYNCHRONOUS Single false false
5 ASYNCHRONOUS SHUFFLED false true