search

sbabcoc / TestNG-Foundation

TestNG Foundation is a lightweight collection of TestNG listeners, interfaces, and static utility classes that supplement and augment the functionality provided by the TestNG API.
Apache License 2.0
13 stars 5 forks source link
parameters retrying testng testng-listener timeout-control

readme

Maven Central

INTRODUCTION

TestNG Foundation is a lightweight collection of TestNG listeners, interfaces, and static utility classes that supplement and augment the functionality provided by the TestNG API. The facilities provided by TestNG Foundation include two types of runtime listener hooks, test artifact capture, automatic test context attribute propagation, automatic retry of failed tests, and test execution timeout management.

Future releases of TestNG Foundation will add target platform support. See ExecutionFlowController for more information.

TestNG Listeners

Interfaces

Implementing ArtifactType
package com.example;

import java.nio.file.Path;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.testng.ITestResult;

import com.nordstrom.automation.testng.ArtifactType;

public class MyArtifactType extends ArtifactType {

    private static final String ARTIFACT_PATH = "artifacts";
    private static final String EXTENSION = "txt";
    private static final String ARTIFACT = "This text artifact was captured for '%s'";
    private static final Logger LOGGER = LoggerFactory.getLogger(MyArtifactType.class);

    @Override
    public boolean canGetArtifact(ITestResult result) {
        return true;
    }

    @Override
    public byte[] getArtifact(ITestResult result) {
        return String.format(ARTIFACT, result.getName()).getBytes().clone();
    }

    @Override
    public Path getArtifactPath(ITestResult result) {
        return super.getArtifactPath(result).resolve(ARTIFACT_PATH);
    }

    @Override
    public String getArtifactExtension() {
        return EXTENSION;
    }

    @Override
    public Logger getLogger() {
        return LOGGER;
    }
}
Creating a type-specific artifact collector
package com.example;

import com.nordstrom.automation.testng.ArtifactCollector;

public class MyArtifactCapture extends ArtifactCollector<MyArtifactType> {

    public MyArtifactCapture() {
        super(new MyArtifactType());
    }

}

The preceding code is an example of how the artifact type definition is assigned as the type parameter in a subclass of ArtifactCollector. Because TestNG listeners are specified solely by their class, type-specific artifact collectors must be declared this way.

Annotations

Static Utility Classes

ExecutionFlowController, ListenerChain, and the ServiceLoader

If ExecutionFlowController is the only listener you need, or if the order in which your listeners are invoked is inconsequential, the TestNG @Listeners annotation is a perfectly fine method to activate your listeners. However, if you need to activate multiple listeners that must be invoked in a specific order, use ListenerChain and activate it via the ServiceLoader as described in the TestNG documentation:

org.testng.ITestNGListener
com.nordstrom.automation.testng.ListenerChain

In a Maven project, the preceding file is stored in the src/main/resources folder:

com.testng.ITestNGListener

Once this file is added to your project, ListenerChain will be loaded automatically whenever you run your tests. To link listeners into the chain, you have two options:

  1. Specify listeners to attach via the ListenerChain service loader.
  2. Mark your test class with the @LinkedListeners annotation.

These options are not mutually exclusive; you can freely apply both within the same project.

Specifying listeners to attach via the ListenerChain service loader

Listeners that you wish to attach via the ListenerChain service loader must implement the LinkedListener interface:

Service-loaded listener example
package com.example

import org.testng.IClassListener;

import com.nordstrom.automation.testng.LinkedListener;

public class ServiceLoadedListener implements IClassListener, LinkedListener {

    @Override
    public void onBeforeClass(ITestClass testClass) {
        ...
    }

    @Override
    public void onAfterClass(ITestClass testClass) {
        ...
    }
}

To specify the listener(s) you wish to attach via the ListenerChain service loader, create a file at location
META-INF/services/com.nordstrom.automation.testng.LinkedListener and indicate the listener(s) you want to be linked in:

com.nordstrom.automation.testng.LinkedListener
com.example.ServiceLoadedListener

In a Maven project, the preceding file is stored in the src/test/resources folder:

com.testng.ITestNGListener

In this example, we've specified a single listener (ServiceLoadedListener) that should be attached via the ListenerChain service loader. If additional listeners had been specified, each of them would be attached in the order they're specified.

Marking your test class with the @LinkedListeners annotation

With the @LinkedListeners annotation, you specify one or more listener types to attach to the ListenerChain:

LinkedListeners annotation
package com.example;

import com.nordstrom.automation.selenium.listeners.DriverListener;
import com.nordstrom.automation.testng.ExecutionFlowController;
import com.nordstrom.automation.testng.LinkedListeners;
import com.nordstrom.automation.testng.ListenerChain;

@LinkedListeners({DriverListener.class, ExecutionFlowController.class})
public class ExampleTest {

    ...

}

As shown above, we use the @LinkedListeners annotation to attach DriverListener and ExecutionFlowController. The order in which listener methods are invoked is determined by the order in which listener objects are added to the chain. Listener before methods are invoked in last-added-first-called order. Listener after methods are invoked in first-added-first-called order. Only one instance of any given listener class will be included in the chain.

ExecutionFlowController managed features: Method timeout and retry analyzer

The annotation transformer of ExecutionFlowController applies the configuration for two managed features to their corresponding attributes in the @Test annotation:

When a positive retry count and valid retry analyzer are specified, the indicated analyzer is attached to every test method that doesn't already specify a retry analyzer. Note that until you create and populate the provider configuration file, RetryManager will always return false. Consequently, no failed tests will be retried. The IRetryAnalyzer implementations in the classes specified by the configuration file determine whether or not any given failed test is retried.

Attaching retry analyzers via RetryManager

As indicated above, RetryManager is a TestNG retry analyzer that provides a framework for invoking collections of scenario-specific analyzers that are installed via the ServiceLoader:

Prior to retrying a failed test, RetryManager emits a debug-level message in this format:

### RETRY ### [suite-name/test-name] className.methodName(parmValue...)

The class/method portion of these messages is produced by the InvocationRecord class. The content of each parmValue item (if any) represents the actual value passed to the corresponding argument of a failed invocation of a parameterized test.

Declining automatic retry support

Once automatic retry is enabled, RetryManager will be attached to every method that doesn't already specify a retry analyzer. However, there may be test methods or classes that you don't wish to be retried. For example, you may have a long-running test method that would delay completion of the suite, or you have an entire class of tests that rely on externally-managed resources that must be replenished between runs.

For these sorts of scenarios, you can mark test methods or classes with the @NoRetry annotation:

@Test
@NoRetry
public void testLongRunning() {
    // test implementation goes here
}

Redacting sensitive values in RetryManager messages

Sensitive values can be redacted from the messages that RetryManager emits by marking their test method arguments with the @RedactValue annotation:

@Test
@Parameters({"username", "password"})
public void testLogin(String username, @RedactValue String password) {
    // test implementation goes here
}

The retry message for this method would include the actual user name, but redact the password:

### RETRY ### [MySuite/MyTest] AccountTest.testLogin(john.doe, |:arg1:|)

Using RetryManager in another framework

Typically, scenario-specific retry analyzers are installed via the service loader. However, if you plan to use RetryManager in another framework, we recommend that you extend this class and override the isRetriable(ITestResult) method instead of registering your retry analyzer via the service loader. This strategy enables clients of your framework to add their own analyzers without disconnecting yours. Just make sure to invoke the overridden method in RetryManager if your analyzer declines to request method retry:

@Override
protected boolean isRetriable(ITestResult result) {
    if (isRetriableInFramework(result)) {
        return true;
    }
    return super.isRetriable(result);
}

Remember to override the value of the RETRY_ANALYZER setting with the fully-qualified class name of your framework-specific extension of RetryManager to enable activation of your analyzer by ExecutionFlowController.

Propagation of Test Attributes

ExecutionFlowController propagates test context attributes from one phase of test execution to the next. This feature enables tests to attach context-specific values that are accessible throughout the entire lifecycle of the test.
The attribute propagation feature of ExecutionFlowController produces many-to-one-to-many behavior:

Managing Object Reference Attributes

This attribute propagation feature provides an easy way for tests to maintain context-specific values. For any attribute whose value is an object reference, this behavior can result in the creation of additional references that will prevent the object from being marked for garbage collection until the entire suite of tests completes. For these sorts of attributes, TestNG Foundation provides the TrackedObject class.
TrackedObject is a reference-tracking wrapper used by PropertyManager to record the test result objects to which an object reference is propagated. This enables the client to release all references when the object is no longer needed:

private static final DRIVER = "Driver";

public void setDriver(WebDriver driver) {
    ITestResult result = Reporter.getCurrentTestResult();
    if (driver != null) {
        new TrackedObject<>(result, DRIVER, driver);
    } else {
        Object val = result.getAttribute(DRIVER);
        if (val instanceof TrackedObject) {
            ((TrackedObject<?>) val).release();
        } else {
            result.setAttribute(DRIVER, null);
            result.removeAttribute(DRIVER);
        }
    }
}

public WebDriver getDriver() {
    Object obj;
    ITestResult result = Reporter.getCurrentTestResult();
    Object val = result.getAttribute(DRIVER);
    if (val instanceof TrackedObject) {
        obj = ((TrackedObject<?>) val).getValue();
    } else {
        obj = val;
    }
    return (WebDriver) obj;
}

In this example, a Selenium driver attribute is stored as a tracked object. When the driver is no longer needed, specifying a null value will signal that all propagated references should be released. To retrieve the driver reference from the test attribute, extract it with the TrackedObject.getValue() method.