How to control test case execution using testng.xml?

sapins commented 6 years ago

How to control test case execution using testng.xml? Can we control the test case execution using testng.xml while executing appium test case on AWS device farm?

In my local system i am able to perform case execution using testng.xml, however it's not working on AWS device farm.

jamesknowsbest commented 6 years ago

currently we can't control the test case execution. Each test is independently executed in Device Farm's environment.

rekha110254 commented 6 years ago

Hi! Device farm creating new appium instance for each test case in testng.xml. How can i control it?

I want to run all test classes in sequence without creating new appium instance.

jamesknowsbest commented 6 years ago

@rekha110254 Unfortunately I'm afraid that's not possible in Device Farm currently. However, I have found a workaround which might be applicable to the use case here.

Workaround:

With TestNG you can execute tests from a test using a TestNG object. So you can make one test method which then runs the other test classes. Then select only that one test using a testng.xml file. The caveat with this is that Device Farm will see it as one test in the SDK/CLI and web console. So you will need to rely on the generated test results of TestNG using the custom artifacts feature of device farm.

You could also use some other custom reporting frameworks like:

extent reports
Allure reports (see example below on how to use) see this SO post for reference.

Note: Using this workaround, other features like priority and the dependsOnMethods work.

Here is an example test method which calls other test classes in this sample project.

public class DeviceFarmTests {
    @Test
    public void testMultipleTests(){
        TestNG runner = new TestNG();
        runner.setOutputDirectory(System.getenv("WORKING_DIRECTORY")); //uses the custom artifacts to get the results
        runner.setPreserveOrder(true); 
        List<String> suitefiles = new ArrayList<String>();
        //add the classes that need testing
        runner.setTestClasses(new Class[] { HomePageTest.class});
        //execute the tests
        runner.run();
    }
}

Note: if you need to only execute certain methods within the classes you will need to create a virtual xml file as describe in the TestNG docs.

Here is an example of what that would looks like:

public class DeviceFarmTests {
    @Test
    public void testMultipleTests(){

        //create a virtual testng.xml file to list the specific method to be executed
        XmlSuite suite = new XmlSuite();
        suite.setName("TmpSuite");

        XmlTest test = new XmlTest(suite);
        test.setName("TmpTest");
        List<XmlClass> classes = new ArrayList<XmlClass>();

        //get methods for homepage test
        XmlClass homepagetest = new XmlClass("Tests.HomePageTest");
        List<XmlInclude> homepageMethods = new ArrayList<XmlInclude>();
        homepageMethods.add(new XmlInclude("testHomePageSubheader"));
        homepagetest.setIncludedMethods(homepageMethods);

        //add this class to the list of classes to execute
        classes.add(homepagetest);

        //do this for every test classes that needs to be tested, one by one

        //run the tests
        test.setXmlClasses(classes);
        TestNG runner = new TestNG();
        //Using Allure for reports so we won't need to set this now.
        runner.setOutputDirectory(System.getenv("WORKING_DIRECTORY")); //uses the custom artifacts to get the results
        runner.setPreserveOrder(true); 
        List<XmlSuite> suitefiles = new ArrayList<XmlSuite>();
        suitefiles.add(suite);

        runner.setXmlSuites(suitefiles);
        //execute the tests
        runner.run();
    }
}

Then the testng.xml file would look like:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Default Suite">
    <test name="test" preserve-order="true">
        <classes>
            <class name="Tests.DeviceFarmTests"/> <!--Package.ClassName-->
        </classes>
    </test>
</suite>

Here is instructions on how to include the testng.xml file in a test package.

In Device Farm this looks like one test.

However, using the custom artifacts you can see that all the tests passed but in the same Appium session.

So this might not work for all use cases, but I hope that this helps.

Using Allure test report framework

To use the allure test report framework in Device Farm and specifically with this workaround first:

Create a allure.properties file in the ./src/test/resources directory as shown in Allure's example testng project.
- Once created, set the output directory to be /tmp/allure-results (Note: this should work with any directory locally but using the device host machine in Device Farm it's recommended to use the tmp directory to avoid permission issues).

So the contents of the file would look like: allure.results.directory=/tmp/allure-results

Add the necessary dependencies to the maven pom.xml file as documented here in the Allure reports WiKi.

Note: after copies the dependencies you may need to update your local maven repository using this command: mvn clean install -U -DskipTests=true

POM.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
...
    <properties>
        <aspectj.version>1.7.4</aspectj.version><!--I'm not using aspectj here but just copied cause it was in docs -->
        <allure.version>1.5.4</allure.version><!--Look for most recent version here: https://mvnrepository.com/artifact/ru.yandex.qatools.allure/allure-testng-adaptor -->
    </properties>

    <dependencies>
        ...
        <dependency>
            <groupId>ru.yandex.qatools.allure</groupId>
            <artifactId>allure-testng-adaptor</artifactId>
            <version>${allure.version}</version>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            ...
            <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-surefire-plugin</artifactId>
            <version>2.14</version>
            <configuration>
                <testFailureIgnore>false</testFailureIgnore>
                <argLine>
                    -javaagent:${settings.localRepository}/org/aspectj/aspectjweaver/${aspectj.version}/aspectjweaver-${aspectj.version}.jar
                </argLine>
                <!--only for 1.3.* TestNG adapters. Since 1.4.0.RC4, the listener adds via ServiceLoader-->
                <properties>
                    <property>
                        <name>listener</name>
                        <value>ru.yandex.qatools.allure.testng.AllureTestListener</value>
                    </property>
                </properties>
            </configuration>
            <dependencies>
                <dependency>
                    <groupId>org.aspectj</groupId>
                    <artifactId>aspectjweaver</artifactId>
                    <version>${aspectj.version}</version>
                </dependency>
            </dependencies>
        </plugin>
        </plugins>
    </build>
</project>

In the DeviceFarmTest class, we can remove the reference for the output directory of the default TestNG report framework:

Note: Allure reports should automatically apply itself to all tests so it's not dependent on this workaround to work in Device Farm.

public class DeviceFarmTests {
    @Test
    public void testMultipleTests(){
        TestNG runner = new TestNG();
        //Using Allure for reports so we won't need to set this now.
        //runner.setOutputDirectory(System.getenv("WORKING_DIRECTORY")); //uses the custom artifacts to get the results
        runner.setPreserveOrder(true); 
        List<String> suitefiles = new ArrayList<String>();
        //add the classes that need testing
        runner.setTestClasses(new Class[] { HomePageTest.class});
        //execute the tests
        runner.run();
    }
}

Package the tests using mvn clean package -DskipTests=true and schedule a run with /tmp/allure-results set as a directory to export in the Specify Device State page of the Device Farm web console(see attached screenshot below).

Once the tests have completed you should be able to view the xml reports generated in the customer artifacts download link(see below screenshots).

The xml files are then used to make the HTML report. http://allure.qatools.ru/

During report generation (second step), the XML files are transformed to a HTML report. This can be done with a command line tool, a plugin for CI or a build tool. See examples and documentation for more details.

I was able to do this via the command line tool for allure.

commands used to install and execute:

# install the package
npm install allure-commandline --save-dev -g
# add it to an alias or system var depending which is easiest for you
code ~/.bash_profile
alias allure="/c/Users/$(whoami)/AppData/Roaming/npm/node_modules/allure-commandline/bin/allure"
# execute the command to generate the HTML report
allure generate .relative/path/to/allure-results/directory --clean -o /output/directory/allure-report

Then I opened the index.html page generated in FireFox. Note: this page doesn't seem to work in chrome.

Extent reports

DarshanLoga commented 6 years ago

@jamesknowsbest Hi James,

How to use allure report format in the AWS device farm?

jamesknowsbest commented 6 years ago

@Logakarti I've updated my answer to show how to use allure reports in Device Farm.

jamesknowsbest commented 6 years ago

@sapins @rekha110254 @Logakarti @jpeddicord @ahawker Device Farm just released a new feature which allows to run the tests the same way they're executed locally. https://docs.aws.amazon.com/devicefarm/latest/developerguide/custom-test-environments.html

aws-samples / aws-device-farm-appium-tests-for-sample-app