Open sapins opened 6 years ago
currently we can't control the test case execution. Each test is independently executed in Device Farm's environment.
Hi! Device farm creating new appium instance for each test case in testng.xml. How can i control it?
I want to run all test classes in sequence without creating new appium instance.
@rekha110254 Unfortunately I'm afraid that's not possible in Device Farm currently. However, I have found a workaround which might be applicable to the use case here.
With TestNG you can execute tests from a test using a TestNG object. So you can make one test method which then runs the other test classes. Then select only that one test using a testng.xml file. The caveat with this is that Device Farm will see it as one test in the SDK/CLI and web console. So you will need to rely on the generated test results of TestNG using the custom artifacts feature of device farm.
You could also use some other custom reporting frameworks like:
Note: Using this workaround, other features like priority and the dependsOnMethods work.
Here is an example test method which calls other test classes in this sample project.
public class DeviceFarmTests {
@Test
public void testMultipleTests(){
TestNG runner = new TestNG();
runner.setOutputDirectory(System.getenv("WORKING_DIRECTORY")); //uses the custom artifacts to get the results
runner.setPreserveOrder(true);
List<String> suitefiles = new ArrayList<String>();
//add the classes that need testing
runner.setTestClasses(new Class[] { HomePageTest.class});
//execute the tests
runner.run();
}
}
Note: if you need to only execute certain methods within the classes you will need to create a virtual xml file as describe in the TestNG docs.
Here is an example of what that would looks like:
public class DeviceFarmTests {
@Test
public void testMultipleTests(){
//create a virtual testng.xml file to list the specific method to be executed
XmlSuite suite = new XmlSuite();
suite.setName("TmpSuite");
XmlTest test = new XmlTest(suite);
test.setName("TmpTest");
List<XmlClass> classes = new ArrayList<XmlClass>();
//get methods for homepage test
XmlClass homepagetest = new XmlClass("Tests.HomePageTest");
List<XmlInclude> homepageMethods = new ArrayList<XmlInclude>();
homepageMethods.add(new XmlInclude("testHomePageSubheader"));
homepagetest.setIncludedMethods(homepageMethods);
//add this class to the list of classes to execute
classes.add(homepagetest);
//do this for every test classes that needs to be tested, one by one
//run the tests
test.setXmlClasses(classes);
TestNG runner = new TestNG();
//Using Allure for reports so we won't need to set this now.
runner.setOutputDirectory(System.getenv("WORKING_DIRECTORY")); //uses the custom artifacts to get the results
runner.setPreserveOrder(true);
List<XmlSuite> suitefiles = new ArrayList<XmlSuite>();
suitefiles.add(suite);
runner.setXmlSuites(suitefiles);
//execute the tests
runner.run();
}
}
Then the testng.xml file would look like:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Default Suite">
<test name="test" preserve-order="true">
<classes>
<class name="Tests.DeviceFarmTests"/> <!--Package.ClassName-->
</classes>
</test>
</suite>
Here is instructions on how to include the testng.xml file in a test package.
In Device Farm this looks like one test.
However, using the custom artifacts you can see that all the tests passed but in the same Appium session.
So this might not work for all use cases, but I hope that this helps.
To use the allure test report framework in Device Farm and specifically with this workaround first:
allure.properties
file in the ./src/test/resources
directory as shown in Allure's example testng project.
/tmp/allure-results
(Note: this should work with any directory locally but using the device host machine in Device Farm it's recommended to use the tmp
directory to avoid permission issues).So the contents of the file would look like: allure.results.directory=/tmp/allure-results
Note: after copies the dependencies you may need to update your local maven repository using this command:
mvn clean install -U -DskipTests=true
POM.xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
...
<properties>
<aspectj.version>1.7.4</aspectj.version><!--I'm not using aspectj here but just copied cause it was in docs -->
<allure.version>1.5.4</allure.version><!--Look for most recent version here: https://mvnrepository.com/artifact/ru.yandex.qatools.allure/allure-testng-adaptor -->
</properties>
<dependencies>
...
<dependency>
<groupId>ru.yandex.qatools.allure</groupId>
<artifactId>allure-testng-adaptor</artifactId>
<version>${allure.version}</version>
</dependency>
</dependencies>
<build>
<plugins>
...
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.14</version>
<configuration>
<testFailureIgnore>false</testFailureIgnore>
<argLine>
-javaagent:${settings.localRepository}/org/aspectj/aspectjweaver/${aspectj.version}/aspectjweaver-${aspectj.version}.jar
</argLine>
<!--only for 1.3.* TestNG adapters. Since 1.4.0.RC4, the listener adds via ServiceLoader-->
<properties>
<property>
<name>listener</name>
<value>ru.yandex.qatools.allure.testng.AllureTestListener</value>
</property>
</properties>
</configuration>
<dependencies>
<dependency>
<groupId>org.aspectj</groupId>
<artifactId>aspectjweaver</artifactId>
<version>${aspectj.version}</version>
</dependency>
</dependencies>
</plugin>
</plugins>
</build>
</project>
Note: Allure reports should automatically apply itself to all tests so it's not dependent on this workaround to work in Device Farm.
public class DeviceFarmTests {
@Test
public void testMultipleTests(){
TestNG runner = new TestNG();
//Using Allure for reports so we won't need to set this now.
//runner.setOutputDirectory(System.getenv("WORKING_DIRECTORY")); //uses the custom artifacts to get the results
runner.setPreserveOrder(true);
List<String> suitefiles = new ArrayList<String>();
//add the classes that need testing
runner.setTestClasses(new Class[] { HomePageTest.class});
//execute the tests
runner.run();
}
}
mvn clean package -DskipTests=true
and schedule a run with /tmp/allure-results
set as a directory to export in the Specify Device State
page of the Device Farm web console(see attached screenshot below).The xml files are then used to make the HTML report. http://allure.qatools.ru/
During report generation (second step), the XML files are transformed to a HTML report. This can be done with a command line tool, a plugin for CI or a build tool. See examples and documentation for more details.
I was able to do this via the command line tool for allure.
commands used to install and execute:
# install the package
npm install allure-commandline --save-dev -g
# add it to an alias or system var depending which is easiest for you
code ~/.bash_profile
alias allure="/c/Users/$(whoami)/AppData/Roaming/npm/node_modules/allure-commandline/bin/allure"
# execute the command to generate the HTML report
allure generate .relative/path/to/allure-results/directory --clean -o /output/directory/allure-report
Then I opened the index.html page generated in FireFox. Note: this page doesn't seem to work in chrome.
@jamesknowsbest Hi James,
How to use allure report format in the AWS device farm?
@Logakarti I've updated my answer to show how to use allure reports in Device Farm.
@sapins @rekha110254 @Logakarti @jpeddicord @ahawker Device Farm just released a new feature which allows to run the tests the same way they're executed locally. https://docs.aws.amazon.com/devicefarm/latest/developerguide/custom-test-environments.html
@jamesknowsbest
I was able to do this via the command line tool for allure.
Does your allure report built from device farm contain the steps correctly?
I am missing the steps in my report and I'm struggling to find why.
How to control test case execution using testng.xml? Can we control the test case execution using testng.xml while executing appium test case on AWS device farm?
In my local system i am able to perform case execution using testng.xml, however it's not working on AWS device farm.