search

qmetry / qaf

Quality Automation Framework for web, mobileweb, mobile native and rest web-service using Selenium, webdrier, TestNG and Java Jersey
https://qmetry.github.io/qaf
MIT License
253 stars 137 forks source link

QAFTestStep not logged to test-results (custom TestNG framework). #445

Open mialeska opened 1 year ago

mialeska commented 1 year ago

QAF Version

Latest release - 3.2.0

We have existing maven solution with large amount of selenium tests written in java with testNG. Now we would like to import execution results into QMetry for Jira. We decided to integrate QAF package into our existing solution, to get results in qas format, with failure screenshots and steps included.

As a starter, I followed the example in your repository: https://github.com/qmetry/qtm4j-automation-sample-projects/blob/master/QAF/pom.xml a I managed to set our driver instance to qaf, so currently we have got test results with screenshots. But now I still have an issue with it:

I started marking our steps (we call them pageActions) with @QAFTestStep annotations, to see them as checkpoints in the generated test result jsons. I have also added resources/application.properties with the parameter step.provider.pkg=pageActions (pageActions is a full name of our package with steps. I have added related plugins and dependencies for aspectj and qaf into our pom.xml . I can see that aspects has been generated (AjcClosure classes generated in target directory after the build). But unfortunately steps are not getting added to test result jsons. Could you please suggest where could be the problem or how to achieve what we want?

Steps To Reproduce

1.  Add QAF dependencies to maven project
2. Extend BaseTest from WebDriverTestCase and initialize browser at TestBaseProvider in beforeMethod:    
        TestBaseProvider.instance().get()
                .setDriver("chrome", new QAFExtendedWebDriver(DriverManager.getDriver()));
4. Add @QAFTestStep annotation to desired test steps
5. Run the maven command mvn clean test

Expected behavior

Method marked with @QAFTestStep annotation gets added as checkpoint to test-results/ ... .json file

Actual behavior

Only errors and assertions are present in test-results/ ... .json file

Is the issue reproducible on runner?

Test case sample

I prepared a temp repository with sample maven project with our dependencies, where the issue can be reproduced: https://github.com/mialeska/tmp-ui-tests/tree/test-results

Please find generated test results in the test-results branch.

mialeska commented 1 year ago

Please find the debug log below:


16:53:21.924 [pool-2-thread-1] DEBUG com.qmetry.qaf.automation.testng.pro.QAFInvokedMethodListener2 -- afterInvocation: beforeMethodSetUp - base.BaseTest is test:false
16:53:21.924 [pool-2-thread-1] DEBUG com.qmetry.qaf.automation.testng.pro.QAFInvokedMethodListener2 -- afterInvocation: Done
16:53:21.926 [pool-2-thread-1] INFO com.qmetry.qaf.automation.integration.ResultUpdator -- QAF Json Reporter: {description=, name=beforeMethodSetUp}
16:53:21.926 [pool-1-thread-1] DEBUG com.qmetry.qaf.automation.integration.ResultUpdator -- QAF Json Reporter: started to update result
16:53:21.957 [pool-2-thread-1] DEBUG com.qmetry.qaf.automation.testng.pro.QAFResultListener -- onTestStart: start
16:53:21.958 [pool-2-thread-1] DEBUG com.qmetry.qaf.automation.testng.pro.QAFInvokedMethodListener2 -- beforeInvocation: exampleTest
16:53:21.960 [pool-2-thread-1] INFO 1 -- I am example step
16:53:21.963 [pool-2-thread-1] DEBUG com.qmetry.qaf.automation.testng.pro.QAFInvokedMethodListener2 -- afterInvocation: exampleTest - tests.ExampleTest is test:true
16:53:21.971 [pool-2-thread-1] INFO com.qmetry.qaf.automation.ui.WebDriverCommandLogger -- Executing screenshot parameters: {}
16:53:22.112 [pool-2-thread-1] DEBUG com.qmetry.qaf.automation.core.QAFTestBase -- Capturing screen shotExampleTest_ej4k32a1eprx26292.png
16:53:22.113 [pool-2-thread-1] DEBUG com.qmetry.qaf.automation.testng.pro.QAFInvokedMethodListener2 -- afterInvocation: Done
16:53:22.121 [pool-2-thread-1] INFO com.qmetry.qaf.automation.integration.ResultUpdator -- QAF Json Reporter: {description=exampleTest, name=exampleTest, sign=ExampleTest.exampleTest()[pri:0, instance:tests.ExampleTest@1846579f]}
16:53:22.121 [pool-1-thread-1] DEBUG com.qmetry.qaf.automation.integration.ResultUpdator -- QAF Json Reporter: started to update result
16:53:22.127 [pool-2-thread-1] DEBUG com.qmetry.qaf.automation.testng.pro.QAFInvokedMethodListener2 -- beforeInvocation: updateTestStatus
16:53:22.128 [pool-2-thread-1] INFO 1 -- Executing logTestStatusToReport() method
16:53:22.136 [pool-2-thread-1] INFO 1 -- Capturing the screenshot :: takeScreenshot
16:53:22.267 [pool-2-thread-1] INFO 1 -- The screenshot is saved at D:\tmp-ui-tests/extent-report/screenshots/exampleTest_screenshot.png
16:53:22.273 [pool-2-thread-1] INFO 1 -- Screenshot available at the location and trying to attach to the report
16:53:22.426 [pool-2-thread-1] DEBUG com.qmetry.qaf.automation.testng.pro.QAFInvokedMethodListener2 -- afterInvocation: updateTestStatus - base.BaseTest is test:false
16:53:22.426 [pool-2-thread-1] DEBUG com.qmetry.qaf.automation.testng.pro.QAFInvokedMethodListener2 -- afterInvocation: Done
16:53:22.433 [pool-2-thread-1] INFO com.qmetry.qaf.automation.integration.ResultUpdator -- QAF Json Reporter: {description=, name=updateTestStatus}
16:53:22.433 [pool-1-thread-1] DEBUG com.qmetry.qaf.automation.integration.ResultUpdator -- QAF Json Reporter: started to update result
16:53:22.433 [pool-2-thread-1] DEBUG com.qmetry.qaf.automation.testng.pro.QAFInvokedMethodListener2 -- beforeInvocation: afterMethod
16:53:22.435 [pool-2-thread-1] DEBUG com.qmetry.qaf.automation.testng.pro.QAFInvokedMethodListener2 -- afterInvocation: afterMethod - com.qmetry.qaf.automation.ui.AbstractTestCase is test:false
16:53:22.435 [pool-2-thread-1] DEBUG com.qmetry.qaf.automation.testng.pro.QAFInvokedMethodListener2 -- afterInvocation: Done
16:53:22.436 [pool-2-thread-1] DEBUG com.qmetry.qaf.automation.testng.pro.QAFInvokedMethodListener2 -- beforeInvocation: afterClass
16:53:22.436 [pool-2-thread-1] DEBUG com.qmetry.qaf.automation.testng.pro.QAFInvokedMethodListener2 -- afterInvocation: afterClass - com.qmetry.qaf.automation.ui.AbstractTestCase is test:false
16:53:22.436 [pool-2-thread-1] DEBUG com.qmetry.qaf.automation.testng.pro.QAFInvokedMethodListener2 -- afterInvocation: Done
16:53:22.437 [main] DEBUG com.qmetry.qaf.automation.testng.pro.QAFInvokedMethodListener2 -- beforeInvocation: afterTest
16:53:22.437 [main] DEBUG com.qmetry.qaf.automation.testng.pro.QAFInvokedMethodListener2 -- afterInvocation: afterTest - com.qmetry.qaf.automation.ui.AbstractTestCase is test:false
16:53:22.437 [main] DEBUG com.qmetry.qaf.automation.testng.pro.QAFInvokedMethodListener2 -- afterInvocation: Done
FAILED: exampleTest
        exampleTest
java.lang.AssertionError: Some assertion expected [false] but found [true]
cjayswal commented 1 year ago

Refer pom file from sample maven project.

mialeska commented 1 year ago

@cjayswal yes I used it for the reference, but add some customizations for project specific:

cjayswal commented 1 year ago

Maven aspectj-maven-plugin is not specific to eclipse. If you don't want to use aspectj-maven-plugin refer configuration used in plugin.

mialeska commented 1 year ago

@cjayswal I am using the aspectj-maven-plugin. But my configuration is a little different, as with yours config Lombok dependency is not working. Lombok requires aspectJ to be configured in the maven-surefire-plugin's argline:

      <argLine>
            -javaagent:"${settings.localRepository}/org/aspectj/aspectjweaver/${aspectj.version}/aspectjweaver-${aspectj.version}.jar"
      </argLine>

With your configuration added, I see the following error:

[WARNING] You aren't using a compiler supported by lombok, so lombok will not work and has been disabled. Your processor is: org.aspectj.org.eclipse.jdt.internal.compiler.apt.dispatch.BatchProcessingEnvImpl Lombok supports: OpenJDK javac, ECJ <unknown source file>:<no line information>

[WARNING] advice defined in com.qmetry.qaf.automation.step.JavaStepReporter has not been applied [Xlint:adviceDidNotMatch] C:\Users\a2.meleshko.m2\repository\com\qmetry\qaf\3.2.0\qaf-3.2.0.jar!com\qmetry\qaf\automation\step\JavaStepReporter.class:0

Below is my configuration for plugins section:

       <plugins>
            <plugin>
                <groupId>org.codehaus.mojo</groupId>
                <artifactId>aspectj-maven-plugin</artifactId>
                <version>1.14.0</version>
                <executions>
                    <execution>
                        <id>default-compile</id>
                        <phase>process-classes</phase>
                        <goals>
                            <goal>compile</goal>
                        </goals>
                        <configuration>
                            <weaveDirectories>
                                <weaveDirectory>${project.build.directory}/classes</weaveDirectory>
                            </weaveDirectories>
                        </configuration>
                    </execution>
                    <execution>
                        <id>default-testCompile</id>
                        <phase>process-test-classes</phase>
                        <goals>
                            <goal>test-compile</goal>
                        </goals>
                        <configuration>
                            <weaveDirectories>
                                <weaveDirectory>${project.build.directory}/test-classes</weaveDirectory>
                            </weaveDirectories>
                        </configuration>
                    </execution>
                </executions>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                    <showWeaveInfo>true</showWeaveInfo>
                    <verbose>true</verbose>
                    <complianceLevel>1.8</complianceLevel>
                    <aspectLibraries>
                        <aspectLibrary>
                            <groupId>com.qmetry</groupId>
                            <artifactId>qaf</artifactId>
                        </aspectLibrary>
                    </aspectLibraries>
                    <sources/>
                    <excludes>
                        <exclude>**/*.java</exclude>
                    </excludes>
                    <forceAjcCompile>true</forceAjcCompile>
                </configuration>
                <dependencies>
                    <dependency>
                        <groupId>org.aspectj</groupId>
                        <artifactId>aspectjtools</artifactId>
                        <version>${aspectj.version}</version>
                    </dependency>
                </dependencies>
            </plugin>
        </plugins>