extent-framework / extentreports-java

Extent Reporting Library, Java
http://extentreports.com
Apache License 2.0
221 stars 126 forks source link

Results getting mixed when using with Grid and TestngListener #88

Closed eymasar closed 5 years ago

eymasar commented 5 years ago

We have been using ExtentReport with Testng Listeners and Selenium grid with 45 sessions total. I create report on BeforeMethod for each test and log through page object system. Then onListeners, I remove tests onSkip and I take screenshot onfailure and attach to the report. On execution, I flush it. But even though I tried everything I could, like, flush every onFinish, or flush it afterMethod, Scaling screenshot to minimum, using hashmap or threadlocal in extent report setting. But couldn't find any solutions. It is still getting mixed when I take screenshot at onFailure method.

image

@BeforeMethod(alwaysRun = true)
    @Parameters(value = { "browser", "testName", "testCategory", "Priority","Coverage" })
    public synchronized void beforeMethod(String browser, String testName, String testCategory, String Priority,String Coverage)
            throws InterruptedException, IOException {
        try {
            RemoteWebDriver driverDocker = null;
            testNameFromXml.set(testName);
            testSuiteName = testCategory;
            browserName = browser;
            priority = Priority;
            coverage = Coverage;
            isBrowserDead=false;
            ReportFactory.getTest(testNameFromXml.get(), testSuiteName, browserName,priority,coverage);
            logger.info("Log4j appender configuration is successful !!");
........................

@Override
    public void onTestFailure(ITestResult iTestResult) {
        try {
        testName  = testNameFromXml.get();
        this.sendStatus(iTestResult,"FAIL",iTestResult.getThrowable().toString());
        System.out.println("I am in onTestFailure method " +  iTestResult.getMethod().getMethodName() + " failed");
        methodName = iTestResult.getMethod().getMethodName();
        testReporter = ReportFactory.getTest();
        int retryCount = iTestResult.getTestContext().getSkippedTests().size();
            if (testReporter == null)
                ReportFactory.getTest(testName,
                        testSuiteName,browserName,coverage,priority);
            testReporter = ReportFactory.getTest();
            logger.error( methodName +" : This test case has been retried : " + retryCount + " time!");
            testReporter.log(Status.INFO,
                    MarkupHelper.createLabel( " This test case has been retried : " + retryCount + " time!", ExtentColor.BLUE));
            testReporter.log(Status.FAIL,MarkupHelper.createLabel( "Test failed ! ", ExtentColor.RED));
            testReporter.log(Status.FAIL, iTestResult.getThrowable() );
            WebDriver driver = getDriver();
            if(driver!=null) {
                testReporter.log(Status.FAIL,"I am at "+ testName+" ," +methodName);                        
                imagePath=takeScreenShotOfFailure( driver,testReporter,testName) ;              

                }
            else {
                testReporter.log( Status.FAIL , "Driver return null !! ");

            }

            } catch ( Throwable e) {
                try {
                    testReporter.log( Status.FAIL , "Got an error !! ");        
                    testReporter.log( Status.FAIL , e.getMessage());                    

                } catch (Throwable e1) {
                    System.out.println("GOT AN ERROR !!! ");
                }
            }

    }
`@Override
    public void onExecutionFinish() {
        try {
        Thread.sleep(240000);   
        ReportFactory.getExtentReport().flush();
        ReportFactory.closeTest(ReportFactory.getTest());
        Runtime.
        getRuntime().
        exec("cmd /c start \"\" cleanDrivers.bat");
//      if (isHostNameNotLocal)     
//              AfterAllSuites.sendEmail(ReportFactory.Path.toString(), testSuiteName, log4JSetUp.logPath.toString());
//              
        } catch (IOException  | InterruptedException e) {
            e.printStackTrace();
        }
    }
`

    public static synchronized String takeScreenshot(ExtentTest test, WebDriver testDriver, String testClassName,
            String pageClassName, String pageMethodName, String baseMethodName) throws IOException, HeadlessException, AWTException {

        String dest = null;

        try {
        DateFormat df = new SimpleDateFormat("dd-MM-yyyy--HH");
        DateFormat df1 = new SimpleDateFormat("dd-MM-yyyy--HH--mm--ss");

        Date date = new Date();
        Calendar cal = Calendar.getInstance();
        cal.setTime(date);

        if (!isHostNameNotLocal)
            dest = System.getProperty("user.dir") + "\\Reports" + "\\screenshots\\"+ df.format(cal.getTime()).toString()+"_"+reportFolder ;
        else
            dest = "\\\\" + "Istpweb30t01v\\web30reports" + "\\screenshots\\"+ df.format(cal.getTime()).toString()+"_"+reportFolder ;

        File folder = new File(dest);
        if (!folder.exists()) {
            folder.mkdirs();
        }
        dest = dest + "\\"
                + baseMethodName + testClassName + pageMethodName
                + df1.format(cal.getTimeInMillis()).toString() + ".png";

        File outputFile = new File(dest);

        Screenshot screenshot = new AShot().shootingStrategy(ShootingStrategies.scaling(3)).takeScreenshot(testDriver);     

        ImageIO.write(screenshot.getImage(), "PNG", outputFile);    
        }catch (Throwable e) {
              test.log(Status.FAIL,"Exception is --- "+ e.getMessage());
        }
        return dest;
    }

Using: Extent reports - 4.0.9 TestNG - 6.14.3 Selenium - 3.14.0

virenv commented 5 years ago

From the looks of this much of code, it seems that your code is not thread-safe. This is something you need to handle in your code. Can you share the ReportFactory and WebDriver generation logic of your code?

eymasar commented 5 years ago

ReportFactory: PS: I have used concurrent hashmap, collections.synchronizedMap and hashmap before.

package ExtentFactory;

import java.io.File;
import java.net.UnknownHostException;
import java.util.Collections;
import java.util.HashMap;
import java.util.Map;
import com.aventstack.extentreports.ExtentReports;
import com.aventstack.extentreports.ExtentTest;
import com.aventstack.extentreports.reporter.ExtentHtmlReporter;
import com.aventstack.extentreports.reporter.configuration.Theme;

import base.TestBase;

public class ReportFactory extends TestBase {

    public static ExtentReports reporter;
    public static String Path = null;
    public static ExtentHtmlReporter htmlReporter;  
    protected static ThreadLocal<ExtentTest> test = new ThreadLocal<ExtentTest>();

    public synchronized static ExtentReports getExtentReport() throws UnknownHostException {
        if (reporter == null) {
            Path = System.getProperty("user.dir") + "\\Reports" + "\\extentReportsWeb3";            
             File folder = new File(Path);
            if (!folder.exists()) {
                folder.mkdir();
            }

            Path = Path+"\\"+reportFolder;

            new File(Path).mkdir();

            Path = Path +"\\"+"Report_"+reportFolder+".html";
            htmlReporter = new ExtentHtmlReporter(Path);
            htmlReporter.config().setTheme(Theme.STANDARD);
            reporter = new ExtentReports();
            reporter.attachReporter(htmlReporter);

        }
        return reporter;
    }
    public synchronized static void getTest(String testName, String testCategory, String browser,String priority_, String coverage_)
            throws UnknownHostException {

         ExtentTest parent = getExtentReport().createTest(testName).assignCategory(testCategory.toUpperCase()).
                 assignCategory(browser.toUpperCase()).assignCategory(priority_.toUpperCase()).assignCategory(coverage_.toUpperCase());
         test.set(parent);
    }

    public synchronized static ExtentTest getTest() {
        return test.get();
    }

    public synchronized static void closeTest(ExtentTest test) throws UnknownHostException {
        if (test != null) {
            reporter.flush();

        }
    }

    public synchronized static void removeTest(ExtentTest test) throws UnknownHostException {
        if (test != null) {
            reporter.removeTest(test);

        }
    }

    public synchronized static void closeTest() throws UnknownHostException {
        closeTest(test.get());
    }

}
eymasar commented 5 years ago

And BeforeMethod :


    protected static ThreadLocal<RemoteWebDriver> dr = new ThreadLocal<>();

    @BeforeMethod(alwaysRun = true)
    @Parameters(value = { "browser", "testName", "testCategory", "Priority","Coverage" })
    public synchronized void beforeMethod(String browser, String testName, String testCategory, String Priority,String Coverage)
            throws InterruptedException, IOException {
        try {
            RemoteWebDriver driverDocker = null;
            testNameFromXml.set(testName);
            testSuiteName = testCategory;
            browserName = browser;
            priority = Priority;
            coverage = Coverage;
            isBrowserDead=false;
            ReportFactory.getTest(testNameFromXml.get(), testSuiteName, browserName,priority,coverage);
            logger.info("Log4j appender configuration is successful !!");

            DesiredCapabilities capabilities = new DesiredCapabilities();
            ChromeOptions chromeOptions = new ChromeOptions();
            Map<String, Object> prefs = new HashMap<String, Object>();

            if (runOption.equalsIgnoreCase("GRID")) {
                switch (browser) {
                case "firefox":
                ...                 
                case "chrome":
                ...
                case "ie":
                ...
                case "opera":
                ...
                case "yandex":
                ...

                }

                driverDocker = new RemoteWebDriver(new URL("....:4444/wd/hub"), capabilities);
                driverDocker.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);
                setWebDriver(driverDocker);
                getDriver().navigate().to(CONFIG.getProperty("baseUrl"));
                String implicitlyWait = CONFIG.getProperty("default_implicitWait");
                getDriver().manage().timeouts().implicitlyWait(Long.parseLong(implicitlyWait), TimeUnit.SECONDS);
                getDriver().manage().timeouts().pageLoadTimeout(Long.parseLong(CONFIG.getProperty("default_pageLoadWait")),
                        TimeUnit.SECONDS);}

public synchronized static WebDriver getDriver() {

            return dr.get();

    }

public void setWebDriver(RemoteWebDriver driver) {
        dr.set(driver);
}
virenv commented 5 years ago

@eymasar everything seems fine here. Code looks to be thread safe too. Is it possible to create a small project to reproduce this problem. It is hard for me to understand the issue with so much of code clutter. Most of the code is specific to your project.

eymasar commented 5 years ago

Hi again, So I have figured it out where we having trouble. When I create a report, I create it at shared folder on another server, so the application hold all the data in cache (probably) and transfer them to that server at the same time, and this approach causes data loss over network or i/o problems. Right now I create screenshots with base64 feature and attach the report which is on local machine then I transfer whole report to another server. But my point is, why extentreport get confused everytime it has problems with screenshots? So the driver is not null, testReporter logs fail message then tries to attach screenshot, but fails at some point and couldn't do it and Catch messages logs in another test case.

if(driver!=null) {
                testReporter.log(Status.FAIL,"I am at "+ testName+" ," +methodName);                        
                imagePath=takeScreenShotOfFailure( driver,testReporter,testName) ;              

                }
            else {
                testReporter.log( Status.FAIL , "Driver return null !! ");

            }

            } catch ( Throwable e) {
                try {
                    testReporter.log( Status.FAIL , "Got an error !! ");        
                    testReporter.log( Status.FAIL , e.getMessage());                    

                } catch (Throwable e1) {
                    System.out.println("GOT AN ERROR !!! ");
                }
            }
virenv commented 5 years ago

@eymasar : The problem that you have described is a classical symptoms of multithreading issues. Looking at your production code can give me only this much information because I am looking at code snippets only. For example testReporter.log, what exactly does it do inside. Similary I cant see the usage of imagePath. It will be really nice if I can get structured information and also if you can provide me with the version of Extentreports you are using.

stale[bot] commented 5 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs.

stale[bot] commented 5 years ago

This issue has been automatically closed because of inactivity.