search

azachar / protractor-screenshoter-plugin

A brand new jasmine2 protractor plugin that captures for each browser instance a screenshot, console logs, raw HTML and DB dumps. The snapshot is made optionally for each expect or spec. Plugins comes with a beautiful angular based analytics tool to visually check and fix tests results.
Other
89 stars 24 forks source link

Reports are wrong on parallel (sharded) tests #85

Open jdmwood opened 5 years ago

jdmwood commented 5 years ago

Bug report

exports.config = {
  multiCapabilities: [
    {
      shardTestFiles: true,
      maxInstances: 2,
      browserName: 'chrome',
    }
  ],

  specs: [
    '../test/screenshottest/**/*.js'
  ],

  // Default URL to be passed to Protractor
  //baseUrl: 'http://localhost:3000',

  // Framework to be used by Protractor
  framework: 'jasmine2',

  plugins: [{
    package: 'protractor-screenshoter-plugin',
    screenshotPath: './REPORTS/e2e',
    screenshotOnExpect: 'failure+success',
    screenshotOnSpec: 'none',
    withLogs: true,
    writeReportFreq: 'asap',
    imageToAscii: 'none',
    clearFoldersBeforeTest: true
  }],

  // Setup
  onPrepare: () => {
    browser.waitForAngularEnabled(false);

    // returning the promise makes protractor wait for the reporter config before executing tests
    return global.browser.getProcessedConfig().then(function(config) {
      //it is ok to be empty
    });
  }
};

The report shows the wrong number of tests - it shows whichever tests were run in one of the browser instances

  1. Unzip the Zip
  2. cd into the folder
  3. npm install && npm test
  4. Open up REPORTS/e2e/index.html
  5. It should show all 4 test failures, but it only shows 2. E.g.:

Screenshot 2019-03-29 at 12 20 14

It looks like the report gets cleared whenever one of the browser instances closes?

PS: I also tried with later protractor and webdriver-manager (^6.0.0 and webdriver-manager ^13.0.0). You can see this in * protractor-bug-1.zip. Not sure this is working correctly as I get weird errors.

Galaloo commented 5 years ago

I was running into the same issue. Opted to set clearFoldersBeforeTest: false and clear the directory with beforeLaunch. This way the directory is cleaned out when the test run first starts but not with each sharded test.

// config.js
let rmDir = require('../helpers/rmDir');
let path = require('path');
let testReportsDir = path.resolve(__dirname, '../reports/testreports');
...
    beforeLaunch() {
        // clean out the test results directory
        rmDir(testReportsDir);
    },
// rmDir.js
var fs = require('fs');

async function rmDir(dirPath)
{
  try { var files = fs.readdirSync(dirPath); }
  catch(e) { return; }
  if (files.length > 0)
    for (var i = 0; i < files.length; i++) {
      var filePath = dirPath + '/' + files[i];
      if (fs.statSync(filePath).isFile())
        fs.unlinkSync(filePath);
      else
        rmDir(filePath);
    }
  return fs.rmdirSync(dirPath);
};

  module.exports = rmDir;