Very last step of very last scenario is not recorded

[sbrugnoni]

Folks,

I just discovered scenarioo. It's a really great tool, and we're thinking about adopting it at our company. I have written a small e2e spec using protractor/jasmine and scenaroo-js reporter. I'm using the async/await syntax for my spec, and everything is working great.

Except for one thing.: The XML-document of very the last step of the very last scenario does not get written to disk. The corresponding screenshot gets written.

recordLastStepForStatus.success =true/false doesn't influence the behaviour.

I'm not sure If I have misconfigured something, of if this is a bug.

Things I have tried:

• ran the tests in different order => same behavior. problem is use-case independent
• adding a final await step('scenario success') to the last scenario will record an additional screenshot, but not generate an additional step.

Here is my setup:

package.json of the test runner

  "dependencies": {
    "@types/jasmine": "^3.3.9",
    "protractor": "^5.4.2",
    "scenarioo-js": "^3.0.2",
    "ts-node": "^8.0.3",
    "typescript": "^3.3.3"
  }

protractor.conf.js

'use strict';

var scenarioo = require('scenarioo-js');
var _ = require('./specs/test-label-definitions');

var exportsConfig = {
  allScriptsTimeout: 10000,
  specs: ['specs/*.spec.ts'],
  SELENIUM_PROMISE_MANAGER: false,
  capabilities: {
    browserName: 'chrome',
    chromeOptions: {
      args: ['--no-sandbox', '--headless', '--window-size=1024,768'],
    },
    shardTestFiles: false,
    maxInstances: 1,
  },

  beforeLaunch: function() {
    require('ts-node').register({
      project: 'tsconfig.json',
    });
  },

  onPrepare: function() {
    by.addLocator('marker', (marker, parentElement) => {
      const scope = parentElement || document;
      const matches = scope.querySelectorAll(`[selenium-id="${marker}"]`);
      if (matches.length === 0) {
        return null;
      } else if (matches.length === 1) {
        return matches[0];
      }
      return matches;
    });

    var gitVersion = process.env.COMMIT || readVersionFromGit();

    scenarioo.setupJasmineReporter(jasmine, {
      targetDirectory: './target/scenariooReports',
      cleanBuildOnStart: true,
      branchName: process.env.BRANCH || readBranchNameFromGit(),
      branchDescription: process.env.DESCRIPTION || '',
      buildName: 'build_' + gitVersion + '_' + new Date().getTime(),
      revision: gitVersion,
      pageNameExtractor: function(url) {
        let segments = url.href.split('/').filter(el => el && el.length > 0);
        return segments[segments.length - 1];
      },
      reportStepOnExpectationFailed: true,
      recordLastStepForStatus: {
        failed: true,
        success: false,
      },
      disableScenariooLogOutput: false,
    });

    function readVersionFromGit() {
      var execSync = require('child_process').execSync;
      var version = '' + execSync('git rev-parse --short HEAD');
      console.log('git commit: ' + version);
      return version.substring(0, version.length - 1);
    }

    function readBranchNameFromGit() {
      var execSync = require('child_process').execSync;
      var branch = '' + execSync('git symbolic-ref --short HEAD');
      console.log('git branch: ' + branch);
      return branch.substring(0, branch.length - 1);
    }
  },

  jasmineNodeOpts: {
    showColors: true,
    includeStackTrace: true,
    defaultTimeoutInterval: 50000,
  },
};

exports.config = exportsConfig;

login.spec.ts (page objects omitted)

import { useCase, scenario } from 'scenarioo-js';
import { LoginPage } from '../pages/loginPage';
import { DashboardPage } from '../pages/dashboardPage';
import { NavigationPage } from '../pages/navigationPage';

useCase('Logout')
  .description('Logged-in user wants to log out')
  .describe(async () => {
    const loginPage = new LoginPage();
    const dashboardPage = new DashboardPage();
    const navigationPage = new NavigationPage();

    beforeEach(async () => {
      await loginPage.goTo();
    });

    await scenario('Logout via user menu')
      .description('Users land on login page after logout')
      .labels(['happy'])
      .it(async () => {
        await loginPage.loginNoStep('*****', '******');
        await dashboardPage.assertIsVisible();
        await navigationPage.logoutViaUserMenu();
        await loginPage.assertIsVisible();
      });
  });

useCase('Login')
  .description('User wants to log in')
  .describe(() => {
    const loginPage = new LoginPage();
    const dashboardPage = new DashboardPage();

    beforeEach(async () => {
      await loginPage.goTo();
    });

    scenario('Login with valid credentials')
      .description('Users with valid credentials land on dashboard after login')
      .labels(['happy'])
      .it(async () => {
        await loginPage.login('******', '*******');
        await dashboardPage.assertIsVisible();
      });

    scenario('Login with invalid credentials')
      .description('Users with invalid credentials are rejected')
      .labels(['happy'])
      .it(async () => {
        await loginPage.login('******', '******');
        await loginPage.assertLoginFailedErrorIsDisplayed();
      });
  });

[davidlahner]

Hi Silvano,

I'm not the expert on the javascript library of Scenarioo so I checked the behavior of our demo and there the last step of the last scenario is properly recorded.

Looking over our examples I noticed two differences to your listings: 1) We have an explicit await step('<Step information>'); direction after every test statement which executes a step. 2) We have recordLastStepForStatus enabled for success and failure recordLastStepForStatus: { failed: true, success: true }

For a complete example see https://github.com/scenarioo/scenarioo/blob/develop/scenarioo-client/test/e2e/specs/use_breadcrumbs.ts

What I did notice however, was that in every scenario we have one more screenshot than we have step xmls.

I'll try to get some insight from one of the experts on this.

[adiherzog]

@sbrugnoni @eggerschwiler This might be fixed on the develop branch already. In that case we could just create a new release of the library. We need to verify the assumption first though, and I don't know when I'll find time to do that.

[adiherzog]

Tried my own example project (JavaScript with Jasmine syntax, not DSL) with scenarioo-js 3.0.2 (the latest release) and could not reproduce the failure. Both the PNG and XML of the last step are written.

I also looked into the output written by our examples in the revision that corresponds to version 3.0.2 and can't reproduce the problem there.

Then I looked into the code to see how saveStep is implemented. The screenshot and the XML file are written in parallel. There must be some error happening inside writeStepXml. This error might happen in getPageNameFromUrl which uses the pageNameExtractor defined in the protractor.conf.js file:

  pageNameExtractor: function(url) {
    let segments = url.href.split('/').filter(el => el && el.length > 0);
    return segments[segments.length - 1];
  },

Could it be that segments is ever empty? I don't really see how this is possible though...

Could you please write a minimal test to reproduce the issue and send us the complete source code? The test should run against a publicly available website, so we can execute it as well.

[sbrugnoni]

Hey Folks,

Thanks for looking into this issue.

To work around it for now I have added a 1 second sleep after every test suite. This gives the reporter enough time to write all files. However, this is a brittle solution for obvious reasons :)

afterAll(async () => {
  await browser.sleep(1000);
});

I will provide a minimal test suite soon.

Another note. I tried using the develop version of the scenarioo-js library to see if the issue is fixed there, but the current code fails due to a syntax error in scenarioo.js.

you can reproduce this by installing the develop version from github and running any test suite:

npm install https://github.com/scenarioo/scenarioo-js\#develop

[bruderol]

Hi @sbrugnoni,

thanks for your great feedback, may I ask where you heared about us? Would be cool if you spread the word further about scenarioo.org and let us know about any further feedback.

You can tweet with and about us here https://twitter.com/scenarioo_org

And by the way: would be great if you give us a star on github :-)

your feedback and the mentioned issue is important and I therefore assigned it to the next release milestone, such that we look into it further and whether we can fix it, if we can reproduce it.

As soon as you send us a minimal example to reproduce it (as PR or github repo, if possible) we can have a further look. Thanks.

What we have to look at in any cae your last comment about develop branch currently not working, which is why I opened another issue #83

Thanks.

[sbrugnoni]

I finally managed to find time to create the minimal example, see link below.

• https://github.com/sbrugnoni/scenarioo-js-minimal-example

I discovered the project at a selenium meetup in Zurich in January, where @adiherzog presented the tool.

Cheers.

[davidlahner]

@sbrugnoni Thank you for the example. I was able to replicate and fix the problem. As soon as the pull request #85 is merged it will be available in develop (which compiles again, thanks for the hint that we had an issue there), if you want to take another look. We hope to release a new version of scenarioo and scenarioo-js in the next couple of months, which would then include these fixes.