Surprising Failures w/ Data-Driven Tests

[aedenj]

Below is a data-driven test that I've written that compares screen shots using the compare program that comes with ImageMagick.

var exec = require('child_process').execFile;
var fs = require('fs');
var viewports = [
  {
    width: 320,
    height: 568
  },
  {
    width: 768,
    height: 1024
  },
  {
    width: 1024,
    height: 768
  },
  {
    width: 1366,
    height: 768
  }
];

viewports.forEach(function(view) {
  describe('when navigating to the '.concat(view.width, 'x', view.height, ' poa page in English'), function() {
    var actualScreenshot = './tmp/actual-'.concat(view.width, 'x', view.height, '-en.png');
    var expectedScreenshot =  './opening-screen/'.concat(view.width, 'x', view.height, '-en.png');
    before(function() {
      var url = 'http://10.12.12.12:5000/poa?lang=en';
      casper.start(url);
      casper.viewport(view.width, view.height, function() {
        casper.capture(actualScreenshot);
      });
    });

    it('matches the expected UI', function(done) {
      var compareArgs = [
        '-metric',
        'PSNR',
        actualScreenshot,
        expectedScreenshot,
        'difference.png'];
      exec('compare', compareArgs, null, function(error, stdout, stderr) {
        //Unfortunately the compare utility redirects output to stderr instead of 
        //stdout.  The string "inf" means the images are identical.
        expect(stderr).to.equal("inf");
        done(error);
      });
    }); //end of it
  }); //end of describe
}); //end of viewports

The tests above all pass, but when I add another describe block like the one above except I replace the language en with fr every test after the fifth, of the now eight total, fails. The output from the runner for both sets of tests in the same file is,

when navigating to the 320x568 poa page in English
    ✓ matches the expected UI (63ms)

  when navigating to the 768x1024 poa page in English
    ✓ matches the expected UI (189ms)

  when navigating to the 1024x768 poa page in English
    ✓ matches the expected UI (190ms)

  when navigating to the 1366x768 poa page in English
    ✓ matches the expected UI (249ms)

  when navigating to the 320x568 poa page in French
    ✓ matches the expected UI (56ms)

  when navigating to the 768x1024 poa page in French
    1) "before all" hook

  when navigating to the 1024x768 poa page in French
    2) "before all" hook

  when navigating to the 1366x768 poa page in French
    3) "before all" hook

  5 passing (15s)
  3 failing

  1) when navigating to the 768x1024 poa page in French "before all" hook:
     Load timeout of (10000ms)

  2) when navigating to the 1024x768 poa page in French "before all" hook:
     Load timeout of (10000ms)

  3) when navigating to the 1366x768 poa page in French "before all" hook:
     Load timeout of (10000ms)

Now if I put the two tests in separate files and run them with two different commands like this

> mocha-casperjs english-test.js
> mocha-casperjs french-test.js

then both pass, but strangely if I run them like this mocha-casperjs english-test.js french-test.js I get the same failure pattern as above. Why do you think the tests are falling?

[nathanboktae]

You can only call casper.start only once. It resets a bunch of state when you do. I think since your doing a start on the same URL you are getting away with it initially.

Try casper.navigate or casper.page.navigate (I could be wrong on a mobile ATM) to do subsequent page navigations.

More of an issue with Casper iirc, I'll keep this open and follow up.

[aedenj]

The functions casper.navigate and casper.page.navigate don't exist as far as I can tell. I did find and use casper.thenOpen and so my test looks like this,

casper.start();

viewports.forEach(function(view) {
  describe('when navigating to the '.concat(view.width, 'x', view.height, ' poa page in English'), function() {
    var actualScreenshot = './tmp/actual-'.concat(view.width, 'x', view.height, '-en.png');
    var expectedScreenshot =  './opening-screen/'.concat(view.width, 'x', view.height, '-en.png');
    before(function() {
      var url = 'http://10.12.12.12:5000/poa?lang=en';
      casper.thenOpen(url, function() { 
        casper.viewport(view.width, view.height, function() {
          casper.capture(actualScreenshot);
        });
      });
    });

    it('matches the expected UI', function(done) {
      var compareArgs = [
        '-metric',
        'PSNR',
        actualScreenshot,
        expectedScreenshot,
        'difference.png'];
      exec('compare', compareArgs, null, function(error, stdout, stderr) {
        //Unfortunately the compare utility redirects output to stderr instead of 
        //stdout.  The string "inf" means the images are identical.
        expect(stderr).to.equal("inf");
        done(error);
      });
    }); //end of it
  }); //end of describe
}); //end of viewports

which still consistently results in this failure pattern where only the sixth one is failing. Bizarre!

 when navigating to the 320x568 poa page in English
    ✓ matches the expected UI (55ms)

  when navigating to the 768x1024 poa page in English
    ✓ matches the expected UI (189ms)

  when navigating to the 1024x768 poa page in English
    ✓ matches the expected UI (196ms)

  when navigating to the 1366x768 poa page in English
    ✓ matches the expected UI (254ms)

  when navigating to the 320x568 poa page in French
    ✓ matches the expected UI (58ms)

  when navigating to the 768x1024 poa page in French
    1) "before all" hook

  when navigating to the 1024x768 poa page in French
    ✓ matches the expected UI (192ms)

  when navigating to the 1366x768 poa page in French
    ✓ matches the expected UI (249ms)

  7 passing (15s)
  1 failing

  1) when navigating to the 768x1024 poa page in French "before all" hook:
     Load timeout of (10000ms)

[nathanboktae]

Yup casper.thenOpen is what you want (I couldn't remember off of the top of my head)

No reason to do a pageload for every variation of screen widths. Browsers can change the viewport size on the fly :tada:

[aedenj]

I'm not sure what you mean by "No reason to do a pageload for every variation of screen widths. Browsers can change the viewport size on the fly"

[aedenj]

I think I see what you mean. I don't need to call casper.thenOpen for every viewport size.

[aedenj]

If I increase the timeout then all of the tests pass. So that begs the question of when the clock for the load timeout starts and ends. Any ideas?

[aedenj]

What's even more bizarre is if I set casper.options.timeout = 1000 then the first test will fail like so,

1) when navigating to the 320x568 poa page in English "before all" hook:
     Load timeout of (1000ms)

but all the other tests will pass. So something strange is happening with timeout behavior it seems.

[nathanboktae]

If I increase the timeout then all of the tests pass. So that begs the question of when the clock for the load timeout starts and ends. Any ideas?

Could it be that your site sometimes takes that long to load? --log-level=debug says a lot. I always use it.

As you mentioned in #28 I have seen this before. Are you on the latest version? That being said, I could add more test cases as this is tricky to get right.

Also, anytime you do something with casper in a test or hook, you should make sure it is put into a step so mocha-casperjs knows to hold up mocha on casper running. I need to expand the readme with this and tips for writing tests.

[aedenj]

I am on the latest, 0.44 ....

I think I understand what's happening, but don't know why. If you take a look below the timeout clock doesn't start and stop after each describe. Lines such as [info] [phantom] Done 5 steps in 1767ms indicate that the time just keeps accumulating after each test fixture/describe block. I would expect the timeout clock to stop after each describe block. Thoughts?

when navigating to the 320x568 poa page in English
[info] [phantom] Running suite: 3 steps
....
[info] [phantom] Step anonymous 5/5: done in 1752ms.
[info] [phantom] Done 5 steps in 1767ms
    ✓ matches the expected UI forEnglish (42ms)

  when navigating to the 768x1024 poa page in English
[info] [phantom] Running suite: 8 steps
....
[info] [phantom] Step anonymous 10/10: done in 3538ms.
[info] [phantom] Done 10 steps in 3554ms
    ✓ matches the expected UI forEnglish (42ms)

when navigating to the 1024x768 poa page in English
[info] [phantom] Running suite: 13 steps
....
[info] [phantom] Step anonymous 15/15: done in 5642ms.
[info] [phantom] Done 15 steps in 5658ms
    ✓ matches the expected UI forEnglish (42ms)

  when navigating to the 1366x768 poa page in English
[info] [phantom] Running suite: 18 steps
....
[info] [phantom] Step anonymous 20/20: done in 7551ms.
[info] [phantom] Done 20 steps in 7566ms
    ✓ matches the expected UI forEnglish (42ms)

[nathanboktae]

casper.timeout is one overall timeout - see #32