Running Individual Tests Produces Different Results Than Initial Run All

[niedzielski]

Node v6.4.0, Gemini v4.8.0, Gemini GUI v4.3.2

When I launch the GUI to run all tests, they all pass. I can keep running all tests successfully. When I start running individual tests, some fail and some pass. When I try to run all after an individual failure, the results come back much more quickly and always fail. The command line output also shows an incorrect total.

$ http-server -c-1&
$ phantomjs -w&
$ gemini-gui foo.js
/home/stephen/.node/lib/node_modules
GUI is running at http://localhost:8000
✓ zzz-card 160x144 [phantomjs]
✓ zzz-card 320x240 [phantomjs]
✓ zzz-card 480x320 [phantomjs]
✓ zzz-card 640x480 [phantomjs]
✓ zzz-card 1024x768 [phantomjs]
✓ zzz-photo 160x144 [phantomjs]
✓ zzz-photo 320x240 [phantomjs]
✓ zzz-photo 480x320 [phantomjs]
✓ zzz-photo 640x480 [phantomjs]
✓ zzz-photo 1024x768 [phantomjs]
Total: 10 Passed: 10 Failed: 0 Skipped: 0 Retries: 0
✓ zzz-card 160x144 [phantomjs]
✓ zzz-card 320x240 [phantomjs]
✓ zzz-card 480x320 [phantomjs]
✓ zzz-card 640x480 [phantomjs]
✓ zzz-card 1024x768 [phantomjs]
✓ zzz-photo 160x144 [phantomjs]
✓ zzz-photo 320x240 [phantomjs]
✓ zzz-photo 480x320 [phantomjs]
✓ zzz-photo 640x480 [phantomjs]
✓ zzz-photo 1024x768 [phantomjs]
Total: 10 Passed: 10 Failed: 0 Skipped: 0 Retries: 0
✓ zzz-card 160x144 [phantomjs]
✓ zzz-card 320x240 [phantomjs]
✓ zzz-card 480x320 [phantomjs]
✓ zzz-card 640x480 [phantomjs]
✓ zzz-card 1024x768 [phantomjs]
✓ zzz-photo 160x144 [phantomjs]
✓ zzz-photo 320x240 [phantomjs]
✓ zzz-photo 480x320 [phantomjs]
✓ zzz-photo 640x480 [phantomjs]
✓ zzz-photo 1024x768 [phantomjs]
Total: 10 Passed: 10 Failed: 0 Skipped: 0 Retries: 0
✓ zzz-card 160x144 [phantomjs]
✓ zzz-card 320x240 [phantomjs]
✓ zzz-card 480x320 [phantomjs]
✓ zzz-card 640x480 [phantomjs]
✓ zzz-card 1024x768 [phantomjs]
✓ zzz-photo 160x144 [phantomjs]
✓ zzz-photo 320x240 [phantomjs]
✓ zzz-photo 480x320 [phantomjs]
✓ zzz-photo 640x480 [phantomjs]
✓ zzz-photo 1024x768 [phantomjs]
Total: 10 Passed: 10 Failed: 0 Skipped: 0 Retries: 0
✓ zzz-card 160x144 [phantomjs]
Total: 1 Passed: 1 Failed: 0 Skipped: 0 Retries: 0
✓ zzz-card 160x144 [phantomjs]
✓ zzz-card 320x240 [phantomjs]
✓ zzz-card 480x320 [phantomjs]
✓ zzz-card 640x480 [phantomjs]
✓ zzz-card 1024x768 [phantomjs]
✓ zzz-photo 160x144 [phantomjs]
✓ zzz-photo 320x240 [phantomjs]
✓ zzz-photo 480x320 [phantomjs]
✓ zzz-photo 640x480 [phantomjs]
✓ zzz-photo 1024x768 [phantomjs]
Total: 10 Passed: 10 Failed: 0 Skipped: 0 Retries: 0
✘ zzz-card 320x240 [phantomjs]
Total: 1 Passed: 0 Failed: 1 Skipped: 0 Retries: 0
✓ zzz-card 160x144 [phantomjs]
✓ zzz-card 320x240 [phantomjs]
✓ zzz-card 480x320 [phantomjs]
✓ zzz-card 640x480 [phantomjs]
✓ zzz-card 1024x768 [phantomjs]
✓ zzz-photo 160x144 [phantomjs]
✓ zzz-photo 320x240 [phantomjs]
✓ zzz-photo 480x320 [phantomjs]
✓ zzz-photo 640x480 [phantomjs]
✓ zzz-photo 1024x768 [phantomjs]
Total: 10 Passed: 10 Failed: 0 Skipped: 0 Retries: 0
✘ zzz-card 320x240 [phantomjs]
Total: 1 Passed: 0 Failed: 1 Skipped: 0 Retries: 0
✘ zzz-card 320x240 [phantomjs]
Total: 1 Passed: 0 Failed: 1 Skipped: 0 Retries: 0
✘ zzz-card 320x240 [phantomjs]
Total: 1 Passed: 0 Failed: 1 Skipped: 0 Retries: 0
✘ zzz-card 320x240 [phantomjs]
Total: 1 Passed: 0 Failed: 1 Skipped: 0 Retries: 0
✘ zzz-card 320x240 [phantomjs]
Total: 1 Passed: 0 Failed: 1 Skipped: 0 Retries: 0
✘ zzz-card 320x240 [phantomjs]
Total: 1 Passed: 0 Failed: 1 Skipped: 0 Retries: 0

.gemini.yml

rootUrl: http://localhost:8080
browsers:
  phantomjs:
    gridUrl: http://localhost:8910
    desiredCapabilities:
      browserName: phantomjs

foo.js

const phantom = require('phantomjs')

const viewports = [{w: 160, h: 144}, {w: 320, h: 240}, {w: 480, h: 320},
  {w: 640, h: 480}]

gemini.suite('zzz-card', function(suite) {
  suite
    .setUrl('/test/screenshot/fixture/zzz-card.html')
    .setCaptureElements('body')
    .before(function(actions, find) {
      actions.wait(1000)
    })
  viewports.forEach(function(viewport) {
    suite
      .capture('zzz-card-' + viewport.w + 'x' + viewport.h, function(actions, find) {
        actions.setWindowSize(viewport.w, viewport.h)
      })
  })
})

[sipayRT]

hi. can you reproduce it now? Can you provide your page for testing?

[niedzielski]

o/

Looks like it's still happening with Node v6.5.0, Gemini v4.9.4, and Gemini GUI v4.3.3 :/ You are welcome to view my testing repo in full (just run npm i && make, I think). There's not much there. Thanks!

[sipayRT]

got error while trying to run tests:

(master)% make                                                                                                                                            ~/sites/tmp/lizzztlib
parallelshell 'http-server -c-1' 'phantomjs -w' 'gemini-gui test/screenshot'
Starting up http-server, serving ./
Available on:
  http://127.0.0.1:8080
  http://5.45.206.152:8080
Hit CTRL-C to stop the server
[ERROR - 2016-10-07T08:51:29.183Z] GhostDriver - main.fail - {"line":85,"sourceURL":"phantomjs://code/main.js","stack":"global code@phantomjs://code/main.js:85:56"}

  phantomjs://platform/console++.js:263 in error
http-server stopped.
make: *** [test] Error 1

[sipayRT]

in your case I got different results because of image loading:

So while gemini getting block coordinates image does not load yet

[niedzielski]

Thank you for looking into this! I'm sure your analysis is correct but I'm afraid I'm not as aware of the internal workings of Polymer as I should be. I don't know if that means this issue is user error and the test just "happened to work" for the initial execution due to a race condition. If so, please close this issue. If there's room for improvement in the Gemini GUI individual test implementation, please leave this open. Thanks!

[sipayRT]

To avoid those errors you can use base64 for images instead of <img> tag