Inconsistent browser behaviour

xcambar commented 8 years ago

Hi, I use Browserstack (and browserstack-local) to take screenshots and do visual regression testing. The tests are run on many (14) browsers, each configured within nightwatch.js.

When run in CI (CircleCI), from one build to another, there is usually a couple of browsers that do not start then quit immediately without the possibility of running any test.

Has this behaviour even been reported before? I'm more than happy to help in finding the root cause of this and figure out a solution (or at least a workaround).

Best,

tr4n2uil commented 8 years ago

Glad you reported, but its not any clear what exactly the issue you are facing ? Could you please elaborate, with steps on reproducing the issue.

xcambar commented 8 years ago

Sure! At the end is the (partial) config nightwatch.js uses to run the tests.

Here's an example of output in CI (output is by browserstack-local):

Started child process for: chrome 52 environment 
Started child process for: chrome 51 environment 
Started child process for: edge 14 environment 
Started child process for: edge 13 environment 
Started child process for: ie 11 environment 
Started child process for: ie 10 environment 
Started child process for: ie 9 environment 
Started child process for: safari 9.1 environment 
Started child process for: safari 9 environment 
Started child process for: safari 8 environment 
Started child process for: safari 7.1 environment 
Started child process for: safari 7 environment 

  >> edge 14 environment finished.  
  >> safari 7.1 environment finished.  
  >> chrome 52 environment finished.  
  >> edge 13 environment finished.  
  >> ie 10 environment finished.  
  >> chrome 51 environment finished.  
  >> ie 11 environment finished.  
  >> safari 7 environment finished.  
  >> safari 9.1 environment finished.  
  >> safari 9 environment finished.  
  >> ie 9 environment finished.  
  >> safari 8 environment finished.

The weird thing is that for a browser that runs as expected, I get a lot of output (~50 screenshots per browser), but, despite the "BROWSER_NAME environment finished" log message, some browsers have not executed a single assertion, which is proven by the lack of output for the assertions.

In summary: the expected behaviour is : log messages for every assertion + many screenshots.

For some (seemingly random at each build) browsers, no log nor screenshots are produced.

Disclaimer: I would blame myself and my code if at every run, the same browser was failing. But as it os something Firefox, sometimes Chrome, sometimes any other browser, I start to think the error is not on my side.

Note: A coworker reported to me that we have only 2 workers, and we are trying to run 12 browsers. I don't know if this is relevant for you.

Here's the config for nightwatch:

{
   "selenium" : {
      "start_process" : false,
      "port" : 80,
      "host" : "hub.browserstack.com"
   },
   "test_settings" : {
      "safari 9.1" : {
         "selenium_port" : 80,
         "desiredCapabilities" : {
            "browser_version" : "9.1",
            "javascriptEnabled" : true,
            "browserstack.user" : "MY_USERNAME",
            "browserstack.key" : "MY_KEY",
            "browser" : "safari",
            "acceptSslCerts" : true,
            "browserstack.local" : true
         },
         "silent" : true,
         "screenshots" : {
            "enabled" : true,
            "path" : ""
         },
         "launch_url" : "http://hub.browserstack.com",
         "selenium_host" : "hub.browserstack.com"
      },
      "safari 7.1" : {
         "selenium_host" : "hub.browserstack.com",
         "silent" : true,
         "desiredCapabilities" : {
            "browser_version" : "7.1",
            "javascriptEnabled" : true,
            "browserstack.user" : "MY_USERNAME",
            "browserstack.key" : "MY_KEY",
            "browser" : "safari",
            "acceptSslCerts" : true,
            "browserstack.local" : true
         },
         "selenium_port" : 80,
         "launch_url" : "http://hub.browserstack.com",
         "screenshots" : {
            "enabled" : true,
            "path" : ""
         }
      },
      "ie 11" : {
         "selenium_host" : "hub.browserstack.com",
         "selenium_port" : 80,
         "desiredCapabilities" : {
            "browser_version" : "11",
            "javascriptEnabled" : true,
            "browserstack.user" : "MY_USERNAME",
            "browserstack.key" : "MY_KEY",
            "browser" : "internet explorer",
            "acceptSslCerts" : true,
            "browserstack.local" : true
         },
         "silent" : true,
         "screenshots" : {
            "enabled" : true,
            "path" : ""
         },
         "launch_url" : "http://hub.browserstack.com"
      },
      "safari 9" : {
         "screenshots" : {
            "enabled" : true,
            "path" : ""
         },
         "launch_url" : "http://hub.browserstack.com",
         "selenium_port" : 80,
         "silent" : true,
         "desiredCapabilities" : {
            "browserstack.user" : "MY_USERNAME",
            "browser_version" : "9",
            "javascriptEnabled" : true,
            "browser" : "safari",
            "browserstack.key" : "MY_KEY",
            "browserstack.local" : true,
            "acceptSslCerts" : true
         },
         "selenium_host" : "hub.browserstack.com"
      },
      "ie 9" : {
         "selenium_port" : 80,
         "desiredCapabilities" : {
            "browserstack.user" : "MY_USERNAME",
            "browser_version" : "9",
            "javascriptEnabled" : true,
            "browser" : "internet explorer",
            "browserstack.key" : "MY_KEY",
            "browserstack.local" : true,
            "acceptSslCerts" : true
         },
         "silent" : true,
         "screenshots" : {
            "path" : "",
            "enabled" : true
         },
         "launch_url" : "http://hub.browserstack.com",
         "selenium_host" : "hub.browserstack.com"
      },
      "edge 13" : {
         "silent" : true,
         "desiredCapabilities" : {
            "acceptSslCerts" : true,
            "browserstack.local" : true,
            "browserstack.key" : "MY_KEY",
            "browser" : "edge",
            "browser_version" : "13",
            "javascriptEnabled" : true,
            "browserstack.user" : "MY_USERNAME"
         },
         "selenium_port" : 80,
         "launch_url" : "http://hub.browserstack.com",
         "screenshots" : {
            "enabled" : true,
            "path" : ""
         },
         "selenium_host" : "hub.browserstack.com"
      },
      "safari 8" : {
         "selenium_host" : "hub.browserstack.com",
         "launch_url" : "http://hub.browserstack.com",
         "screenshots" : {
            "path" : "",
            "enabled" : true
         },
         "desiredCapabilities" : {
            "acceptSslCerts" : true,
            "browserstack.local" : true,
            "browser_version" : "8",
            "javascriptEnabled" : true,
            "browserstack.user" : "MY_USERNAME",
            "browserstack.key" : "MY_KEY",
            "browser" : "safari"
         },
         "silent" : true,
         "selenium_port" : 80
      },
      "edge 14" : {
         "selenium_host" : "hub.browserstack.com",
         "desiredCapabilities" : {
            "browserstack.user" : "MY_USERNAME",
            "javascriptEnabled" : true,
            "browser_version" : "14",
            "browser" : "edge",
            "browserstack.key" : "MY_KEY",
            "browserstack.local" : true,
            "acceptSslCerts" : true
         },
         "silent" : true,
         "selenium_port" : 80,
         "launch_url" : "http://hub.browserstack.com",
         "screenshots" : {
            "enabled" : true,
            "path" : ""
         }
      },
      "chrome 51" : {
         "screenshots" : {
            "enabled" : true,
            "path" : ""
         },
         "launch_url" : "http://hub.browserstack.com",
         "selenium_port" : 80,
         "desiredCapabilities" : {
            "browser" : "chrome",
            "browserstack.key" : "MY_KEY",
            "browserstack.user" : "MY_USERNAME",
            "javascriptEnabled" : true,
            "browser_version" : "51",
            "browserstack.local" : true,
            "acceptSslCerts" : true
         },
         "silent" : true,
         "selenium_host" : "hub.browserstack.com"
      },
      "chrome 52" : {
         "desiredCapabilities" : {
            "browserstack.key" : "MY_KEY",
            "browser" : "chrome",
            "browser_version" : "52",
            "javascriptEnabled" : true,
            "browserstack.user" : "MY_USERNAME",
            "acceptSslCerts" : true,
            "browserstack.local" : true
         },
         "silent" : true,
         "selenium_port" : 80,
         "launch_url" : "http://hub.browserstack.com",
         "screenshots" : {
            "enabled" : true,
            "path" : ""
         },
         "selenium_host" : "hub.browserstack.com"
      },
      "safari 7" : {
         "screenshots" : {
            "enabled" : true,
            "path" : ""
         },
         "launch_url" : "http://hub.browserstack.com",
         "selenium_port" : 80,
         "silent" : true,
         "desiredCapabilities" : {
            "acceptSslCerts" : true,
            "browserstack.local" : true,
            "javascriptEnabled" : true,
            "browser_version" : "7",
            "browserstack.user" : "MY_USERNAME",
            "browserstack.key" : "MY_KEY",
            "browser" : "safari"
         },
         "selenium_host" : "hub.browserstack.com"
      },
      "ie 10" : {
         "selenium_host" : "hub.browserstack.com",
         "selenium_port" : 80,
         "silent" : true,
         "desiredCapabilities" : {
            "browserstack.local" : true,
            "acceptSslCerts" : true,
            "browserstack.user" : "MY_USERNAME",
            "browser_version" : "10",
            "javascriptEnabled" : true,
            "browser" : "internet explorer",
            "browserstack.key" : "MY_KEY"
         },
         "screenshots" : {
            "path" : "",
            "enabled" : true
         },
         "launch_url" : "http://hub.browserstack.com"
      }
   },
   "browserstack_template" : {
      "desiredCapabilities" : {
         "browserstack.local" : true,
         "acceptSslCerts" : true,
         "browserstack.key" : "MY_KEY",
         "browserstack.user" : "MY_USERNAME",
         "javascriptEnabled" : true
      },
      "silent" : true,
      "selenium_port" : 80,
      "launch_url" : "http://hub.browserstack.com",
      "screenshots" : {
         "enabled" : true,
         "path" : ""
      },
      "selenium_host" : "hub.browserstack.com"
   },
   "test_workers" : {
      "enabled" : true,
      "workers" : "2"
   },
   "src_folders" : [
      "./tests"
   ]
}

xcambar commented 8 years ago

Run some nightwatch.js tests with this config, and that should do.

If you want a full example, I can still do it, but it will take more time, obviously.

tr4n2uil commented 8 years ago

Ran and it works for me. Can you share one of your the test files ?

tr4n2uil commented 8 years ago

Please refer for proper integration of nightwatch with browserstack-local: nightwatch-browserstack

xcambar commented 8 years ago

I'll build a complete demo. please give me a couple of days to do so. Meanwhile, here's a complete log from CircleCI, as you can see, many browsers are started, but only a few of them have output data.

For instance, in this build, Chrome has started and finished, but did not output anything.

BrowserStack Local started
Started child process for: chrome 52 environment 
Started child process for: chrome 51 environment 
Started child process for: edge 14 environment 
Started child process for: edge 13 environment 
Started child process for: ie 11 environment 
Started child process for: ie 10 environment 
Started child process for: ie 9 environment 
Started child process for: safari 9.1 environment 
Started child process for: safari 9 environment 
Started child process for: safari 8 environment 
Started child process for: safari 7.1 environment 
Started child process for: safari 7 environment 

  >> edge 14 environment finished.  

  >> safari 7 environment finished.  

  >> ie 10 environment finished.  

  >> edge 13 environment finished.  

  >> safari 7.1 environment finished.  

  >> chrome 51 environment finished.  

  >> chrome 52 environment finished.  

  >> ie 9 environment finished.  

  >> ie 11 environment finished.  

  >> safari 9 environment finished.  

  >> safari 9.1 environment finished.  

;37m    at Object.ok (/home/ubuntu/peopledoc-ui/node_modules/nightwatch/lib/core/api.js:67:24)
    at /home/ubuntu/peopledoc-ui/tests/visual-capture.js:118:45
    at process._tickCallback (node.js:368:9)
 ie 9   ✖ Failed [ok]: (Error: Block fieldset has been altered by 18.23%.)  - expected "true" but got: false
 ie 9       at Object.ok (/home/ubuntu/peopledoc-ui/node_modules/nightwatch/lib/core/api.js:67:24)
    at /home/ubuntu/peopledoc-ui/tests/visual-capture.js:118:45
    at process._tickCallback (node.js:368:9)
 ie 9   ✖ Failed [ok]: (Error: Block file-upload has been altered by 6.64%.)  - expected "true" but got: false
 ie 9       at Object.ok (/home/ubuntu/peopledoc-ui/node_modules/nightwatch/lib/core/api.js:67:24)
    at /home/ubuntu/peopledoc-ui/tests/visual-capture.js:118:45
    at process._tickCallback (node.js:368:9)
 ie 9   ✖ Failed [ok]: (Error: Block filter has been altered by 14.38%.)  - expected "true" but got: false
 ie 9       at Object.ok (/home/ubuntu/peopledoc-ui/node_modules/nightwatch/lib/core/api.js:67:24)
    at /home/ubuntu/peopledoc-ui/tests/visual-capture.js:118:45
    at process._tickCallback (node.js:368:9)
 ie 9   ✖ Failed [ok]: (Error: Block footer has been altered by 2.46%.)  - expected "true" but got: false
 ie 9       at Object.ok (/home/ubuntu/peopledoc-ui/node_modules/nightwatch/lib/core/api.js:67:24)
    at /home/ubuntu/peopledoc-ui/tests/visual-capture.js:118:45
    at process._tickCallback (node.js:368:9)
 ie 9   ✖ Failed [ok]: (Error: Block form-error has been altered by 27.84%.)  - expected "true" but got: false
 ie 9       at Object.ok (/home/ubuntu/peopledoc-ui/node_modules/nightwatch/lib/core/api.js:67:24)
    at /home/ubuntu/peopledoc-ui/tests/visual-capture.js:118:45
    at process._tickCallback (node.js:368:9)
 ie 9   ✖ Failed [ok]: (Error: Block form-wizard has been altered by 6.84%.)  - expected "true" but got: false
 ie 9       at Object.ok (/home/ubuntu/peopledoc-ui/node_modules/nightwatch/lib/core/api.js:67:24)
    at /home/ubuntu/peopledoc-ui/tests/visual-capture.js:118:45
    at process._tickCallback (node.js:368:9)
 ie 9   ✖ Failed [ok]: (Error: Block header has been altered by 38.78%.)  - expected "true" but got: false
 ie 9       at Object.ok (/home/ubuntu/peopledoc-ui/node_modules/nightwatch/lib/core/api.js:67:24)
    at /home/ubuntu/peopledoc-ui/tests/visual-capture.js:118:45
    at process._tickCallback (node.js:368:9)
 ie 9   ✖ Failed [ok]: (Error: Block horizontal-form has been altered by 6.36%.)  - expected "true" but got: false
 ie 9       at Object.ok (/home/ubuntu/peopledoc-ui/node_modules/nightwatch/lib/core/api.js:67:24)
    at /home/ubuntu/peopledoc-ui/tests/visual-capture.js:118:45
    at process._tickCallback (node.js:368:9)
 ie 9   ✖ Failed [ok]: (Error: Block icon has been altered by 3.09%.)  - expected "true" but got: false
    at Object.ok (/home/ubuntu/peopledoc-ui/node_modules/nightwatch/lib/core/api.js:67:24)
    at /home/ubuntu/peopledoc-ui/tests/visual-capture.js:118:45
    at process._tickCallback (node.js:368:9)
 ie 9   ✖ Failed [ok]: (Error: Block input has been altered by 11.78%.)  - expected "true" but got: false
    at Object.ok (/home/ubuntu/peopledoc-ui/node_modules/nightwatch/lib/core/api.js:67:24)
    at /home/ubuntu/peopledoc-ui/tests/visual-capture.js:118:45
    at process._tickCallback (node.js:368:9)
 ie 9   ✖ Failed [ok]: (Error: Block label has been altered by 9.07%.)  - expected "true" but got: false
 ie 9       at Object.ok (/home/ubuntu/peopledoc-ui/node_modules/nightwatch/lib/core/api.js:67:24)
    at /home/ubuntu/peopledoc-ui/tests/visual-capture.js:118:45
    at process._tickCallback (node.js:368:9)
 ie 9   ✖ Failed [ok]: (Error: Block main has been altered by 3.84%.)  - expected "true" but got: false
 ie 9       at Object.ok (/home/ubuntu/peopledoc-ui/node_modules/nightwatch/lib/core/api.js:67:24)
    at /home/ubuntu/peopledoc-ui/tests/visual-capture.js:118:45
    at process._tickCallback (node.js:368:9)
 ie 9   ✖ Failed [ok]: (Error: Block modal has been altered by 11.01%.)  - expected "true" but got: false
 ie 9       at Object.ok (/home/ubuntu/peopledoc-ui/node_modules/nightwatch/lib/core/api.js:67:24)
    at /home/ubuntu/peopledoc-ui/tests/visual-capture.js:118:45
    at process._tickCallback (node.js:368:9)
 ie 9   ✖ Failed [ok]: (Error: Block popover has been altered by 11.12%.)  - expected "true" but got: false
 ie 9       at Object.ok (/home/ubuntu/peopledoc-ui/node_modules/nightwatch/lib/core/api.js:67:24)
    at /home/ubuntu/peopledoc-ui/tests/visual-capture.js:118:45
    at process._tickCallback (node.js:368:9)
 ie 9   ✖ Failed [ok]: (Error: Block progress-bar has been altered by 9.88%.)  - expected "true" but got: false
 ie 9       at Object.ok (/home/ubuntu/peopledoc-ui/node_modules/nightwatch/lib/core/api.js:67:24)
    at /home/ubuntu/peopledoc-ui/tests/visual-capture.js:118:45
    at process._tickCallback (node.js:368:9)
 ie 9   ✖ Failed [ok]: (Error: Block radio has been altered by 3.91%.)  - expected "true" but got: false
 ie 9       at Object.ok (/home/ubuntu/peopledoc-ui/node_modules/nightwatch/lib/core/api.js:67:24)
    at /home/ubuntu/peopledoc-ui/tests/visual-capture.js:118:45
    at process._tickCallback (node.js:368:9)
 ie 9   ✖ Failed [ok]: (Error: Block search has been altered by 15.00%.)  - expected "true" but got: false
 ie 9       at Object.ok (/home/ubuntu/peopledoc-ui/node_modules/nightwatch/lib/core/api.js:67:24)
    at /home/ubuntu/peopledoc-ui/tests/visual-capture.js:118:45
    at process._tickCallback (node.js:368:9)
 ie 9   ✖ Failed [ok]: (Error: Block select has been altered by 11.28%.)  - expected "true" but got: false
 ie 9       at Object.ok (/home/ubuntu/peopledoc-ui/node_modules/nightwatch/lib/core/api.js:67:24)
    at /home/ubuntu/peopledoc-ui/tests/visual-capture.js:118:45
    at process._tickCallback (node.js:368:9)
 ie 9   ✖ Failed [ok]: (Error: Block sidebar has been altered by 17.33%.)  - expected "true" but got: false
 ie 9       at Object.ok (/home/ubuntu/peopledoc-ui/node_modules/nightwatch/lib/core/api.js:67:24)
    at /home/ubuntu/peopledoc-ui/tests/visual-capture.js:118:45
    at process._tickCallback (node.js:368:9)
 ie 9   ✖ Failed [ok]: (Error: Block standout-text has been altered by 2.85%.)  - expected "true" but got: false
 ie 9       at Object.ok (/home/ubuntu/peopledoc-ui/node_modules/nightwatch/lib/core/api.js:67:24)
    at /home/ubuntu/peopledoc-ui/tests/visual-capture.js:118:45
    at process._tickCallback (node.js:368:9)
 ie 9   ✖ Failed [ok]: (Error: Block table has been altered by 10.24%.)  - expected "true" but got: false
 ie 9       at Object.ok (/home/ubuntu/peopledoc-ui/node_modules/nightwatch/lib/core/api.js:67:24)
    at /home/ubuntu/peopledoc-ui/tests/visual-capture.js:118:45
    at process._tickCallback (node.js:368:9)
 ie 9   ✖ Failed [ok]: (Error: Block tabs has been altered by 16.17%.)  - expected "true" but got: false
 ie 9       at Object.ok (/home/ubuntu/peopledoc-ui/node_modules/nightwatch/lib/core/api.js:67:24)
    at /home/ubuntu/peopledoc-ui/tests/visual-capture.js:118:45
    at process._tickCallback (node.js:368:9)
 ie 9   ✖ Failed [ok]: (Error: Block textarea has been altered by 25.03%.)  - expected "true" but got: false
 ie 9       at Object.ok (/home/ubuntu/peopledoc-ui/node_modules/nightwatch/lib/core/api.js:67:24)
    at /home/ubuntu/peopledoc-ui/tests/visual-capture.js:118:45
    at process._tickCallback (node.js:368:9)
 ie 9   ✖ Failed [ok]: (Error: Block title has been altered by 1.42%.)  - expected "true" but got: false
 ie 9       at Object.ok (/home/ubuntu/peopledoc-ui/node_modules/nightwatch/lib/core/api.js:67:24)
    at /home/ubuntu/peopledoc-ui/tests/visual-capture.js:118:45
    at process._tickCallback (node.js:368:9)
 ie 9   ✖ Failed [ok]: (Error: Block tooltip has been altered by 8.35%.)  - expected "true" but got: false
 ie 9       at Object.ok (/home/ubuntu/peopledoc-ui/node_modules/nightwatch/lib/core/api.js:67:24)
    at /home/ubuntu/peopledoc-ui/tests/visual-capture.js:118:45
    at process._tickCallback (node.js:368:9)
 ie 9   ✖ Failed [ok]: (Error: Block tree has been altered by 6.74%.)  - expected "true" but got: false
 ie 9       at Object.ok (/home/ubuntu/peopledoc-ui/node_modules/nightwatch/lib/core/api.js:67:24)
    at /home/ubuntu/peopledoc-ui/tests/visual-capture.js:118:45
    at process._tickCallback (node.js:368:9)
 ie 9   ✖ Failed [ok]: (Error: Block vertical-menu has been altered by 34.75%.)  - expected "true" but got: false
 ie 9       at Object.ok (/home/ubuntu/peopledoc-ui/node_modules/nightwatch/lib/core/api.js:67:24)
    at /home/ubuntu/peopledoc-ui/tests/visual-capture.js:118:45
    at process._tickCallback (node.js:368:9)
 ie 9   FAILED:  44 assertions failed (6m 22s / 382606ms)
 ie 9   
 ie 9   ----------------------------------------------------
 ie 9   TEST FAILURE: 44 assertions failed, 0 passed (6m 23s)
 ie 9    ✖ visual-capture
 ie 9   - capture
 ie 9   Failed [ok]: (Error: Block alert has been altered by 32.13%.) - Expected "true" but got: "false"
 ie 9   Failed [ok]: (Error: Block avatar has been altered by 4.81%.) - Expected "true" but got: "false"
 ie 9   Failed [ok]: (Error: Block badge has been altered by 6.70%.) - Expected "true" but got: "false"
 ie 9   Failed [ok]: (Error: Block button has been altered by 16.14%.) - Expected "true" but got: "false"
 ie 9   Failed [ok]: (Error: Block button-group has been altered by 11.36%.) - Expected "true" but got: "false"
 ie 9   Failed [ok]: (Error: Block calendar has been altered by 6.91%.) - Expected "true" but got: "false"
 ie 9   Failed [ok]: (Error: Block card has been altered by 23.13%.) - Expected "true" but got: "false"
 ie 9   Failed [ok]: (Error: Block carousel has been altered by 5.96%.) - Expected "true" but got: "false"
 ie 9   Failed [ok]: (Error: Block checkable-block has been altered by 6.42%.) - Expected "true" but got: "false"
 ie 9   Failed [ok]: (Error: Block checkbox has been altered by 5.95%.) - Expected "true" but got: "false"
 ie 9   Failed [ok]: (Error: Block date-picker has been altered by 9.39%.) - Expected "true" but got: "false"
 ie 9   Failed [ok]: (Error: Block document-thumbnail has been altered by 6.63%.) - Expected "true" but got: "false"
 ie 9   Failed [ok]: (Error: Block document-viewer has been altered by 33.13%.) - Expected "true" but got: "false"
 ie 9   Failed [ok]: (Error: Block draggable has been altered by 13.59%.) - Expected "true" but got: "false"
 ie 9   Failed [ok]: (Error: Block dropdown has been altered by 15.69%.) - Expected "true" but got: "false"
     Failed [ok]: (Error: Block droppable has been altered by 8.42%.) - Expected "true" but got: "false"
 ie 9   Failed [ok]: (Error: Block dynamic-list has been altered by 35.07%.) - Expected "true" but got: "false"
 ie 9   Failed [ok]: (Error: Block fieldset has been altered by 18.23%.) - Expected "true" but got: "false"
 ie 9   Failed [ok]: (Error: Block file-upload has been altered by 6.64%.) - Expected "true" but got: "false"
 ie 9   Failed [ok]: (Error: Block filter has been altered by 14.38%.) - Expected "true" but got: "false"
 ie 9   Failed [ok]: (Error: Block footer has been altered by 2.46%.) - Expected "true" but got: "false"
 ie 9   Failed [ok]: (Error: Block form-error has been altered by 27.84%.) - Expected "true" but got: "false"
 ie 9   Failed [ok]: (Error: Block form-wizard has been altered by 6.84%.) - Expected "true" but got: "false"
 ie 9   Failed [ok]: (Error: Block header has been altered by 38.78%.) - Expected "true" but got: "false"
 ie 9   Failed [ok]: (Error: Block horizontal-form has been altered by 6.36%.) - Expected "true" but got: "false"
 ie 9   Failed [ok]: (Error: Block icon has been altered by 3.09%.) - Expected "true" but got: "false"
 ie 9   Failed [ok]: (Error: Block input has been altered by 11.78%.) - Expected "true" but got: "false"
 ie 9   Failed [ok]: (Error: Block label has been altered by 9.07%.) - Expected "true" but got: "false"
 ie 9   Failed [ok]: (Error: Block main has been altered by 3.84%.) - Expected "true" but got: "false"
 ie 9   Failed [ok]: (Error: Block modal has been altered by 11.01%.) - Expected "true" but got: "false"
 ie 9   Failed [ok]: (Error: Block popover has been altered by 11.12%.) - Expected "true" but got: "false"
 ie 9   Failed [ok]: (Error: Block progress-bar has been altered by 9.88%.) - Expected "true" but got: "false"
 ie 9   Failed [ok]: (Error: Block radio has been altered by 3.91%.) - Expected "true" but got: "false"
 ie 9   Failed [ok]: (Error: Block search has been altered by 15.00%.) - Expected "true" but got: "false"
     Failed [ok]: (Error: Block select has been altered by 11.28%.) - Expected "true" but got: "false"
 ie 9   Failed [ok]: (Error: Block sidebar has been altered by 17.33%.) - Expected "true" but got: "false"
 ie 9   Failed [ok]: (Error: Block standout-text has been altered by 2.85%.) - Expected "true" but got: "false"
 ie 9   Failed [ok]: (Error: Block table has been altered by 10.24%.) - Expected "true" but got: "false"
 ie 9   Failed [ok]: (Error: Block tabs has been altered by 16.17%.) - Expected "true" but got: "false"
 ie 9   Failed [ok]: (Error: Block textarea has been altered by 25.03%.) - Expected "true" but got: "false"
 ie 9   Failed [ok]: (Error: Block title has been altered by 1.42%.) - Expected "true" but got: "false"
 ie 9   Failed [ok]: (Error: Block tooltip has been altered by 8.35%.) - Expected "true" but got: "false"
 ie 9   Failed [ok]: (Error: Block tree has been altered by 6.74%.) - Expected "true" but got: "false"
 ie 9   Failed [ok]: (Error: Block vertical-menu has been altered by 34.75%.) - Expected "true" but got: "false"
 ie 9   
 safari 9.1   [Visual Capture] Test Suite
 safari 9.1   ===========================
 safari 9.1   
 safari 9.1   Results for:  capture
 safari 9.1   ✔ Passed [ok]: No reference for block alert, skipping
 safari 9.1   ✔ Passed [ok]: No reference for block avatar, skipping
 safari 9.1   ✔ Passed [ok]: No reference for block badge, skipping
 safari 9.1   ✔ Passed [ok]: No reference for block button, skipping
 safari 9.1   ✔ Passed [ok]: No reference for block button-group, skipping
 safari 9.1   ✔ Passed [ok]: No reference for block calendar, skipping
 safari 9.1   ✔ Passed [ok]: No reference for block card, skipping
 safari 9.1   ✔ Passed [ok]: No reference for block carousel, skipping
 safari 9.1   ✔ Passed [ok]: No reference for block checkable-block, skipping
 safari 9.1   ✔ Passed [ok]: No reference for block checkbox, skipping
 safari 9.1   ✔ Passed [ok]: No reference for block date-picker, skipping
 safari 9.1   ✔ Passed [ok]: No reference for block document-thumbnail, skipping
 safari 9.1   ✔ Passed [ok]: No reference for block document-viewer, skipping
 safari 9.1   ✔ Passed [ok]: No reference for block draggable, skipping
 safari 9.1   ✔ Passed [ok]: No reference for block dropdown, skipping
 safari 9.1   ✔ Passed [ok]: No reference for block droppable, skipping
 safari 9.1   ✔ Passed [ok]: No reference for block dynamic-list, skipping
 safari 9.1   ✔ Passed [ok]: No reference for block fieldset, skipping
 safari 9.1   ✔ Passed [ok]: No reference for block file-upload, skipping
 safari 9.1   ✔ Passed [ok]: No reference for block filter, skipping
 safari 9.1   ✔ Passed [ok]: No reference for block footer, skipping
 safari 9.1   ✔ Passed [ok]: No reference for block form-error, skipping
 safari 9.1   ✔ Passed [ok]: No reference for block form-wizard, skipping
 safari 9.1   ✔ Passed [ok]: No reference for block header, skipping
 safari 9.1   ✔ Passed [ok]: No reference for block horizontal-form, skipping
 safari 9.1   ✔ Passed [ok]: No reference for block icon, skipping
 safari 9.1   ✔ Passed [ok]: No reference for block input, skipping
 safari 9.1   ✔ Passed [ok]: No reference for block label, skipping
 safari 9.1   ✔ Passed [ok]: No reference for block main, skipping
 safari 9.1   ✔ Passed [ok]: No reference for block modal, skipping
 safari 9.1   ✔ Passed [ok]: No reference for block popover, skipping
 safari 9.1   ✔ Passed [ok]: No reference for block progress-bar, skipping
 safari 9.1   ✔ Passed [ok]: No reference for block radio, skipping
 safari 9.1   ✔ Passed [ok]: No reference for block search, skipping
 safari 9.1   ✔ Passed [ok]: No reference for block select, skipping
 safari 9.1   ✔ Passed [ok]: No reference for block sidebar, skipping
 safari 9.1   ✔ Passed [ok]: No reference for block standout-text, skipping
 safari 9.1   ✔ Passed [ok]: No reference for block table, skipping
 safari 9.1   ✔ Passed [ok]: No reference for block tabs, skipping
 safari 9.1   ✔ Passed [ok]: No reference for block textarea, skipping
 safari 9.1   ✔ Passed [ok]: No reference for block title, skipping
 safari 9.1   ✔ Passed [ok]: No reference for block tooltip, skipping
 safari 9.1   ✔ Passed [ok]: No reference for block tree, skipping
 safari 9.1   ✔ Passed [ok]: No reference for block vertical-menu, skipping
 safari 9.1   OK. 44 assertions passed. (11m 29s / 689363ms)
 safari 9.1   
 safari 9   [Visual Capture] Test Suite
 safari 9   ===========================
 safari 9   
 safari 9   Results for:  capture
 safari 9   ✔ Passed [ok]: Block alert is unchanged
 safari 9   ✔ Passed [ok]: Block avatar is unchanged
 safari 9   ✔ Passed [ok]: Block badge is unchanged
 safari 9   ✔ Passed [ok]: Block button is unchanged
 safari 9   ✔ Passed [ok]: Block button-group is unchanged
 safari 9   ✔ Passed [ok]: Block calendar is unchanged
 safari 9   ✔ Passed [ok]: Block card is unchanged
 safari 9   ✔ Passed [ok]: Block carousel is unchanged
 safari 9   ✔ Passed [ok]: Block checkable-block is unchanged
 safari 9   ✔ Passed [ok]: Block checkbox is unchanged
 safari 9   ✔ Passed [ok]: Block date-picker is unchanged
 safari 9   ✔ Passed [ok]: Block document-thumbnail is unchanged
 safari 9   ✔ Passed [ok]: Block document-viewer is unchanged
 safari 9   ✔ Passed [ok]: Block draggable is unchanged
 safari 9   ✔ Passed [ok]: Block dropdown is unchanged
 safari 9   ✔ Passed [ok]: Block droppable is unchanged
 safari 9   ✔ Passed [ok]: Block dynamic-list is unchanged
 safari 9   ✔ Passed [ok]: Block fieldset is unchanged
 safari 9   ✔ Passed [ok]: Block file-upload is unchanged
 safari 9   ✔ Passed [ok]: Block filter is unchanged
 safari 9   ✔ Passed [ok]: Block footer is unchanged
 safari 9   ✔ Passed [ok]: Block form-error is unchanged
 safari 9   ✔ Passed [ok]: Block form-wizard is unchanged
 safari 9   ✔ Passed [ok]: Block header is unchanged
 safari 9   ✔ Passed [ok]: Block horizontal-form is unchanged
 safari 9   ✔ Passed [ok]: Block icon is unchanged
 safari 9   ✔ Passed [ok]: Block input is unchanged
 safari 9   ✔ Passed [ok]: Block label is unchanged
 safari 9   ✔ Passed [ok]: Block main is unchanged
 safari 9   ✔ Passed [ok]: Block modal is unchanged
 safari 9   ✔ Passed [ok]: Block popover is unchanged
 safari 9   ✔ Passed [ok]: Block progress-bar is unchanged
 safari 9   ✔ Passed [ok]: Block radio is unchanged
 safari 9   ✔ Passed [ok]: Block search is unchanged
 safari 9   ✔ Passed [ok]: Block select is unchanged
 safari 9   ✔ Passed [ok]: Block sidebar is unchanged
 safari 9   ✔ Passed [ok]: Block standout-text is unchanged
 safari 9   ✔ Passed [ok]: Block table is unchanged
 safari 9   ✔ Passed [ok]: Block tabs is unchanged
 safari 9   ✔ Passed [ok]: Block textarea is unchanged
 safari 9   ✔ Passed [ok]: Block title is unchanged
 safari 9   ✔ Passed [ok]: Block tooltip is unchanged
 safari 9   ✔ Passed [ok]: Block tree is unchanged
 safari 9   ✔ Passed [ok]: Block vertical-menu is unchanged
 safari 9   OK. 44 assertions passed. (11m 27s / 687402ms)
 safari 9   
 safari 8   [Visual Capture] Test Suite
 safari 8   ===========================
 safari 8   
 safari 8   Results for:  capture
 safari 8   ✔ Passed [ok]: Block alert is unchanged
 safari 8   ✔ Passed [ok]: Block avatar is unchanged
 safari 8   ✔ Passed [ok]: Block badge is unchanged
 safari 8   ✔ Passed [ok]: Block button is unchanged
 safari 8   ✔ Passed [ok]: Block button-group is unchanged
 safari 8   ✔ Passed [ok]: Block calendar is unchanged
 safari 8   ✔ Passed [ok]: Block card is unchanged
 safari 8   ✔ Passed [ok]: Block carousel is unchanged
 safari 8   ✔ Passed [ok]: Block checkable-block is unchanged
 safari 8   ✔ Passed [ok]: Block checkbox is unchanged
 safari 8   ✔ Passed [ok]: Block date-picker is unchanged
 safari 8   ✔ Passed [ok]: Block document-thumbnail is unchanged
 safari 8   ✔ Passed [ok]: Block document-viewer is unchanged
 safari 8   ✔ Passed [ok]: Block draggable is unchanged
 safari 8   ✔ Passed [ok]: Block dropdown is unchanged
 safari 8   ✔ Passed [ok]: Block droppable is unchanged
 safari 8   ✔ Passed [ok]: Block dynamic-list is unchanged
 safari 8   ✔ Passed [ok]: Block fieldset is unchanged
 safari 8   ✔ Passed [ok]: Block file-upload is unchanged
 safari 8   ✔ Passed [ok]: Block filter is unchanged
 safari 8   ✔ Passed [ok]: Block footer is unchanged
 safari 8   ✔ Passed [ok]: Block form-error is unchanged
 safari 8   ✔ Passed [ok]: Block form-wizard is unchanged
 safari 8   ✔ Passed [ok]: Block header is unchanged
 safari 8   ✔ Passed [ok]: Block horizontal-form is unchanged
 safari 8   ✔ Passed [ok]: Block icon is unchanged
 safari 8   ✔ Passed [ok]: Block input is unchanged
 safari 8   ✔ Passed [ok]: Block label is unchanged
 safari 8   ✔ Passed [ok]: Block main is unchanged
 safari 8   ✔ Passed [ok]: Block modal is unchanged
 safari 8   ✔ Passed [ok]: Block popover is unchanged
 safari 8   ✔ Passed [ok]: Block progress-bar is unchanged
 safari 8   ✔ Passed [ok]: Block radio is unchanged
 safari 8   ✔ Passed [ok]: Block search is unchanged
 safari 8   ✔ Passed [ok]: Block select is unchanged
 safari 8   ✔ Passed [ok]: Block sidebar is unchanged
 safari 8   ✔ Passed [ok]: Block standout-text is unchanged
 safari 8   ✔ Passed [ok]: Block table is unchanged
 safari 8   ✔ Passed [ok]: Block tabs is unchanged
 safari 8   ✔ Passed [ok]: Block textarea is unchanged
 safari 8   ✔ Passed [ok]: Block title is unchanged
 safari 8   ✔ Passed [ok]: Block tooltip is unchanged
 safari 8   ✔ Passed [ok]: Block tree is unchanged
 safari 8   ✔ Passed [ok]: Block vertical-menu is unchanged
 safari 8   OK. 44 assertions passed. (17m 49s / 1069930ms)
 safari 8   
 safari 7.1   [Visual Capture] Test Suite
 safari 7.1   ===========================
 safari 7.1   
 safari 7.1   Results for:  capture
 safari 7.1   ✔ Passed [ok]: Block alert is still conform to its baseline.
 safari 7.1   ✔ Passed [ok]: Block avatar is still conform to its baseline.
 safari 7.1   ✔ Passed [ok]: Block badge is still conform to its baseline.
 safari 7.1   ✔ Passed [ok]: Block button is still conform to its baseline.
 safari 7.1   ✔ Passed [ok]: Block button-group is still conform to its baseline.
 safari 7.1   ✔ Passed [ok]: Block calendar is still conform to its baseline.
 safari 7.1   ✔ Passed [ok]: Block card is still conform to its baseline.
 safari 7.1   ✔ Passed [ok]: Block carousel is still conform to its baseline.
 safari 7.1   ✔ Passed [ok]: Block checkable-block is still conform to its baseline.
 safari 7.1   ✔ Passed [ok]: Block checkbox is still conform to its baseline.
 safari 7.1   ✔ Passed [ok]: Block date-picker is still conform to its baseline.
 safari 7.1   ✔ Passed [ok]: Block document-thumbnail is still conform to its baseline.
 safari 7.1   ✖ Failed [ok]: (Error: Block document-viewer has been altered by 25.98%.)  - expected "true" but got: false
 safari 7.1       at Object.ok (/home/ubuntu/peopledoc-ui/node_modules/nightwatch/lib/core/api.js:67:24)
    at /home/ubuntu/peopledoc-ui/tests/visual-capture.js:118:45
    at process._tickCallback (node.js:368:9)
 safari 7.1   ✔ Passed [ok]: Block draggable is still conform to its baseline.
 safari 7.1   ✔ Passed [ok]: Block dropdown is still conform to its baseline.
 safari 7.1   ✔ Passed [ok]: Block droppable is still conform to its baseline.
 safari 7.1   ✔ Passed [ok]: Block dynamic-list is still conform to its baseline.
 safari 7.1   ✔ Passed [ok]: Block fieldset is still conform to its baseline.
 safari 7.1   ✔ Passed [ok]: Block file-upload is still conform to its baseline.
 safari 7.1   ✔ Passed [ok]: Block filter is still conform to its baseline.
 safari 7.1   ✔ Passed [ok]: Block footer is still conform to its baseline.
 safari 7.1   ✔ Passed [ok]: Block form-error is still conform to its baseline.
 safari 7.1   ✔ Passed [ok]: Block form-wizard is still conform to its baseline.
 safari 7.1   ✔ Passed [ok]: Block header is still conform to its baseline.
 safari 7.1   ✔ Passed [ok]: Block horizontal-form is still conform to its baseline.
 safari 7.1   ✔ Passed [ok]: Block icon is still conform to its baseline.
 safari 7.1   ✔ Passed [ok]: Block input is still conform to its baseline.
 safari 7.1   ✔ Passed [ok]: Block label is still conform to its baseline.
 safari 7.1   ✔ Passed [ok]: Block main is still conform to its baseline.
 safari 7.1   ✔ Passed [ok]: Block modal is still conform to its baseline.
 safari 7.1   ✔ Passed [ok]: Block popover is still conform to its baseline.
 safari 7.1   ✔ Passed [ok]: Block progress-bar is still conform to its baseline.
 safari 7.1   ✔ Passed [ok]: Block radio is still conform to its baseline.
 safari 7.1   ✔ Passed [ok]: Block search is still conform to its baseline.
 safari 7.1   ✔ Passed [ok]: Block select is still conform to its baseline.
 safari 7.1   ✔ Passed [ok]: Block sidebar is still conform to its baseline.
 safari 7.1   ✔ Passed [ok]: Block standout-text is still conform to its baseline.
 safari 7.1   ✔ Passed [ok]: Block table is still conform to its baseline.
 safari 7.1   ✔ Passed [ok]: Block tabs is still conform to its baseline.
 safari 7.1   ✔ Passed [ok]: Block textarea is still conform to its baseline.
 safari 7.1   ✔ Passed [ok]: Block title is still conform to its baseline.
 safari 7.1   ✔ Passed [ok]: Block tooltip is still conform to its baseline.
 safari 7.1   ✔ Passed [ok]: Block tree is still conform to its baseline.
 safari 7.1   ✔ Passed [ok]: Block vertical-menu is still conform to its baseline.
 safari 7.1   FAILED:  1 assertions failed and 43 passed (4m 11s / 251538ms)
 safari 7.1   
 safari 7.1   ----------------------------------------------------
 safari 7.1   TEST FAILURE: 1 assertions failed, 43 passed (4m 12s)
 safari 7.1    ✖ visual-capture
 safari 7.1   - capture
 safari 7.1   Failed [ok]: (Error: Block document-viewer has been altered by 25.98%.) - Expected "true" but got: "false"
 safari 7.1   
 safari 7   [Visual Capture] Test Suite
 safari 7   ===========================
 safari 7   
 safari 7   Results for:  capture
 safari 7   ✔ Passed [ok]: No reference for block alert, skipping
 safari 7   ✔ Passed [ok]: No reference for block avatar, skipping
 safari 7   ✔ Passed [ok]: No reference for block badge, skipping
 safari 7   ✔ Passed [ok]: No reference for block button, skipping
 safari 7   ✔ Passed [ok]: No reference for block button-group, skipping
 safari 7   ✔ Passed [ok]: No reference for block calendar, skipping
 safari 7   ✔ Passed [ok]: No reference for block card, skipping
 safari 7   ✔ Passed [ok]: No reference for block carousel, skipping
 safari 7   ✔ Passed [ok]: No reference for block checkable-block, skipping
 safari 7   ✔ Passed [ok]: No reference for block checkbox, skipping
 safari 7   ✔ Passed [ok]: No reference for block date-picker, skipping
 safari 7   ✔ Passed [ok]: No reference for block document-thumbnail, skipping
 safari 7   ✔ Passed [ok]: No reference for block document-viewer, skipping
 safari 7   ✔ Passed [ok]: No reference for block draggable, skipping
 safari 7   ✔ Passed [ok]: No reference for block dropdown, skipping
 safari 7   ✔ Passed [ok]: No reference for block droppable, skipping
 safari 7   ✔ Passed [ok]: No reference for block dynamic-list, skipping
 safari 7   ✔ Passed [ok]: No reference for block fieldset, skipping
 safari 7   ✔ Passed [ok]: No reference for block file-upload, skipping
 safari 7   ✔ Passed [ok]: No reference for block filter, skipping
 safari 7   ✔ Passed [ok]: No reference for block footer, skipping
 safari 7   ✔ Passed [ok]: No reference for block form-error, skipping
 safari 7   ✔ Passed [ok]: No reference for block form-wizard, skipping
 safari 7   ✔ Passed [ok]: No reference for block header, skipping
 safari 7   ✔ Passed [ok]: No reference for block horizontal-form, skipping
 safari 7   ✔ Passed [ok]: No reference for block icon, skipping
 safari 7   ✔ Passed [ok]: No reference for block input, skipping
 safari 7   ✔ Passed [ok]: No reference for block label, skipping
 safari 7   ✔ Passed [ok]: No reference for block main, skipping
 safari 7   ✔ Passed [ok]: No reference for block modal, skipping
 safari 7   ✔ Passed [ok]: No reference for block popover, skipping
 safari 7   ✔ Passed [ok]: No reference for block progress-bar, skipping
 safari 7   ✔ Passed [ok]: No reference for block radio, skipping
 safari 7   ✔ Passed [ok]: No reference for block search, skipping
 safari 7   ✔ Passed [ok]: No reference for block select, skipping
 safari 7   ✔ Passed [ok]: No reference for block sidebar, skipping
 safari 7   ✔ Passed [ok]: No reference for block standout-text, skipping
 safari 7   ✔ Passed [ok]: No reference for block table, skipping
 safari 7   ✔ Passed [ok]: No reference for block tabs, skipping
 safari 7   ✔ Passed [ok]: No reference for block textarea, skipping
 safari 7   ✔ Passed [ok]: No reference for block title, skipping
 safari 7   ✔ Passed [ok]: No reference for block tooltip, skipping
 safari 7   ✔ Passed [ok]: No reference for block tree, skipping
 safari 7   ✔ Passed [ok]: No reference for block vertical-menu, skipping
 safari 7   OK. 44 assertions passed. (1m 16s / 76822ms)
 safari 7   
Runner failed. See above for details
[Error: Visual regression testng has failed.]
make: *** [visual-capture] Error 1
npm ERR! Test failed.  See above for more details.

npm test returned exit code 1

tr4n2uil commented 8 years ago

A sample demo would be perfect for us to debug better. Can you share one of the test case in the suite to understand what you are testing/asserting that is failing as per above logs ?

xcambar commented 8 years ago

Good news: I solved it! Before closing, a couple of comments, for future reference:

Safari only works with a limited number of ports (see http://stackoverflow.com/questions/28660703/testing-localhost-on-browserstack-safari-8-on-yosemite-cant-open-my-site)
The browsers available on Live are not necessarily available in Automate (Edge 14, Firefox 48, as of this writing)
A very annoying feature: When not specifying an os, Browserstack randomly picks one for you.

The last point is especially annoying when taking screen captures of maximised browsers (we want to do visual regression testing) because Windows and OSX don't maximise windows the same way, which leads to inconsistent results. This also applies to font rendering, FYI.

@tr4n2uil Thanks for your help, because as you were asking for a demo, that allowed me to actually isolate a number of situations that led to this resolution. So, thanks mate!

browserstack / browserstack-local-nodejs

Inconsistent browser behaviour #18