search

nightwatchjs / nightwatch

Integrated end-to-end testing framework written in Node.js and using W3C Webdriver API. Developed at @browserstack
https://nightwatchjs.org
MIT License
11.79k stars 1.31k forks source link

--retries does not run testcases under it #565

Closed LucioFranco closed 9 years ago

LucioFranco commented 9 years ago

It seems that when you use --retries and it reruns a testcase and then that testcase passes it does not continue to run the rest of the testcases in the file. It seems to just skip to the after function. This creates the issue that some of our unit tests are not being reached on some of the testsuites.

sknopf commented 9 years ago

I couldn't reproduce. What does your configuration look like? An example would help too.

LucioFranco commented 9 years ago

Here is the nightwatch.json

{
  "src_folders" : ["tests"],
  "output_folder" : "reports",
  "custom_commands_path" : "commands",
  "custom_assertions_path" : "assertions",
  "page_objects_path" : "",
  "globals_path" : "./globals/globals.js",

  "selenium" : {
    "start_process" : true,
    "server_path" : "bin/selenium-server-standalone-2.45.0.jar",
    "log_path" : "",
    "host" : "127.0.0.1",
    "port" : 4444,
    "cli_args" : {
      "webdriver.chrome.driver" : "bin/chromedriver",
      "webdriver.ie.driver" : ""
    }
  },

  "test_settings" : {
    "default" : {
      "launch_url" : "http://localhost",
      "selenium_port"  : 4444,
      "selenium_host"  : "localhost",
      "silent": true,
      "screenshots" : {
        "enabled" : false,
        "path" : ""
      },
      "desiredCapabilities": {
        "browserName": "firefox",
        "javascriptEnabled": true,
        "acceptSslCerts": true
      }
    },

    "chrome" : {
      "end_session_on_fail": false,
      "desiredCapabilities": {
        "browserName": "chrome",
        "javascriptEnabled": true,
        "acceptSslCerts": true
      }
    },

    "local-chrome": {
      "end_session_on_fail": false,
      "desiredCapabilities": {
        "browserName": "chrome",
        "javascriptEnabled": true,
        "acceptSslCerts": true
      }
    },

    "local-firefox": {
      "end_session_on_fail": false,
      "desiredCapabilities": {
        "browserName": "firefox",
        "javascriptEnabled": true,
        "acceptSslCerts": true
      }
    },

    "staging-chrome": {
      "end_session_on_fail": false,
      "desiredCapabilities": {
        "browserName": "chrome",
        "javascriptEnabled": true,
        "acceptSslCerts": true
      }
    },

    "staging-firefox": {
      "end_session_on_fail": false,
      "desiredCapabilities": {
        "browserName": "firefox",
        "javascriptEnabled": true,
        "acceptSslCerts": true
      }
    }

  }
}

I can not give you the exact test as it has sensitive code but I just checked the logs and it seems that when it retries it just skips the rest of the testcases in the suite. If there is anything else that might help and debug please let me know. Thanks!

sknopf commented 9 years ago

Thanks. Can you try this simple test case and see if you can reproduce with this?

module.exports = {
  testSometimesFails : function(client) {
    client.assert.ok(Math.random() >= 0.5);
  },

  testPass : function(client) {
    client.assert.ok(true);
  }
};

The first test will fail half the time, so you may need to run a few times to get a the scenario in question (it helps to set the # of retries to a high number like 4).

Example output (works as expected):

$ bin/nightwatch --test myTest.js --retries 2

Running:  testSometimesFails
 ✖ Failed [ok]: (false == true)  - expected "true" but got: false

Retrying (1/2):  testSometimesFails
 ✔ Passed [ok]: true ok

OK. 1 assertions passed. (2ms)

Running:  testPass
 ✔ Passed [ok]: true ok

OK. 1 assertions passed. (1ms)

OK. 2 total assertions passed. (1.22s)
LucioFranco commented 9 years ago 
nightwatch --test test.js -e local-chrome --retries 2
Starting selenium server... started - PID:  71770

[Test] Test Suite
=================

Running:  testSometimesFails
 ✖ Failed [ok]: (false == true)  - expected "true" but got: false

Retrying (1/2):  testSometimesFails
 ✖ Failed [ok]: (false == true)  - expected "true" but got: false

Retrying (2/2):  testSometimesFails
 ✔ Passed [ok]: true ok

OK. 1 assertions passed. (1ms)

This is what I get and I updated the global version of nightwtach to 0.7.8

Here is another one:

nightwatch --test test.js -e local-chrome --retries 2
Starting selenium server... started - PID:  71733

[Test] Test Suite
=================

Running:  testSometimesFails
 ✖ Failed [ok]: (false == true)  - expected "true" but got: false

Retrying (1/2):  testSometimesFails
 ✔ Passed [ok]: true ok

OK. 1 assertions passed. (1ms)
sknopf commented 9 years ago

OK I figured out how to reproduce - I'll look into the issue.

LucioFranco commented 9 years ago

Ok thank you!

sknopf commented 9 years ago

@LucioFranco the issue looks to be coming from how you have your configuration setup for unit tests. When I set it up properly, I don't see any issues. Take a look at the example here (unittests env) for using a a combined configuration. I think that will fix it

LucioFranco commented 9 years ago

@sknopf I am still a bit confused. What exactly is wrong with the configuration I have. Do I have to add a filter property to my env? Doesn't quite make sense and again Thanks for the help!

sknopf commented 9 years ago

Yeah, you'll want to add a filter but also the selenium settings - you don't want to start the selenium server.

On Fri, Jul 17, 2015 at 4:57 PM, Lucio Franco notifications@github.com wrote:

@sknopf https://github.com/sknopf I am still a bit confused. What exactly is wrong with the configuration I have. Do I have to add a filter property to my env? Doesn't quite make sense and again Thanks for the help!

— Reply to this email directly or view it on GitHub https://github.com/nightwatchjs/nightwatch/issues/565#issuecomment-122414187 .

LucioFranco commented 9 years ago

So nightwatch can not manage the selenium process if you have the retries argument?

sknopf commented 9 years ago

No it works fine handling the selenium process with retries. The issue here is that you are running unit tests, so you should have start_process set to false and start_process set to false like in the example config. With that proper setup, retries should work fine.

LucioFranco commented 9 years ago

@sknopf so here is my issue we are actually not writing unit test we are actually writing functional tests with nightwatch. So we need nightwatch to start selenium. So setting start_process to false makes the tests fail. and when I run the example you gave me to run i get:

nightwatch --test test.js -e local-chrome --retries 2

[Test] Test Suite
=================

Running:  testSometimesFails
 ✖ Failed [ok]: (false == true)  - expected "true" but got: false

Error retrieving a new session from the selenium server
Error: connect ECONNREFUSED
    at errnoException (net.js:904:11)
    at Object.afterConnect [as oncomplete] (net.js:895:19)

Connection refused! Is selenium server started?

With the env of:

"local-chrome": {
      "selenium" : {
        "start_process" : false
      },
      "end_session_on_fail": false,
      "desiredCapabilities": {
        "browserName": "chrome",
        "javascriptEnabled": true,
        "acceptSslCerts": true
      }
    }
sknopf commented 9 years ago

Yes but that's because you have start_process set to false and start_session set to true there.

So you're not running unit tests and indeed need Selenium. Going back to your original configuration, does it work as expected if you set skip_testcases_on_fail to false?

LucioFranco commented 9 years ago

@sknopf That worked! Is there anyway to specify this for every env or do i have to specify this for every env?

sknopf commented 9 years ago

Excellent. Yes you can set it in the default env.

On Fri, Jul 17, 2015 at 6:11 PM, Lucio Franco notifications@github.com wrote:

@sknopf https://github.com/sknopf That worked! Is there anyway to specify this for every env or do i have to specify this for every env?

— Reply to this email directly or view it on GitHub https://github.com/nightwatchjs/nightwatch/issues/565#issuecomment-122431321 .

LucioFranco commented 9 years ago

@sknopf Okay awesome and thanks for the help!

himanshuteotia commented 8 years ago

This is what I'm using and it works fine

{ "src_folders" : ["tests"], "output_folder" : "/reports", "custom_commands_path" : "./examples/custom-commands", "custom_assertions_path" : "", "globals_path" : "",

"selenium" : { "start_process" : false, "server_path" : "lib/selenium-server-standalone-2.52.0.jar", "log_path" : "", "host" : "127.0.0.1", "port" : 4444, "cli_args" : { "webdriver.chrome.driver" : "lib/chromedriver.exe", "webdriver.ie.driver" : "" }
},

"test_settings" : { "default" : { "launch_url" : "http://localhost", "selenium_port" : 4444, "selenium_host" : "localhost", "silent": true, "screenshots" : { "enabled" : false, "path" : "" }, "desiredCapabilities": { "browserName": "chrome", "javascriptEnabled": true, "acceptSslCerts": true, "platform" : "LINUX" } } } }