search

NickTomlin / protractor-flake

Rerun potentially flakey protractor tests before failing.
MIT License
80 stars 51 forks source link

Protractor-flake running again the tests that have passed #54

Closed rafaelcs closed 7 years ago

rafaelcs commented 7 years ago

How can I configure to run only those tests that failed? Currentlyits running even the tests that passed the first time

wswebcreation commented 7 years ago

Hi @rafaelcs ,

What do you exactly mean? Does it run alle the spec / step files again, including the spec / step-files that hold no failed tests?

protractor-flake can only run a spec / stepfile which holds at least 1 failed test. It can not specifically run the failed test.This due to the log parser that will check which test from which file failed, it will then collect all the files with failed specs and run them again, leaving out the spec / step-files that don't hold any failed tests.

rafaelcs commented 7 years ago

I mean that, for example, I have spec1.js and spec2.js...if only the spec1.js is failing, why spec2.js run again too?

Its my problem

Em 8 de abr de 2017 4:20 AM, "wswebcreation" notifications@github.com escreveu:

Hi @rafaelcs https://github.com/rafaelcs ,

What do you exactly mean? Does it run alle the spec / step files again, including the spec / step-files that hold no failed tests?

protractor-flake can only run a spec / stepfile which holds at least 1 failed test. It can not specifically run the failed test.This due to the log parser that will check which test from which file failed, it will then collect all the files with failed specs and run them again, leaving out the spec / step-files that don't hold any failed tests.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/NickTomlin/protractor-flake/issues/54#issuecomment-292700770, or mute the thread https://github.com/notifications/unsubscribe-auth/AHvNk7hgQHZ_DIiIHxtaWLKr4jAPdaeJks5rtzVBgaJpZM4M3ktI .

wswebcreation commented 7 years ago

Can you provide us the following info:

Tnx in advance

rafaelcs commented 7 years ago

Sorry for delay @wswebcreation

My flake file:

#!/usr/bin/env node

var protractorFlake = require('protractor-flake');
// skip first two passed args (node and self)
var protractorArgs = process.argv.splice(2);

protractorFlake({
    protractorPath: 'node_modules/.bin/protractor',
    maxAttempts: 2,
    parser: 'standard',
    nodeBin: 'node',
    protractorArgs: protractorArgs
}, function(status, output) {
    process.exit(status);
});

My conf.js


exports.config = {

    directConnect: true,

    specs: ['tests/UI/specs/*.js'],
    seleniumAddress: 'http://localhost:4445/wd/hub',
    framework: 'jasmine2',

    onPrepare: function(){
        // set implicit wait times in ms...
        browser.manage().timeouts().implicitlyWait(5000);

        browser.ignoreSynchronization = true;

        // set browser size...
        browser.manage().window().maximize();

        // better jasmine 2 reports...
        let SpecReporter = require('jasmine-spec-reporter').SpecReporter;
        jasmine.getEnv().addReporter(new SpecReporter({displayStacktrace: 'spec'}));
    },

    capabilities: {
        browserName: 'chrome',
        restartBrowserBetweenTests: true
    },

    jasmineNodeOpts: {
        showColors: true,
        displayStacktrace: true,
        displaySpecDuration: true,
        // overrides jasmine's print method to report dot syntax for custom reports
        print: function () {},
        defaultTimeoutInterval: 999999
    }
};

On this screen, you can see that only two tests failed

image

But when the tests ran again, all the specs were executed instead of just running the specs that failed

image

I'm running the tests using ./flake conf.js

That's my problem so far

wswebcreation commented 7 years ago

Hi @rafaelcs,

Is this the only trace you have from your tests, because I think the specreporter is the problem here. What protractor-flake does is that it:

  1. checks the exitcode after all tests are ran
  2. reads the log and parses it.

When I look at you log there is no logging that matches this regex of the standard parser.

I think there are 3 options for you:

  1. write a custom parser to use and parse your output
  2. change the logging of the specreporter so it will output more logging that will match the parser
  3. write a "hook" that will print the filename after a test has failed so it will be picked up by the standard / custom parser

Hope it helps

Grtz Wim

surendraJ commented 7 years ago

Recently I am also getting this error.

98 specs, 1 failure Finished in 446.454 seconds

[21:37:14] I/launcher - 0 instance(s) of WebDriver still running [21:37:14] I/launcher - chrome #01 failed 1 test(s) [21:37:14] I/launcher - overall: 1 failed spec(s) Closing report [21:37:14] E/launcher - Process exited with error code 1

Using standard to parse output Re-running tests: test attempt 2

Tests failed but no specs were found. All specs will be run again.

This is very frustrating as my all specs are running again even there is only one spec failure. Here is my configuration....
===gulpfile.js ===

gulp.task('e2e', function(){
    return protractorFlake({
        protractorPath: 'node_modules/protractor/bin/protractor',
        maxAttempts: 3,
        parser: 'standard',
        // expects node to be in path
        // set this to wherever the node bin is located
        nodeBin: 'node',
        // set color to one of the colors available at 'chalk' - https://github.com/chalk/ansi-styles#colors
        color: 'magenta',
        protractorArgs: ["e2e/conf/protractor.conf.js"]
    }, function (status, output) {
        process.exit(status);
    });
});

==== package.json=====

"jasmine-core": "^2.6.0",
"protractor": "^5.1.1",
protractor-jasmine2-screenshot-reporter": "^0.3.5",    
"protractor-flake": "2.5.1"

===protractor.conf.js=====

var HtmlScreenshotReporter = require('protractor-jasmine2-screenshot-reporter');
var htmlReporter = new HtmlScreenshotReporter({
        dest: 'e2e/testreports', 
        filename: 'e2eReport.html', 
        reportOnlyFailedSpecs: false,
        captureOnlyFailedSpecs: true, 
        showSummary: true,
        showQuickLinks: true
});
exports.config = {
     .......
    specs: ['../spec/*.js'],
    framework:'jasmine2',
    multiCapabilities: [
        {browserName: 'chrome'}
    ],
    directConnect: true,

    onPrepare: function() {
       .......
        jasmine.getEnv().addReporter(htmlReporter);
        var jasmineReporters = require('jasmine-reporters');
        jasmine.getEnv().addReporter(new jasmineReporters.JUnitXmlReporter({
            consolidateAll: true,
            savePath: 'e2e/testreports',
            filePrefix: 'e2eReport'
        }));
    },
..........

And I will run gulp task to call protractor flake. Would you please check this one what is happening?

wswebcreation commented 7 years ago

Hi @surendraJ

Can you please provide your logging so we can see what protractor-flake should parse?

Tnx in advance

surendraJ commented 7 years ago

@wswebcreation How can I generate the log? Is there any configuration do I need to fix to see the log?

wswebcreation commented 7 years ago

@surendraJ It's you console log / output from your protractor tests

surendraJ commented 7 years ago

@wswebcreation here is my log I am not sure it would be helpful or not...

Stack:
    Error: Failed expectation
        at C:\Jenkins\workspace\mypro\e2e\spec\landing-page.spec.js:39:31
        at ManagedPromise.invokeCallback_ (C:\Jenkins\workspace\mypro\node_modules\selenium-webdriver\lib\promise.js:1366:14)
        at TaskQueue.execute_ (C:\Jenkins\workspace\mypro\node_modules\selenium-webdriver\lib\promise.js:2970:14)
        at TaskQueue.executeNext_ (C:\Jenkins\workspace\mypro\node_modules\selenium-webdriver\lib\promise.js:2953:27)
        at asyncRun (C:\Jenkins\workspace\mypro\node_modules\selenium-webdriver\lib\promise.js:2813:27)
        at C:\Jenkins\workspace\mypro\node_modules\selenium-webdriver\lib\promise.js:676:7
        at process._tickCallback (internal/process/next_tick.js:103:7)

98 specs, 1 failure
Finished in 446.454 seconds

[21:37:14] I/launcher - 0 instance(s) of WebDriver still running
[21:37:14] I/launcher - chrome #01 failed 1 test(s)
[21:37:14] I/launcher - overall: 1 failed spec(s)
Closing report
[21:37:14] E/launcher - Process exited with error code 1

Using standard to parse output Re-running tests: test attempt 2

Tests failed but no specs were found. All specs will be run again.

wswebcreation commented 7 years ago

@surendraJ Is this the full log?

We need to the full log because, as also mentioned above, protractor-flake reads the output from protractor to determine which steps-files failed. Based on your logging I can check what goes wrong, so it would be nice if you can provide the complete output from your tests.

surendraJ commented 7 years ago

@wswebcreation It's very huge log file so all are the success log before this one and failure message and again start 2nd attempt...

Failures:
1) #Landing page - [landing-page.spec.js]  should show autocomplete with first entry of ZIP, 52 when entered Zip in location field
  Message:
[31m    Expected [ 'ZIP, 51', 'ZIP, 52', 'ZIP, 53', 'ZIP, 54', 'ZIP, 55', 'ZIP, 56' ] to contain 'ZIP, 61'.
  Stack:
    Error: Failed expectation
        at C:\Jenkins\workspace\mypro\e2e\spec\landing-page.spec.js:38:31
        at ManagedPromise.invokeCallback_ (C:\Jenkins\workspace\mypro\node_modules\selenium-webdriver\lib\promise.js:1366:14)
        at TaskQueue.execute_ (C:\Jenkins\workspace\mypro\node_modules\selenium-webdriver\lib\promise.js:2970:14)
        at TaskQueue.executeNext_ (C:\Jenkins\workspace\mypro\node_modules\selenium-webdriver\lib\promise.js:2953:27)
        at asyncRun (C:\Jenkins\workspace\mypro\node_modules\selenium-webdriver\lib\promise.js:2813:27)
        at C:\Jenkins\workspace\mypro\node_modules\selenium-webdriver\lib\promise.js:676:7
        at process._tickCallback (internal/process/next_tick.js:103:7)
  Message:
    Expected [ 'ZIP, 51', 'ZIP, 52', 'ZIP, 53', 'ZIP, 54', 'ZIP, 55', 'ZIP, 56' ] to contain 'ZIP, 61'.
  Stack:
    Error: Failed expectation
        at C:\Jenkins\workspace\mypro\e2e\spec\landing-page.spec.js:39:31
        at ManagedPromise.invokeCallback_ (C:\Jenkins\workspace\mypro\node_modules\selenium-webdriver\lib\promise.js:1366:14)
        at TaskQueue.execute_ (C:\Jenkins\workspace\mypro\node_modules\selenium-webdriver\lib\promise.js:2970:14)
        at TaskQueue.executeNext_ (C:\Jenkins\workspace\mypro\node_modules\selenium-webdriver\lib\promise.js:2953:27)
        at asyncRun (C:\Jenkins\workspace\mypro\node_modules\selenium-webdriver\lib\promise.js:2813:27)
        at C:\Jenkins\workspace\mypro\node_modules\selenium-webdriver\lib\promise.js:676:7
        at process._tickCallback (internal/process/next_tick.js:103:7)

98 specs, 1 failure
Finished in 446.454 seconds

[21:37:14] I/launcher - 0 instance(s) of WebDriver still running
[21:37:14] I/launcher - chrome #01 failed 1 test(s)
[21:37:14] I/launcher - overall: 1 failed spec(s)
Closing report
[21:37:14] E/launcher - Process exited with error code 1

Using standard to parse output
Re-running tests: test attempt 2

Tests failed but no specs were found. All specs will be run again.

Report destination:   e2e\testreports\e2eReport.html
[21:37:14] W/driverProviders - Using driver provider directConnect, but also found extra driver provider parameter(s): seleniumServerJar
[21:37:14] I/launcher - Running 1 instances of WebDriver
[21:37:14] I/direct - Using ChromeDriver directly...
wswebcreation commented 7 years ago

If this is the only logging for the failure then the standard / multi parser will not detect any failures in your logging. This means it will rerun all your specs again.

My assumption is that the jasmine-reporter is "cleaning" up the log the shown in the output. In my opinion this is the same problem @rafaelcs has and can only be fixed with a "customer" parser.

@NickTomlin , how do you think about this?

wpalkuri commented 7 years ago

Hi @wswebcreation - @NickTomlin - fyi Please see my total log as below. You should see out of 3 tests, 2 tests have passed and 1 test failed. Expecting protractor-flake to re run ONLY the failed test in 2nd attempt. However, it runs all 3 tests again. Can you please suggest a solution or let me know if i missed something.

Thanks in advance.

C:\Users\wilson.palkuri\Downloads\protractor-example-master\protractor-example-master>protractor-flake --maxAttempts=2 -- conf.js
[14:53:34] I/launcher - Running 1 instances of WebDriver
[14:53:45] I/testLogger -
------------------------------------

[14:53:45] I/testLogger - [chrome #01-0] PID: 11920
[chrome #01-0] Specs: C:\Users\wilson.palkuri\Downloads\protractor-example-master\protractor-example-master\specs\friendSpec.js
[chrome #01-0]
[chrome #01-0] [14:53:38] I/direct - Using ChromeDriver directly...
[chrome #01-0] [14:53:45] I/runnerCli - Cannot find module 'chance'

[14:53:45] I/testLogger -

[14:53:45] E/launcher - Runner process exited unexpectedly with error code: 1
[14:53:46] I/launcher - 1 instance(s) of WebDriver still running
.[14:53:56] I/testLogger -
------------------------------------

[14:53:56] I/testLogger - [chrome #01-1] PID: 10200
[chrome #01-1] Specs: C:\Users\wilson.palkuri\Downloads\protractor-example-master\protractor-example-master\specs\nonAngularLoginSpec.js
[chrome #01-1]
[chrome #01-1] [14:53:49] I/direct - Using ChromeDriver directly...
[chrome #01-1] Jasmine started
[chrome #01-1] No Angular PASS
[chrome #01-1]
[chrome #01-1]   non-angular login
[chrome #01-1]     √ should goto friend pages on successful login
[chrome #01-1]
[chrome #01-1] Executed 1 of 1 spec SUCCESS in 1 sec.

[14:53:56] I/testLogger -

[14:53:56] I/launcher - 1 instance(s) of WebDriver still running
..[14:54:22] I/testLogger -
------------------------------------

[14:54:22] I/testLogger - [chrome #01-2] PID: 152
[chrome #01-2] Specs: C:\Users\wilson.palkuri\Downloads\protractor-example-master\protractor-example-master\specs\qsSpec.js
[chrome #01-2]
[chrome #01-2] [14:54:00] I/direct - Using ChromeDriver directly...
[chrome #01-2] Jasmine started
[chrome #01-2]
[chrome #01-2]   Quality Shepherd blog
[chrome #01-2]     √ should display 5 posts per page
[chrome #01-2]     √ should return search results
[chrome #01-2]
[chrome #01-2] Executed 2 of 2 specs SUCCESS in 17 secs.

[14:54:22] I/testLogger -

[14:54:22] I/launcher - 0 instance(s) of WebDriver still running
[14:54:22] I/launcher - chrome #01-0 failed with exit code: 1
[14:54:22] I/launcher - chrome #01-1 passed
[14:54:22] I/launcher - chrome #01-2 passed
[14:54:22] I/launcher - overall: 1 process(es) failed to complete
[14:54:22] E/launcher - Process exited with error code 100

Using standard to parse output
Re-running tests: test attempt 2

Tests failed but no specs were found. All specs will be run again.

[14:54:27] I/launcher - Running 1 instances of WebDriver
[14:54:38] I/testLogger -
------------------------------------

[14:54:38] I/testLogger - [chrome #01-0] PID: 8196
[chrome #01-0] Specs: C:\Users\wilson.palkuri\Downloads\protractor-example-master\protractor-example-master\specs\friendSpec.js
[chrome #01-0]
[chrome #01-0] [14:54:31] I/direct - Using ChromeDriver directly...
[chrome #01-0] [14:54:38] I/runnerCli - Cannot find module 'chance'

[14:54:38] I/testLogger -

[14:54:38] E/launcher - Runner process exited unexpectedly with error code: 1
[14:54:38] I/launcher - 1 instance(s) of WebDriver still running
.[14:54:49] I/testLogger -
------------------------------------

[14:54:49] I/testLogger - [chrome #01-1] PID: 13060
[chrome #01-1] Specs: C:\Users\wilson.palkuri\Downloads\protractor-example-master\protractor-example-master\specs\nonAngularLoginSpec.js
[chrome #01-1]
[chrome #01-1] [14:54:42] I/direct - Using ChromeDriver directly...
[chrome #01-1] Jasmine started
[chrome #01-1] No Angular PASS
[chrome #01-1]
[chrome #01-1]   non-angular login
[chrome #01-1]     √ should goto friend pages on successful login
[chrome #01-1]
[chrome #01-1] Executed 1 of 1 spec SUCCESS in 1 sec.

[14:54:49] I/testLogger -

[14:54:50] I/launcher - 1 instance(s) of WebDriver still running
..[14:55:18] I/testLogger -
------------------------------------

[14:55:18] I/testLogger - [chrome #01-2] PID: 9324
[chrome #01-2] Specs: C:\Users\wilson.palkuri\Downloads\protractor-example-master\protractor-example-master\specs\qsSpec.js
[chrome #01-2]
[chrome #01-2] [14:54:54] I/direct - Using ChromeDriver directly...
[chrome #01-2] Jasmine started
[chrome #01-2]
[chrome #01-2]   Quality Shepherd blog
[chrome #01-2]     √ should display 5 posts per page
[chrome #01-2]     √ should return search results
[chrome #01-2]
[chrome #01-2] Executed 2 of 2 specs SUCCESS in 18 secs.

[14:55:18] I/testLogger -

[14:55:18] I/launcher - 0 instance(s) of WebDriver still running
[14:55:18] I/launcher - chrome #01-0 failed with exit code: 1
[14:55:18] I/launcher - chrome #01-1 passed
[14:55:18] I/launcher - chrome #01-2 passed
[14:55:18] I/launcher - overall: 1 process(es) failed to complete
[14:55:18] E/launcher - Process exited with error code 100

C:\Users\wilson.palkuri\Downloads\protractor-example-master\protractor-example-master>
wswebcreation commented 7 years ago

Hi @wpalkuri

Can you provide more info about the parser you use and the config you use?

When I look at this part of your log

[14:53:45] I/testLogger - [chrome #01-0] PID: 11920
[chrome #01-0] Specs: C:\Users\wilson.palkuri\Downloads\protractor-example-master\protractor-example-master\specs\friendSpec.js
[chrome #01-0]
[chrome #01-0] [14:53:38] I/direct - Using ChromeDriver directly...
[chrome #01-0] [14:53:45] I/runnerCli - Cannot find module 'chance'

[14:53:45] I/testLogger -

[14:53:45] E/launcher - Runner process exited unexpectedly with error code: 1
[14:53:46] I/launcher - 1 instance(s) of WebDriver still running
.[14:53:56] I/testLogger -

I see a problem in your code you don't want to rerun, but just fix ;-)

The reason why the parser (don't exactly know which one you use) is that no exception is thrown that could be picked up by the parser to determine which spec file to rerun. Are you using SpecReporter or an other tool to do the reporting? If so, try enabling the stacktrace option