Report only includes the results from the last spec executed and not all specs when running WDIO suites

[tzero86]

Hello,

When running WDIO specific suites for example npm run smokeTestSuite, being smokeTest a suite defined in wdio.config.js as per the docs:

suites: { smokeTestSuite: [ "./tests/specs/App_Login.spec.js", "./tests/specs/App_Role.spec.js", "./tests/specs/App_CreateAccount.spec.js", "./tests/specs/App_DeleteAccount.spec.js" ] }

And of course the same suiteName defined as a script in package.json file to execute the suite of tests configured with WDIO.

"scripts": { "smoke": "wdio --suite smokeTestSuite" }

When executing such script: npm run smoke

All test pass without issues and a report is generated. However, the report created only includes the pass/fail results of the last spec file of the suite executed and not the information from all the specs that were executed in that suite. Just to Clarify only 1 report.html file is generated and contains only the test results from the last report.json generated.

Note: Image above was from one of my projects, hence the amount of folders won't match the amount of spec files detailed in the sample suite above.

In this example, let's say the last Spec to run was the App_DeleteAccount.spec.js then that's the only information I would see when opening the report.html file.

Is there any way this can be enhanced so the report generated takes into account all the report.json files created for each spec file under the suite? that was how the original reporter worked on v4. No matter how much spects your suite had, the single report file would include all the test results in that single report.html file.

Just FYI I've configured the Report using the default settings as in the readme of this report's repository.

I hope the details helps for a possible fix.

Thank you!

[rpii]

Elias,

Webdriverio creates a report for every spec file it executes. The report structure for multiple spec files is normally in the latest report version is:

reports/html-reports/suite-0-0/report.html reports/html-reports/suite-0-1/report.html

what is your structure?

Rich

From: Elias notifications@github.com Sent: Wednesday, April 3, 2019 1:17 PM To: rpii/wdio-html-reporter wdio-html-reporter@noreply.github.com Cc: Subscribed subscribed@noreply.github.com Subject: [EXTERNAL] [rpii/wdio-html-reporter] Report only includes the results from the last spec executed and not all specs when running WDIO suites (#2)

Hello,

When running WDIO specific suites for example npm run smokeTestSuite, being smokeTest a suite defined in wdio.config.js as per the docs:

suites: { smokeTestSuite: [ "./tests/specs/App_Login.spec.js", "./tests/specs/App_Role.spec.js", "./tests/specs/App_CreateAccount.spec.js", "./tests/specs/App_DeleteAccount.spec.js" ] }

And of course the same suiteName defined as a script in package.json file to execute the suite of tests configured with WDIO.

"scripts": { "smoke": "wdio --suite smokeTestSuite" }

When executing such script: npm run smoke

All test pass without issues and a report is generated. However, the report created only includes the pass/fail results of the last spec file of the suite executed and not the information from all the specs that were executed in that suite. Just to Clarify only 1 report.html file is generated and contains only the test results from the last report.json generated.

[Image removed by sender. HTMLReport_Issue]https://user-images.githubusercontent.com/3419877/55509904-ae9e5680-5622-11e9-8328-4dd818903968.jpg Note: Image above was from one of my projects, hence the amount of folders won't match the amount of spec files detailed in the sample suite above.

In this example, let's say the last Spec to run was the App_DeleteAccount.spec.js then that's the only information I would see when opening the report.html file.

Is there any way this can be enhanced so the report generated takes into account all the report.json files created for each spec file under the suite? that was how the original reporter worked on v4. No matter how much spects your suite had, the single report file would include all the test results in that single report.html file.

Just FYI I've configured the Report using the default settings as in the readme of this report's repository.

I hope the details helps for a possible fix.

Thank you!

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/rpii/wdio-html-reporter/issues/2, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AQAUEdomsKCM2uVvAQGx-2_HxZLawTLDks5vdQxDgaJpZM4cbWvi.

[tzero86]

Hey Rich,

Thanks for getting back to me on this. I'm not sure if you saw the screen-capture I've attached to the original post. There you can see the file structure that gets generated. So far I only get one single folder called reports/html-reports/suite-0-0/report.html and when I open that file, it only shows the last 7 tests from one of the spec files(the last one to run).

I' attaching here a new capture of the folder structure and the report.html file generated (renamed to report.txt so GitHub lets me upload it) so you can see it only mentions one spec file. report.txt

I've ran everything from scratch to confirm I'm still seeing the same.

Thanks for your time Rich, appreciated. Elias.

[rpii]

the report.json was going into the wrong folder. Ive fixed that. your report must be blowing up in the template processing. You should see the exception in the console log, please attach that. can you attach a report.json that didnt generate an html? Also, setting the debug options will add info at each reporter event, that may provide information

[tzero86]

Rich, thanks for the update.

I Tested it all again with one of the smallest suites I have. Attached you can find everything that was generated under html-reports folder with the updated version of the reporter. I no longer see any 0-0, 0-1 Folders, just a single "suite 0-0" with both report.json and report.html files in there. As for the logs I ran it with debug set to true, and I got a lot of extra details with that, please also find those logs attached as well. Still the report.html is showing only the last 23 tests from 1 of the spec files (in the log file you can see all other specs that were executed).

html-reports.zip Automation_terminalLogs.txt

Thanks for your time. Elias.

[rpii]

Elias, Thanks for providing such a thorough test case. I was able to reproduce what was happening last night. When you run a suite defined in the wdio.conf.js like you have defined, the test control data mutates in a way not anticipated. When I upgraded the html reporter to 5.0 I sort of reverse engineered from the data structures coming in on the report events. I expected the suite uid to be different each time. Instead the suite uid stays the same and the cid increments. So the reports were overwriting themselves. I’ve fixed it for the moment but I need to spend more time and develop some better tests for the report generator. Please let me know if this works in your scenario.

Rich

[tzero86]

Rich,

Thanks for the updated version. I've tested it again and I think we are getting closer. I do see now multiple folders inside the "suite 0-0" folder. However still see multiple report.json and report.html files being generated without any of them containing a single-full report of all specs that were executed. Each report HTML I get is only including results from one spec, while I was expecting to see a single report.html containing the results from all the specs that were executed as a suite.

I'm attaching the newly gathered results (htmlreports folder and terminal logs). html-reports.zip terminal_logs.txt

As usual, thank you in advance for your time invested in troubleshooting this. And please let me know if you need any more details or testing from my side.

Elias.

[rpii]

Elias,

Unfortunately webdriverio doesn’t invoke the reporter function that way. It executes a complete separate report sequence for each suite in the run. That is the reason for so many folders. Not sure if there is easy way to aggregate them all after the run, I haven’t seen any easy way to do this, but it could just be processing all the report.json files to build an aggregate report, then running the html reporter on that..

Rich

From: Elias notifications@github.com Sent: Friday, April 5, 2019 8:55 AM To: rpii/wdio-html-reporter wdio-html-reporter@noreply.github.com Cc: Peters, Richard rpeter@sandia.gov; Comment comment@noreply.github.com Subject: [EXTERNAL] Re: [rpii/wdio-html-reporter] Report only includes the results from the last spec executed and not all specs when running WDIO suites (#2)

Rich,

Thanks for the updated version. I've tested it again and I think we are getting closer. I do see now multiple folders inside the "suite 0-0" folder. However still see multiple report.json and report.html files being generated without any of them containing a single-full report of all specs that were executed. Each report HTML I get is only including results from one spec, while I was expecting to see a single report.html containing the results from all the specs that were executed as a suite.

I'm attaching the newly gathered results (htmlreports folder and terminal logs). html-reports.ziphttps://github.com/rpii/wdio-html-reporter/files/3048705/html-reports.zip terminal_logs.txthttps://github.com/rpii/wdio-html-reporter/files/3048710/terminal_logs.txt

As usual, thank you in advance for your time invested in troubleshooting this. And please let me know if you need any more details or testing from my side.

Elias.

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/rpii/wdio-html-reporter/issues/2#issuecomment-480328241, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AQAUEaW_nLqfI8uGJCZ7h2o_m1_3k__sks5vd3HqgaJpZM4cbWvi.

[tzero86]

Hey Rich sorry for the delay.

Oh I see, I understand there is a limitation that prevents the reporter to work in that way. I thought It would be possible to make it work like with WDIO v4, I will have to look into other reporters to fulfill what I need.

Anyways, thank you so much for the time you took reviewing this and let me know if there is anything I can to help you.

Thank you! Elias.

[rpii]

Actually I have looked at the code of the other reporters to see what they were doing. The all have the same limitation. The only exception might the the allure report which seems to do extensive processing after all runs are complete. I probably need to look at that. I have code that will build a composite report from all the runs. I just havent found a clean way to do it.

[tzero86]

I see, I've also looked into Allure but for my needs it's not what I'm looking for. I just need a simple HTML report that I can use to automatically email after each run and on WDIO V4 the HTML Reporter was my choice. I hope you can find a clean way to do that with your report and I would definitely love to test it a bit.

Thanks a lot Rich,

Elias.

[rpii]

Elias,

I looked at allure and it does all its real work in a post-processor that picks up files that were written by the reporter. I am looking at doing this but I have not had time yet.

Rich

From: Elias notifications@github.com Sent: Wednesday, April 10, 2019 8:52 AM To: rpii/wdio-html-reporter wdio-html-reporter@noreply.github.com Cc: Peters, Richard rpeter@sandia.gov; Comment comment@noreply.github.com Subject: [EXTERNAL] Re: [rpii/wdio-html-reporter] Report only includes the results from the last spec executed and not all specs when running WDIO suites (#2)

I see, I've also looked into Allure but for my needs it's not what I'm looking for. I just need a simple HTML report that I can use to automatically email after each run and on WDIO V4 the HTML Reporter was my choice. I hope you can find a clean way to do that with your report and I would definitely love to test it a bit.

Thanks a lot Rich,

Elias.

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/rpii/wdio-html-reporter/issues/2#issuecomment-481749083, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AQAUEf52uvZH7KFpndTB1CBFGH0En9-8ks5vfgiygaJpZM4cbWvi.

[rpii]

I was able to make a mod to scan the report files after the run and generate a master report. Please try it out and let me know.

[tzero86]

Hi Rich,

Thanks for the updates. I've updated my project to test this new version. I'm getting this error when trying to Run any test: "ReferenceError: ReportAggregator is not defined" Any ideas?

` C:\ElevateUC\ElevateUCAutomation\Elevate_QC\node_modules\@wdio\cli\build\run.js:94 throw e; ^

ReferenceError: ReportAggregator is not defined at Object.onPrepare (C:\ElevateUC\ElevateUCAutomation\Elevate_QC\wdio.conf.js:197:28) at Launcher.run (C:\ElevateUC\ElevateUCAutomation\Elevate_QC\node_modules\@wdio\cli\build\launcher.js:81:18) at process._tickCallback (internal/process/next_tick.js:68:7) at Function.Module.runMain (internal/modules/cjs/loader.js:745:11) at startup (internal/bootstrap/node.js:236:19) at bootstrapNodeJSCore (internal/bootstrap/node.js:560:3) npm ERR! code ELIFECYCLE npm ERR! errno 1 npm ERR! elevateuc-automation@1.0.0 combo: wdio --suite cleanup && wdio --suite smoke npm ERR! Exit status 1 npm ERR! npm ERR! Failed at the elevateuc-automation@1.0.0 combo script. npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in: npm ERR! C:\Users\elmedina\AppData\Roaming\npm-cache_logs\2019-04-12T15_35_59_743Z-debug.log `

I've updated the reporter version, and added it to the reporters section in wdio.conf.js, plus the ReportAggregators handlers on the OnPrepare and OnComplete hooks. As documented here https://www.npmjs.com/package/@rpii/wdio-html-reporter

Let me know if I'm doing something wrong. Thanks, Elias.

[rpii]

that happens when babel doesnt build the js properly . you should be running the js out of the html/reporter build folder. Thats the default, and thats the babel compiled version. make sure you are compiling your wdio.conf.js with babel. if you are using pure js you might have to do it like this:

var wdioHtmlReporter = require("@rpii/wdio-html-reporter");

    let reportAggregator = new wdioHtmlReporter.ReportAggregator({
      outputDir: './reports/html-reports/',
      filename: 'master-report.html',
      reportTitle: 'Master Report',
      showInBrowser: true
    });
    reportAggregator.clean();
    global.reportAggregator = reportAggregator;

[tzero86]

Rich, I do have Babel setup for WDIO tests as per the docs here (I' using Mocha so I did the register as suggested in the docs) https://webdriver.io/docs/babel.html

Adding the require did solve the reportAggregator error, but now I'm getting another one now. This is somewhat familiar, I think I've seen this error before with other reporters that were not compatible with WDIO V5:

` Stderr: [0-0] 2019-04-12T18:08:47.280Z ERROR @wdio/local-runner: Failed launching test session: TypeError: ReporterClass is not a constructor at BaseReporter.initReporter (C:\ElevateUC\ElevateUCAutomation\Elevate_QC\node_modules\@wdio\runner\build\reporter.js:193:14) at Array.map () at new BaseReporter (C:\ElevateUC\ElevateUCAutomation\Elevate_QC\node_modules\@wdio\runner\build\reporter.js:34:39) at Runner.run (C:\ElevateUC\ElevateUCAutomation\Elevate_QC\node_modules\@wdio\runner\build\index.js:89:21) at process.on.m (C:\ElevateUC\ElevateUCAutomation\Elevate_QC\node_modules\@wdio\local-runner\build\run.js:37:20) at process.emit (events.js:187:15) at emit (internal/child_process.js:811:12) at process._tickCallback (internal/process/next_tick.js:63:19)

`

As always, thanks a lot for your time. Elias.

[rpii]

my reporters section looks like this:

import { ReportAggregator, HtmlReporter} from '@rpii/wdio-html-reporter' ;
    reporters: [
        'spec',
        // ['junit', {
        //     outputDir: './reports/junit-reports/',
        //     outputFileFormat: function(opts) { // optional
        //         return `results-${opts.cid}.${opts.capabilities}.xml`
        //     }
        // }
        // ],
        [HtmlReporter, {
            debug: false,
            outputDir: './reports/html-reports/',
            filename: 'report.html',
            reportTitle: 'Web Test Report',
            showInBrowser: true,
            useOnAfterCommandForScreenshot: false

        }
        ]
    ],

[tzero86]

Hello Rich,

I was able to solve the issues and the report is running correctly. I see the master report being generated, thank you!. A couple of observations:

• Report Generated uses "Master Report" as Title, is there a way to Configure that Value?
• The Duration value in the summary section seems to be not correct: Got Duration | 00:00:16.33 and my whole suite (9 specs) runs in 91.75 seconds, I think this is not reflecting the sum of duration from all specs.
• End and Start Date, should probably need to also consider first spec executed and last one executed to provide an accurate value.

I'm attaching the master report generated along with everything else in the reports folder, let me know if my observations make sense.

As usual, Thank you for your time Rich. html-reports.zip

[rpii]

Elias, you can set the title via the property : reportTitle: "Test Master Report', in the reportAggregator constructor.

No doubt the time parameters are incorrect, They will need to be computed from the data.

Rich

[tzero86]

Great, thanks Rich I will continue to test the report and let you know if I see any other issues.

Thanks, Elias.

[tzero86]

Rich,

It seems having the showInBrowser flag set to false is causing the runs to fail. If I have that set to true, each spec I run as part of a suite will trigger this and open each report in the browser. However, setting it to false seems to be the cause for the runs to fail.

• Same Suite with showInBrowser = True: All tests run correctly.
• Same Suite with showInBrowser = False: The execution just fails without any error.
• I've also tried, the same suite and removing the showInBrowser config line: The execution still fails without any error.

reporters: [ "spec", [ wdioHtmlReporter.HtmlReporter, { debug: false, outputDir: "./reports/html-reports/", filename: "report.html", reportTitle: "Elevate UC Automation Results", showInBrowser: true } ] ], One interesting thing to note is that, when I do for example npm run combo (combo is a script name in my package.json file) Which executes more than one script, like this:

"combo": "wdio --suite cleanup && wdio --suite smoke",

If the showInBrowser is set to true, it'll run correctly both cleanup and the smoke suites. On the other hand having the showInBrowser set to false (or removed completely), causes the command to fail after cleanup has been executed and also the total results reported by WDIO are affected (Notice below, 1 run 1 pass, but still the total shows 1 failed).

So when that fails, all the logs I get are these:

` "spec" Reporter:

[chrome #0-0] Spec: C:\ElevateUC\ElevateUCAutomation\Elevate_QC\tests\specs\ElevateUC_Cleanup.spec.js [chrome #0-0] Running: chrome [chrome #0-0] [chrome #0-0] ElevateUC: Cleaning... [chrome #0-0] ✓ Access Cohort Pool Mgmt and delete any pre-existing auto-cohorts. [chrome #0-0] [chrome #0-0] 1 passing (14.7s)

Test Suites: 0 passed, 1 failed, 1 total (100% completed) Time: � 22.93s

npm ERR! code ELIFECYCLE npm ERR! errno 1 npm ERR! elevateuc-automation@1.0.0 combo: wdio --suite cleanup && wdio --suite smoke npm ERR! Exit status 1 npm ERR! npm ERR! Failed at the elevateuc-automation@1.0.0 combo script. npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in: npm ERR! C:\Users\elmedina\AppData\Roaming\npm-cache_logs\2019-04-16T20_15_14_601Z-debug.log `

Do you see the same issue at your end? let me know if you need any more details from me.

Thanks, Elias.

[rpii]

It looks like I overlooked a callback in the non-showBrowser case, which could have had a bad impact. Dont really have a unit test for it since it is integration behaviour with webdriverio. If you can try it this should work better and compute the proper times as well

[tzero86]

Thanks for the Fixes Rich, I can confirm both are resolved using version 0.7.2. showInBrowser set to false no longer breaks the execution and the computed time in the master report is correct.

I'll let you know if I see any other issues.

Thank you, Elias.

[rpii]

great, Ill close this issue then