larrymyers / jasmine-reporters

Reporter classes for the jasmine test framework. Includes JUnitXmlReporter for generating junit xml output for running in CI environments like Jenkins.
MIT License
394 stars 180 forks source link

Multiple failures in testsuite for failures under single testcase #209

Open sayantansls opened 2 years ago

sayantansls commented 2 years ago

Hi all, I am using jasmine-reporters for generating XML reports for protractor E2E tests which are then being used to create HTML reports. There seems to be a bug in the implementation. So, I have a test suite with 3 test cases, of which 2 test cases are success and 1 test case fails. On running the tests with the JUnitXmlReporter, I am getting the following XML node for the particular testsuite.

<testsuite name="Ingestion" timestamp="2022-02-20T02:59:46" hostname="localhost" time="98.866" errors="0" tests="3" skipped="0" disabled="0" failures="3">
  <testcase classname="Ingestion" name="should ingest file type 1" time="25.266"/>
  <testcase classname="Ingestion" name="should ingest file type 2" time="26.258"/>
  <testcase classname="Ingestion" name="should ingest file type 3" time="13.548">
    <failure type="toBeTruthy" message="Expected false to be truthy."><![CDATA[
   ]]></failure>
    <failure type="exception" message="Failed: No element found using locator: By(xpath, //section[@data-test=&quot;sample-classification&quot;]//mat-select//div//span)"><![CDATA[
   ]]></failure>
    <failure type="exception" message="Failed: element click intercepted: Element &lt;button _ngcontent-keo-c276=&quot;&quot; data-test=&quot;close-dialog&quot; mat-icon-button=&quot;&quot; class=&quot;mat-focus-indicator mat-icon-button mat-button-base&quot; style=&quot;margin-bottom: 5px;&quot;&gt;...&lt;/button&gt; is not clickable at point (1366, 102). Other element would receive the click: &lt;section _ngcontent-keo-c276=&quot;&quot; class=&quot;wrapper active ng-star-inserted&quot; style=&quot;&quot;&gt;&lt;/section&gt;
  (Session info: chrome=97.0.4692.71)
  (Driver info: chromedriver=97.0.4692.71 (adefa7837d02a07a604c1e6eff0b3a09422ab88d-refs/branch-heads/4692@{#1247}),platform=Linux 3.10.0-1160.49.1.el7.x86_64 x86_64)"><![CDATA[
   ]]></failure>
  </testcase>
</testsuite>

So, as visible, two of the three tests have succeeded, however, since the failed test case has 3 failure nodes under it, so the testsuite tag shows the failures attribute as 3, which is an incorrect representation, that gets carried on to my HTML report. I think this line 279 in the file junit_reporter.js is causing this bug.

if (isFailed(spec)) { spec._suite._failures += spec.failedExpectations.length; }

Do let me know if this is by design or is an issue that could potentially be fixed, thanks.

putermancer commented 2 years ago

It’s not intentional. I’d need to go re read the spec again, but I suspect failures is supposed to represent the number of specs (it) that failed, not the number of expectations.

There is also a good chance this behavior is similarly flawed in the other reporters. If you make a pull request to address this, please consider each of the reporters in turn.

This project is in heavy maintenance mode, so a pull request is the best option to ensure timely resolution for your use case.

On Sat, Feb 19, 2022 at 12:43 PM Sayantan Ghosh @.***> wrote:

Hi all, I am using jasmine-reporters for generating XML reports for protractor E2E tests which are then being used to create HTML reports. There seems to be a bug in the implementation. So, I have a test suite with 3 test cases, of which 2 test cases are success and 1 test case fails. On running the tests with the JUnitXmlReporter, I am getting the following XML node for the particular testsuite.

So, as visible, two of the three tests have succeeded, however, since the failed test case has 3 failure nodes under it, so the testsuite tag shows the failures attribute as 3, which is an incorrect representation, that gets carried on to my HTML report. I think this line 279 in the file junit_reporter.js is causing this bug.

if (isFailed(spec)) { spec._suite._failures += spec.failedExpectations.length; }

Do let me know if this is by design or is an issue that could potentially be fixed, thanks.

— Reply to this email directly, view it on GitHub https://github.com/larrymyers/jasmine-reporters/issues/209, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACRTNXETKKI2T3CYZNCE6DU37XFTANCNFSM5O3K3AKQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you are subscribed to this thread.Message ID: @.***>