search

Shelex / cypress-allure-plugin

cypress plugin to use allure reporter api in tests
https://shelex.github.io/cypress-allure-plugin-example/
Apache License 2.0
156 stars 43 forks source link

"Broken" tests causing wrong numbers in report #187

Closed jprealini closed 1 year ago

jprealini commented 1 year ago

Hi!

I might be wrong here, since I'm new to this plugin/report.

I tried to "mimic" a broken test just to see how that would look in the report (as described in this post, I created the categories.json file so that my tests are correctly categorized in the report)

In order to "mimic" it, I did some weird coding in one of my tests.... I have two specs, 4 tests in each one, and I basically broke Cypress I guess, haha... and this is what I got:

image

Basically, since Cypress wasn't even able to run the spec due the crazy stuff I did, the report shows it as a third spec, and considers it ONE extra test case...

The spec that shows yellow is the one called Main Spec, I broke it, ran it, saved the history, fixed it, ran it again, and made it pass... So now as I said I have 3 specs (when I really have two) and 9 test cases (when I really have 8)

So here go my questions:

  1. Is this how the report should look when a spec breaks the execution?
  2. What would be considered a "broken test" (Test defect)? Can you share a sample to try to replicate it on my end?

Thanks!

Shelex commented 1 year ago
  1. Is this how the report should look when a spec breaks the execution?

Yes

  1. What would be considered a "broken test" (Test defect)? Can you share a sample to try to replicate it on my end?

The answer is actually in article you have shared:

Categories

Categories is one of the most time-saving features of Allure Report. It provides simple automation for fail resolution. There are two categories of defects by default:

  • Product defects (failed tests)
  • Test defects (broken tests)

So if the problem is with your code - crash, uncaught exception, etc. it is most likely that test was not executed as intended, so there is less sense to treat it as fail, rather than "broken". While if assertion is failed, some product under test behaviour was unexpected - we can treat the result as "failed" test.

However, I see little sense in such categorisation, and in this plugin most likely you should mostly have failed status for tests, excepting a couple of exceptional cases, when pre-requisite hooks are failing resulting in skipping tests by cypress or even not go through the spec file.

Examples could be found in: Repo: https://github.com/Shelex/cypress-allure-plugin-example Report: https://shelex.github.io/cypress-allure-plugin-example/

Most likely you have not deleted previous allure-results and have allure-report for both executions. However there should be 2 separate reports as discussed in https://github.com/Shelex/cypress-allure-plugin/issues/185. If cypress crashed - there is a little sense in parsing all the tests to find out what was not executed just to match some numbers.

jprealini commented 1 year ago

@Shelex man, you rock.. thanks!

So, to summarize, (and to see if I got it correctly), you mean that basically if Cypress breaks for whatever reason, that report should not even be created in the first place? or probably it should be created (otherwise you wouldn't know why Cypress crashed) but not be "sent" to the history... right? I'm not an expert on CI jobs, but I guess that when the Jenkins job is configured, I can tell it something like "Ok, if Cypress returns an exception, don't save this report to the history" or something like that...

Shelex commented 1 year ago

if Cypress breaks for whatever reason, that report should not even be created in the first place

No, I don't mean that. We should have report in each case.

otherwise you wouldn't know why Cypress crashed

That is exact reason why we would still need a report, and even if spec cpde was crashed, plugin will catch an event from cypress and will create some sample empty test with name of the spec file, attach a failure reason and artifacts to better investigate what went wrong.

but not be "sent" to the history... right?

This plugin is running INSIDE Cypress browser window, while other plugins can run ASIDE their runners. This means that we have pros and cons, one of the cons is exactly this case - if cypress instance is failed in node context, not even starting the browser (where the plugin will be initialised) - we will not have information about tests that were not executed. Thus, test results numbers for an execution with crashed spec will differ from regular run.

I'm not an expert on CI jobs, but I guess that when the Jenkins job is configured

Me too. Especially in Jenkins.

"Ok, if Cypress returns an exception, don't save this report to the history" or something like that...

No, you will have a report.
Imagine we have spec file with 2 describe's (suites), each consists of 4 it's (tests).

We had a very good night with our friends, we are very exhausted, and accidentally added a typo in import path of some dependency, so instead of import fs from 'fs:node' it become import fs from 'fs:nose'.

Pushed the code, tests are running, but our spec crashed, we will see in report our broken spec with failure reason, like here, but it will not contain all that 8 tests that were previously, because Cypress did not passed that information to plugin, it is not even started the spec.

Ok, you will not see some of your tests in report. Well that is bad, it is understandable, I see. But I think we have an elephant in a room which we should address first - "hello, the spec is crashed, it is broken, go fix it" :)