[💡 Feature]: Wdio spec reporter should exclude hook failures in test case pass fail count

bhuvi4login commented 3 months ago

Is your feature request related to a problem?

Yes, I am using "@wdio/junit-reporter": "^8.32.4", and getting report with beforeAll Hook and afterAll Hook failed from the spec-reporter report like below. In case of success, it excludes the hooks in spec reporter. So I get 4 passed. In case of failure, it includes the hooks in spec reporter. So I get 6 failed. In success/failure, the total count should not be changed for me. This is shown in Dashboard in graphs making difficult to understand by business team, why it shows count less one day (in case of success) and other day count more (in case of failure). [edge-headless-shell 122.0.2365.52 windows #0-0] Spot - A and P [edge-headless-shell 122.0.2365.52 windows #0-0] Verify whether the user is able to search with a ID in Spot screen [edge-headless-shell 122.0.2365.52 windows #0-0] ✖ Validate A and P tab is Highlighted [edge-headless-shell 122.0.2365.52 windows #0-0] ✖ Validate RMC Menu Details [edge-headless-shell 122.0.2365.52 windows #0-0] ✖ Validate Export RMC Menu [edge-headless-shell 122.0.2365.52 windows #0-0] ✖ "after all" hook [edge-headless-shell 122.0.2365.52 windows #0-0] [edge-headless-shell 122.0.2365.52 windows #0-0] Verify No records text in A and P Tab [edge-headless-shell 122.0.2365.52 windows #0-0] ✖ "before all" hook [edge-headless-shell 122.0.2365.52 windows #0-0] ✖ Validate No Records [edge-headless-shell 122.0.2365.52 windows #0-0] [edge-headless-shell 122.0.2365.52 windows #0-0] 6 failing (27.9s)

Describe the solution you'd like.

Exclude the hooks count in total test cases passed or failed. Hooks are meant to setup initial or final state of the application/test suite. Reference to Discord channel: https://discord.com/channels/1097401827202445382/1097674067601010709/1218898644284080208

Describe alternatives you've considered.

Not sure, as Junit or HTML reports generated on the basis of default spec reporter generated, in the spec reporter itself this issue is seen.

Additional context

No response

Code of Conduct

[X] I agree to follow this project's Code of Conduct

wdio-bot commented 3 months ago

Thanks for raising this issue 🙏

Unfortunately we can't help you without a reproducible example in this matter. Please read our contributing guidelines on how to create a reproducible example. If you can't provide a reproducible example we will close this issue in 7 days.

If you have any questions please reach out to us on our Discord channel. We are happy to help you out there.

erwinheitzman commented 3 months ago

I did a quick check and what we do is we put the failed hook in the list of failed specs. It would require a rewrite of this logic, any help on this is welcome and appreciated.

wdio-bot commented 3 months ago

Thanks for reporting!

We greatly appreciate any contributions that help resolve the bug. While we understand that active contributors have their own priorities, we kindly request your assistance if you rely on this bug being fixed. We encourage you to take a look at our contribution guidelines or join our friendly Discord development server, where you can ask any questions you may have. Thank you for your support, and cheers!

christian-bromann commented 3 months ago

@erwinheitzman can you provide a link to the locations you looked into?

erwinheitzman commented 3 months ago

@christian-bromann of course!

https://github.com/webdriverio/webdriverio/blob/main/packages/wdio-junit-reporter/src/index.ts#L59

christian-bromann commented 3 months ago

The above reporter shows a spec report 🤔 do we see the same behavior across multiple reporters here?

webdriverio / webdriverio