search

Drieam / rspec-github

Formatter for RSpec to show errors in GitHub action annotations
https://drieam.github.io/rspec-github/
MIT License
61 stars 10 forks source link

Output rspec seed if used #15

Closed grobie closed 2 years ago

grobie commented 2 years ago

Rspec supports running the specs in a random order. This helps in surfacing any accidental dependencies and side effects in the test suite. When debugging test failures caused by such side effects, it's often necessary to reproduce the issue by running the specs in the same order again. For that the seed value needs to be known and logged by the formatter.

StefSchenkelaars commented 2 years ago

@grobie Thanks for the PR. Code loods good but I'm doubting if this should be in this formatter. For me it feels that this is another formatter (or an option for that matter).

What about splitting this output to another class completely and load both of them? Does that make sense for you too? You can then even make separate versions: one which outputs it like this one and maybe also one that translates it into a github action annotation. Let me know what you think about those things.

If you don't like that, I'm also fine with merging this.

grobie commented 2 years ago

@StefSchenkelaars Apologies for the late response. How important is it to you to split this out?

The seed in the output has helped us multiple times already to re-run flaky tests in the same order. I expect every rspec user who enables random spec runs to have the same need, so splitting this into multiple formatters seems unnecessarily complex.

My order of preference would be: 1) you merge this :smile: 2) I change it as requested 3) We leave the PR unmerged

StefSchenkelaars commented 2 years ago

@grobie I got my permissions for the repo. Rubocop is complaining about an empty line. Once that's fixed I'll merge it 👍

grobie commented 2 years ago

Thanks @StefSchenkelaars, I made the requested change. It would be very helpful if you could release a new version afterwards, so that we can remove the git dependency to install from my branch. Cheers!

StefSchenkelaars commented 2 years ago

@luukvh Since this is a PR from a fork, reviewdog does not have the permissions to push the results through the Checks API. Could you disable the requirement for rubocop for a merge? This is covered by the reviewdog result itself as well.