search

joshmfrankel / simplecov-check-action

SimpleCov+ Action
MIT License
33 stars 16 forks source link

Support the coverage_drop options #10

Closed multiplegeorges closed 1 year ago

multiplegeorges commented 2 years ago

Hey @joshmfrankel

I'm looking to use this Action, but we're starting off from a fairly low % test suite.

SimpleCov supports the maximum_coverage_drop option. I'd like to use this option enforce that our test coverage get better every time, but not fail at our low overall coverage.

I can't use the existing options because they are hard coded. For example, if I set minimum_suite_coverage to 45% (yeah...) and we improve the coverage to 50%, then a new commit at 48% coverage would pass. I'd like that to fail since it would be a drop of 2%.

I believe this will require caching the previous run json file with a caching action, but I'm not too familiar with the internals of actions.

Thoughts?

joshmfrankel commented 2 years ago

I believe this will require caching the previous run json file with a caching action, but I'm not too familiar with the internals of actions.

I believe you're correct here. You would need to know the previous run coverage amount. This gets tricky because what we need here is the previous coverage amount from the main branch, not from the current PR. This way, you can compare your changeset's coverage percentage versus what the current application coverage is at. GitHub actions have a cache action that would possibly work here: https://github.com/actions/cache.

That being said, I don't have any plans to implement caching in order to keep the action simple. Check out the workaround below to see if that solves your issue

I can't use the existing options because they are hard coded. For example, if I set minimum_suite_coverage to 45% (yeah...) and we improve the coverage to 50%, then a new commit at 48% coverage would pass. I'd like that to fail since it would be a drop of 2%.

Do you mean you can't change the minimum_suite_coverage configuration value for the GitHub workflow, or does your test suite hard-code SimpleCov.minimum_coverage within test_helper.rb?

Workaround If you're able to change minimum_suite_coverage, my suggestion would be to use this as a locking mechanism. From your example above:

  1. Open pull request and add some covering tests,
  2. GitHub action runs, indicating you've brought test suite coverage up to 50%
  3. In the same Pull Request, change minimum_suite_coverage to minimum_suite_coverage: 50 effectively locking passing builds that drop below this
  4. Merge & deploy
  5. Rest of team needs to also ensure that coverage doesn't drop below 50% in order to merge code

Let me know if that solves the issue