simplecov-ruby / simplecov

Code coverage for Ruby with a powerful configuration library and automatic merging of coverage across test suites
MIT License
4.74k stars 550 forks source link

Why: `INSUFFICIENT CODE COVERAGE` is followed by `Code coverage is acceptable` and exit code is `0` #1036

Open drhuffman12 opened 1 year ago

drhuffman12 commented 1 year ago

When I run rspec (or bin/rails coverage:assert_minimum itself), and my coverage is less then the MIN_COVERAGE_PERCENT, I would expect an error code of some positive number, but instead, I get a exit code of 0 and the following INSUFFICIENT/acceptable confusing text:

root@9fe4400f4469:/var/www/our_app# RAILS_ENV=test MIN_COVERAGE_PERCENT=43.7 bin/rails coverage:assert_minimum
----------------------------------------
INSUFFICIENT CODE COVERAGE
Minimum: 43.7%
Actual:  35.79396325459317%
----------------------------------------
*** Code coverage is acceptable: (minimum: 43.7% / actual: 35.79396325459317%) ***

My version info are:

root@9fe4400f4469:/var/www/our_app# ruby -e "puts RUBY_DESCRIPTION"
ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) [x86_64-linux]

root@9fe4400f4469:/var/www/our_app# rails -v
Rails 6.1.7

root@9fe4400f4469:/var/www/our_app# bundler -v
Bundler version 2.3.23

root@9fe4400f4469:/var/www/our_app# gem -v
3.3.7

root@9fe4400f4469:/var/www/our_app# gem list | grep simplecov
simplecov (0.21.2, 0.16.1)
simplecov-html (0.12.3, 0.10.2)
simplecov-json (0.2.3)
simplecov_json_formatter (0.1.4)

root@9fe4400f4469:/var/www/our_app# gem list | grep minitest
minitest (5.16.3, 5.15.0)

root@9fe4400f4469:/var/www/our_app# gem list | grep rspec
rspec (3.11.0)
rspec-core (3.11.0)
rspec-expectations (3.11.1)
rspec-mocks (3.11.1)
rspec-rails (5.1.2)
rspec-support (3.11.1)
rspec_api_documentation (6.1.0)
rubocop-rspec (2.13.1)

Help please; I want to use the exit code to trigger a failure in the CI process.

drhuffman12 commented 1 year ago

Closing; Oops, sorry, apparently, there were some internal modes that broke the exit code.

I guess I can't close my own ticket. Please close this ticket.

zenspider commented 3 weeks ago

Please close