search

simplecov-ruby / simplecov

Code coverage for Ruby with a powerful configuration library and automatic merging of coverage across test suites
MIT License
4.77k stars 552 forks source link

Coverage results generated before tests finish for a gem under development. #1112

Open ThomasOwens opened 2 days ago

ThomasOwens commented 2 days ago

I was able to put together a very simple repository that demonstrates the issue, including committing the coverage files.

https://github.com/ThomasOwens/simplecov-bug-demo

I'm building a gem, which was initialized using the bundler gem tooling. After initializing the gem, I added a dependency on simplecov for development and configured it per instructions. However, when I run commands (like bundle exec rake, which is configured to run tests and rubocop), I get output, as shown below, that indicates the coverage report is being generated before test execution.

At first, I thought it had to do with the initialization of application code before simplecov, but it looks like the only thing being initialized is the standard version.rb file. It appears to be a timing of generation versus test based on the console output.

In the reproduction example repository, I would expect no coverage of lib/simplecov-bug-demo/version.rb, because this file is required in the gemspec. However, I would expect lines 7-8 of lib/simplecov-bug-demo/commands/command.rb to be covered. However, neither line is covered. Only the module, class, and method declaration lines are covered, which is what I would expect for simply loading the file after executing simplecov.

Without a .simplecov file, the console output of bundle exec rake looks like:

Coverage report generated for Minitest to /workspaces/simplecov-bug-demo/coverage.
Line Coverage: 75.0% (9 / 12)
Run options: --seed 44598

# Running:

..

Finished in 0.001718s, 1164.1532 runs/s, 1164.1532 assertions/s.

2 runs, 2 assertions, 0 failures, 0 errors, 0 skips

With a .simplecov file, the console output of bundle exec rake is:

root@72365261d8df:/workspaces/simplecov-bug-demo# bundle exec rake
Warning: Error occurred while trying to load /workspaces/simplecov-bug-demo/.simplecov. Error message: uninitialized constant Simplecov
Run options: --seed 20890

# Running:

..

Finished in 0.001560s, 1281.7818 runs/s, 1281.7818 assertions/s.

2 runs, 2 assertions, 0 failures, 0 errors, 0 skips

I'm using Ruby 3.3 inside a devcontainer built on top of a Ruby 3.3 docker image. Minitest is at version 5.25.1, Rake at 13.2.1, and simplecov at 0.22.0.

The underlying cause may be the same as #1099. However, that issue does talk specifically about Rails, which is not involved in my project.

ThomasOwens commented 3 hours ago

I asked about this in a Stack Overflow question and got an answer from mitch_. The solution was, as described, to add SimpleCov.external_at_exit = true before the SimpleCov.start in the test_helper.rb file. I also did have to convert the start to a block and explicitly filter the vendor and test directories.

If this doesn't represent a bug in SimpleCov, it would be worth updating the readme to describe how to integrate SimpleCov with minitest. If there's a more appropriate solution, it would be good to get that out into the world as well.