search

simplecov-ruby / simplecov

Code coverage for Ruby with a powerful configuration library and automatic merging of coverage across test suites
MIT License
4.78k stars 553 forks source link

Forked process doing cleanup at_exit can cause duplicate coverage reports #228

Open HerbCSO opened 11 years ago

HerbCSO commented 11 years ago

When you run bundle exec rspec on this repo (that is the most minimal test case I can come up with), there will be two SimpleCov reports generated, with very different coverage percentages. The fact that the UtilityConfig class has a forked process that is cleaned up in an at_exit call seems to cause some sort of race condition in SimpleCov (I have not been able to figure out where this occurs).

Sample output on my system:

> bundle exec rspec
.

Finished in 2 seconds
1 example, 0 failures

Randomized with seed 51923

Coverage report generated for RSpec to /opt/pose/test-simplecov/coverage. 23 / 25 LOC (92.0%) covered.
Coverage (92.00%) is below the expected minimum coverage (95.00%).
Coverage report generated for RSpec to /opt/pose/test-simplecov/coverage. 5 / 25 LOC (20.0%) covered.

The second Coverage report is wrong. If it wasn't for the fact that it overwrites the results of the first (correct) one, I wouldn't mind so much... ;]

Please let me know if I can help with any other input/background.

HerbCSO commented 11 years ago

Oh, and I forgot to mention the best bit - if I debug with rspec -d and step into SimpleCov after my tests are done and then continue execution, only a single report gets generated. Sounds a lot like a race condition of some sort, wouldn't you say?

colszowka commented 11 years ago

Hi Carsten, could you please try the 0.8.0 prerelease from rubygems? It implements a lockfile for the resultset merging and should fix your problem

HerbCSO commented 11 years ago

Thanks for the quick response Christoph!

Tried it just now, but unfortunately that doesn't seem to fix the issue. I still get two runs of the coverage report with different percentages:

> bundle exec rspec
.

Finished in 2.01 seconds
1 example, 0 failures

Randomized with seed 55991

Coverage report generated for RSpec to /opt/pose/test-simplecov/coverage. 23 / 25 LOC (92.0%) covered.
Coverage (92.00%) is below the expected minimum coverage (95.00%).
Coverage report generated for RSpec to /opt/pose/test-simplecov/coverage. 5 / 25 LOC (20.0%) covered.

Just to make sure we're on the same page, here's the Ruby version I'm using:

  ruby:
    interpreter:  "ruby"
    version:      "1.9.2p325"
    date:         "2013-04-15"
    platform:     "x86_64-darwin12.3.0"
    patchlevel:   "2013-04-15"
    full_version: "ruby 1.9.2p325 (2013-04-15) [x86_64-darwin12.3.0]"

and a list of my local gems:

>  gem list --local                                                                                                                                                                                       ruby-1.9.2-head@test-simplecov

*** LOCAL GEMS ***

bundler (1.3.5)
diff-lcs (1.2.4)
multi_json (1.7.6)
rake (10.0.4, 0.8.7)
rspec (2.13.0)
rspec-core (2.13.1)
rspec-expectations (2.13.0)
rspec-mocks (2.13.1)
rubygems-bundler (1.1.1)
rubygems-update (1.4.2)
rvm (1.11.3.7, 1.11.3.5)
simplecov (0.8.0.pre, 0.7.1)
simplecov-html (0.7.1)

(I also tried uninstalling 0.7.1 just in case, same result)

I updated my test case repo with the new Gemfile, just in case you want to see that.

Let me know if you need anything else, please.

HerbCSO commented 11 years ago

So I've found a workaround for my code at least. The solution was to add an at_exit { exit! } into my forked process, which caused it to not fire the main at_exit handler in my code anymore, and then gets around SimpleCov triggering its reporting twice. See this diff for what I did in the context of the code.

colszowka commented 10 years ago

@HerbCSO Cool, thanks! I'll keep this open as a placeholder for a standardized solution built into simplecov