Simplecov coverage not calculating

[dgilperez]

Hi,

We're having trouble getting simplecov coverage in conjunction with flatware. Specs are working correctly with and without flatware, but when we use flatware we're getting 0% coverage.

Output without flatware:

$> bundle exec rspec path_to_spec
.........
Finished in 18.56 seconds (files took 5.67 seconds to load)
3 examples, 0 failures
Randomized with seed 1972
Coverage report generated for RSpec to path. 5784 / 24305 LOC (23.8%) covered.

$> bundle exec flatware rspec path_to_spec
......
Finished in 27.71 seconds (files took 22.54 seconds to load)
3 examples, 0 failures
Coverage report generated for RSpec to path. 0 / 30640 LOC (0.0%) covered.

Is this a known problem? Is there any special setup for flatware to play nice with simplecov?

I also find strange that the LOC number differs from both runs for the same spec.

[briandunn]

This is working for me on 2.0. Note that if you use the new before_fork and after_fork features you'll need to make sure to load (or reload) simplecov in the after_fork hook.

[G-Rath]

@briandunn I've been trailing using flatware with an application generated from my works rails-template - so far it seems really promising, but I'm hitting what looks to be pretty much the same issue: when running my tests they're super fast but SimpleCov reports no lines or branches covered out of the box.

My understanding is that our setup is reasonably standard and I've tried playing around with various combinations including with after_fork with some success, but I'm by no means an expert with setting up RSpec and SimpleCov so I could be missing something obvious; the closest I've come is getting loading SimpleCov in after_fork which results in coverage being collected seemingly successfully which can be viewed in the HTML report but SimpleCov itself exits with exit code 2 saying that coverage has not been satisfied along with really giving really ugly output (it feels like maybe there needs to be some "after all tests are done" hook where final single process of SimpleCov can be told to merge all the results before exiting...)

Would you be willing to help with this, even if its just giving some further pointers? I'm happy to create a new issue to track this, generate a sample project from the template, etc - and hopefully contribute back some documentation on setting up SimpleCov.

[briandunn]

Hi @G-Rath! Yes some help would be great. I can reopen this issue. Can you provide a repro script? gem versions, and then what options to rails-template, contents of before_fork and after_fork. It does look like there is a way to merge simplecov results - that after_run hook could be helpful.

Maybe this could also be done by using TEST_ENV_NUMBER in the simplecov config to generate a different resultset.json file per runner, then run a separate script to merge those after flatware exits. Perhaps that script is where you'd fail the run if coverage is too low.

[briandunn]

Added instructions for use with SimpleCov.

[G-Rath]

@briandunn sorry for the delay in responding - that looks like it does the trick so long as SimpleCov.use_merging is also set to true (which I think it is by default but we set it to false in our template since it seems to have cross-cutting behaviour whereby it'll happily merge old coverage results if you're fast enough...)

I've got a separate issue with my system tests failing due to not being able to find Chrome sometimes, which I expect is to do with it being an external process and not expecting parallel calls; that's unrelated to this issue though 😄

[briandunn]

Glad it's working for you! You probably aren't the only person to have issues with launching chrome in parallel. Feel free to open an issue and I'll help if I can.

[mjankowski]

Adding to this issue, I'm curious whether branch data should be properly preserved in the flatware/simplecov scenario?

In my ongoing saga of attempting to get a flatware/simplecov/codecov situation working - https://github.com/mastodon/mastodon/pull/30284#issuecomment-2112633034 - I've hit the point where I've got the speedups from flatware, and I've got the normal line coverage results merging correctly, but it seems like the branch data just gets dropped when run via flatware.

There are a bunch of open simplecov issues/PRs related to branch data, so I'm guessing that the issue is likely on that side - but figured I'd ask here for any pointers as well.