Open franzliedke opened 1 year ago
Please excuse my incorrect idioms. 😆
@franzliedke Thanks for opening this issue and for offering your help!
Apparently, all that needs to be done is exposing two environment variables to the sub processes, as explained here. Would that be feasible? (Happy to send a PR if you approve.)
Sure! Environment variables can be passed in TurboTests::Runner#start_subprocess
.
_I've unclearly described the usage of parallel_tests
in this project. turbo_tests
uses Ruby threads. parallel_tests
are used to group spec files per process. Clarified the difference between two projects in fb33eb4ed03ed6e9ac823365c869e88e74f322f2._
Nice! Thanks for the pointers.
I'm in project mode at work right now. Roughly one month from now, we will have time for stuff like this. See you then. 🙂
Hi, this issue is relevant to me and I could probably pick it up earlier than @franzliedke . I'm slightly confused by the SimpleCov docs, as they seem to imply TEST_ENV_NUMBER
is enough, but turbo_tests
already sets that does it not?
turbo_tests
already sets that does it not
Yes, turbo_tests
sets TEST_ENV_NUMBER
. But there's a bug — value for the first process is incorrect (https://github.com/serpapi/turbo_tests/issues/12).
If it helps anyone, I managed to get simplecov working with turbo_tests by making sure that each test run was named seperately for simplecov - so for example, add this to spec/rails_unit_helper.rb
:
SimpleCov.command_name "test:#{ENV['TEST_ENV_NUMBER']}"
which then results in the following output:
Coverage report generated for test:1, test:2, test:3, test:4 to /../coverage. 12129 / 12784 LOC (94.88%) covered.
Coverage report generated for test:1, test:2, test:3, test:4 to /../coverage.xml. 12129 / 12784 LOC (94.88%) covered
without this, each test runner was overwriting the same coverage files.
@mrmason, thank you! SimpleCov users reported thread-safety bugs with parallel_tests
: https://github.com/simplecov-ruby/simplecov/issues/559. Have you experienced any of these with turbo_tests
?
P.S. SimpleCov.command_name
detects TEST_ENV_NUMBER
automatically: https://github.com/simplecov-ruby/simplecov/blob/216b2d530bee7c4b8a8fe2898684924bfccfa79a/lib/simplecov/command_guesser.rb#L23-L28
P.S.
SimpleCov.command_name
detectsTEST_ENV_NUMBER
automatically: https://github.com/simplecov-ruby/simplecov/blob/216b2d530bee7c4b8a8fe2898684924bfccfa79a/lib/simplecov/command_guesser.rb#L23-L28
Yep, and that's what's currently missing in turbo_tests
, right? That's what this ticket is about, if I remember correctly. 😆
@mrmason, thank you! SimpleCov users reported thread-safety bugs with
parallel_tests
: simplecov-ruby/simplecov#559. Have you experienced any of these withturbo_tests
?P.S.
SimpleCov.command_name
detectsTEST_ENV_NUMBER
automatically: https://github.com/simplecov-ruby/simplecov/blob/216b2d530bee7c4b8a8fe2898684924bfccfa79a/lib/simplecov/command_guesser.rb#L23-L28
This is interesting, because without adding the code that I put above, the outputs over write each other and the test coverage isn't correct - not sure why it's not working.
We're not seeing any direct thread-safety bugs - however I am seeing the following error occasionally when the test suite runs:
Finished in 2 minutes 17.2 seconds (files took 5.22 seconds to load)
3130 examples, 0 failures, 3 pending, 1 error occurred outside of examples
This causes the job to fail with an exit code of 1 - but there are no errors logged that I can see - do you have any suggestions to try and find out where/what the 1 error occurred outside of examples
is ?
Yep, and that's what's currently missing in
turbo_tests
, right? That's what this ticket is about, if I remember correctly.
@franzliedke Right :-) I've missed that SimpleCov relies on PARALLEL_TEST_GROUPS
and TEST_ENV_NUMBER
environment variables for automatic detection: https://github.com/simplecov-ruby/simplecov/blob/216b2d530bee7c4b8a8fe2898684924bfccfa79a/lib/simplecov.rb#L434-L436
@mrmason Is 1 error occurred outside of examples
occurring without SimpleCov too?
Here's what people on StackOverflow suggest:
--bisect
with a specific --seed
: https://stackoverflow.com/a/49120567/1291371example_status_persistence_file_path
: https://stackoverflow.com/a/52873584/1291371P.S. Can be related to https://github.com/serpapi/turbo_tests/pull/20/files/074f68f6d782e1435b779d5631150f114713dc21#diff-c68c40e74f2dc811296e7f0a1088dc810d5cb72d326f703782ad7a22ced5835fR112
Did anyone have success on using simplecov
or any other rails code coverage tool with turbo_tests
?
It's also possible that PARALLEL_TEST_GROUPS
environment variable is required for simplecov
.
I ran into this issue while trying to get coverage to work correctly. Setting command_name
explicitly in spec_helper.rb
as mentioned in https://github.com/serpapi/turbo_tests/issues/24#issuecomment-1503382454 seems to work fine for me.
Setting PARALLEL_TEST_GROUPS
explicitly did not work. While SimpleCov does have built-in support for parallel_tests, setting PARALLEL_TEST_GROUPS
makes SimpleCov think parallel_tests is in use, and it then attempts to rely on other parallel_tests specifics, directly calling methods within that gem that don't seem applicable to turbo_tests:
~/.gem/ruby/3.2.3/gems/parallel_tests-4.6.0/lib/parallel_tests.rb:42:in `fetch': key not found: "PARALLEL_PID_FILE" (KeyError)
Did you mean? "PARALLEL_TEST_GROUPS"
from ~/.gem/ruby/3.2.3/gems/parallel_tests-4.6.0/lib/parallel_tests.rb:42:in `pid_file_path'
from ~/.gem/ruby/3.2.3/gems/parallel_tests-4.6.0/lib/parallel_tests.rb:38:in `pids'
from ~/.gem/ruby/3.2.3/gems/parallel_tests-4.6.0/lib/parallel_tests.rb:90:in `number_of_running_processes'
from ~/.gem/ruby/3.2.3/gems/parallel_tests-4.6.0/lib/parallel_tests.rb:86:in `wait_for_other_processes_to_finish'
from ~/.gem/ruby/3.2.3/gems/simplecov-0.22.0/lib/simplecov.rb:279:in `wait_for_other_processes'
from ~/.gem/ruby/3.2.3/gems/simplecov-0.22.0/lib/simplecov.rb:110:in `result'
from ~/.gem/ruby/3.2.3/gems/simplecov-0.22.0/lib/simplecov/configuration.rb:197:in `block in at_exit'
from ~/.gem/ruby/3.2.3/gems/simplecov-0.22.0/lib/simplecov.rb:189:in `run_exit_tasks!'
from ~/.gem/ruby/3.2.3/gems/simplecov-0.22.0/lib/simplecov.rb:179:in `at_exit_behavior'
from ~/.gem/ruby/3.2.3/gems/simplecov-0.22.0/lib/simplecov/defaults.rb:30:in `block in <top (required)>'
Setting command_name
avoided all these problems, but it would of course be great if Simplecov just worked without any extra config!
https://github.com/serpapi/turbo_tests/issues/24#issuecomment-1503382454 is also working for me, but i'm getting multiple output for simplecov instead of a summarized one. Is there a workaround for that?
were wondering the same, i think thats the last thing missing from this imo
SimpleCov tries to automatically determine the test framework / tools being used based on process name and environment variables.
Since this project is based on
parallel_tests
, for which SimpleCov claims support, I was expecting this to "just work" as well.Apparently, all that needs to be done is exposing two environment variables to the sub processes, as explained here. Would that be feasible? (Happy to send a PR if you approve.)