Closed fj closed 9 years ago
FWIW, I'm running spork
with bundle exec spork rspec
, rake
with bundle exec rake
, and rspec
with bundle exec rspec --drb spec
.
I'm not using Spork myself, but from what I understand is that it always spins up a new process when the test suite should be run.
The Ruby Coverage library SimpleCov uses works by tracking every require
d file AFTER it has been loaded. Therefore, if SimpleCov is loaded AFTER the Rails app and all of it's source files, it won't be able to see them (I think this is what you're getting with the each_run
solution).
So I imagine the solution would be to make Spork require 'simplecov'; SimpleCov.start
before loading the app's environment every time, so the actual code files become visible to SimpleCov. But as I said, I'm not using Spork and I'm not sure how exactly it does it's magic. Maybe the authors of Spork could shed some light onto this?
Quick update: I pinged @timcharper about this, hopefully he can enlighten us a bit :)
I too am having this issue. Seems not many people run into this problem, as I can't find anything anywhere else.
@fj, I bet your problem is your test environment caches classes. See @Estheth's comment in #30.
fwiw, I had been using the hack below in each_run
and setting cache_classes
to true but that doesn't work:
# HACK so we can cache-classes in test.rb
Dir["#{Rails.root}/app/**/*.rb"].each { |f| load f }
require 'simplecov'
SimpleCov.start 'rails'
I have this issue, i put the two lines for simplecov at the top of the Spork.prefork
block. When I use guard no coverage report is generated but when I run rake spec
it works. Would be nice if it worked with guard though.
+1. Same applies to configuration with cucumber. I bet it's spork related
+1. Same config as @Godisemo along with the same issue.
+1 Same here. I swear, it used to work....
+1
simplecov (0.5.4) guard-rspec (0.5.10) guard-spork (0.4.1) spork (0.9.0)
Guys, this obviously is related to #60 - the offender being Spork's requirement to set config.cache_classes = false
in config/environments/test.rb
. Please try to set it to config.cache_classes = true
. As I do not use Spork I don't know what the implications are for spork when you do this, but without it SimpleCov will fail to operate correctly :(
@colszowka I had config.cache_classes
set to true
and false
and it still wouldn't generate the coverage directory/report after guard finishes running tests. Additionally setting config.cache_classes = false
in config/environments/test.rb
causes ActiveRecord::AssociationTypeMismatch
exceptions when using FactoryGirl: http://stackoverflow.com/questions/5954690/strange-activerecordassociationtypemismatch
Something must be wrong in your setups. I successfully use both config.cache_classes=false
and FactoryGirl to get code coverage results under Tork (not Spork).
You guys should try using the Coverage library directly in your setups. If it works, then SimpleCov needs fixing. Else, you know what to do. :)
I'm afraid I don't have any experience with SimpleCov to be able to shed much light on this. But, I can say this:
You shouldn't need config.cache_classes = false
, although that's the easiest way to get your models and such to reload automatically. Read these wiki pages for more info:
https://github.com/sporkrb/spork/wiki/Troubleshooting (specifically: Some changes to files don’t take effect until I restart Spork.)
https://github.com/sporkrb/spork/wiki/Spork.trap_method-Jujutsu
All you should need to do is delay the initialization of SimpleCov until after the process forks, and SimpleCov should be none the wiser.
If a SimpleCov devel wants to do some remove pair programming with me to crack this together, I'd be happy to schedule some time.
On Thu 09 Feb 2012 10:44:56 AM PST, Tim Harper wrote:
You shouldn't need
config.cache_classes = false
, although that's the easiest way to get your models and such to reload automatically.All you should need to do is delay the initialization of SimpleCov until after the process forks, and SimpleCov should be none the wiser.
That is correct. You should call Coverage.start()
after forking. The
coverage results should be collected and processed by an at_exit hook
that was registered once before forking. Here is a working example:
https://github.com/sunaku/tork/blob/coverage/lib/tork/config/coverage.rb#L6-10
FWIW, this is working for me now. I initially had some issues and after a bunch of mucking about it now works (even though my config is right where it was when I started).
The one minor change I made to my code was to skip my custom initializer which bootstrapped some of my custom libs. I now require the initializer manually in the each_run block. This is the complete contents of my test_helper.rb (I'm using minitest):
https://gist.github.com/1820849
I also disabled cache_classes as noted above, though I'm not sure that's absolutely necessary.
So I'm not familiar with Tork, but @sunaku is the ordering of that Coverage.start()
hook before app files are (re)loaded dependent simply on internal load order within Tork? It seems like we'd need something akin to prepend_each_run
in Spork to get things like SimpleCov to run immediately after forking and before (for instance) spork-rails
' each_run
blocks kick off any reloading, etc., does that sound right @timcharper or am I oversimplifying?
@chetan It's not clear what you're doing in the files you require from your Gist -- if you're loading your app env in each_run
every time, you aren't getting much out of using Spork, no?
@ches oops, I forgot to attach test_prefork. Basically I bootstrap the whole rails and test environment w/ the exception of all app/ lib/ and test/ classes. My test runtime has gone from about 10sec (mostly in Bundle.setup) to 1sec. Updated the gist:
I'm having the same issue, following the same logic grid as in the first report. I tried toggling class caching in the test environment to no avail. The coverage report just doesn't run unless I manually run rake spec
. Everything works fine without spork.
Versions:
simplecov 0.6.1
guard 1.0.0
spork 0.9.0
rails 3.2.0
spec_helper.rb https://gist.github.com/1934043
rails (3.2.2)
rspec-rails (2.8.1)
simplecov (0.6.1)
spork (0.9.0)
For me, SimpleCov works when Spork is shut down, but not when Spork is running. I tried all the suggestions above (and from other Google searches on the same problem) to no avail; in particular, setting caching to true or false makes no difference. In the end, I learned two things:
Don't know if this will help anyone, but at least I have a way to get a coverage report without shutting Spork down every time.
This works:
Spork.prefork do
unless ENV['DRB']
require 'simplecov'
SimpleCov.start 'rails'
end
# other code ...
end
Spork.each_run do
if ENV['DRB']
require 'simplecov'
SimpleCov.start 'rails'
end
# other code ...
end
@fredwu nice, it's working for me now.
@fredwu Works for me too!
ruby - 1.9.3 rails - 3.2.2 spork - 1.0.0rc
@fredwu Excellent! That was the piece I needed to unlock the puzzle. There were two other things I had to do:
Now I get identical results for "rake spec" and "rspec spec", with and without Spork running. Thank you! (I still have "config.cache_classes = false" by the way.)
I am getting irregular coverage results with rspec tests while running against spork, despite applying @fredwu 's fix and playing with stuff like:
For now I am simply going to leave it alone since the trouble proved to be too much for any potential gain (for those interested, my setup: https://github.com/prusswan/hw3_rottenpotatoes/tree/hw4)
@prusswan Your repo's Gemfile doesn't have :require => false
. Also, I just checked my test.rb environment, it's config.cache_classes = true
.
@fredwu as I said I have already tried changing those parameters but they did not improve matters, the result with rspec is pretty erratic.
@prusswan and others, the solutions suggested here got me part of the way. I think the problem is when some of the code you want coverage reports for is run for the first time during Rails initialization (for example, if an initializer in config/initializers does some config on models or custom code in your lib directory or whatever). SimpleCov won't pick that stuff up because it's loaded before SimpleCov is. A solution is to split loading and initialization between prefork and each_run by requiring application.rb instead of environment.rb, and then explicitly initializing your app in each_run. This solution works with config.cache_classes = true and if threadsafe mode has not been turned on. A detailed blog post is here: http://rickyrobinson.id.au/2012/07/20/when-spork-puts-a-fork-in-your-cucumber-and-a-spanner-in-your-specs.
I still can't tell much about spork, but on the Ruby Toolbox I've been using spin(+guard-spin) with great success recently. It basically only preloads your app's gems (so probably there's a little time delay compared to spork), but works without any config, and it works smoothly with simplecov so far (of course, only mentioning coverage for the latetst spec that's been run - you can avoid this by expanding merge_timeout
and setting a pseudo-random command_name
)
@colszowka Hey, spin looks pretty cool. Thanks for the pointer. It runs my specs just fine after I take out all the spork-related stuff, but not sure if it'll run my cukes. I note that spin takes the same approach to splitting loading and initialization that my spork hack takes:
We require config/application because that file (typically) loads Rails and any Bundler deps, as well as loading the initialization code for the app, but it doesn't actually perform the initialization. That happens in config/environment.
In my experience that's the best we can do in terms of preloading. Rails and the gem dependencies rarely change and so don't need to be reloaded. But you can't initialize the application because any non-trivial app will involve it's models/controllers, etc. in its initialization, which you definitely don't want to preload.
http://jstorimer.github.com/spin/#section-9
Update: got spin working with cucumber and made a few other changes: http://rickyrobinson.id.au/2012/07/24/spork-to-spin
Having the simplecov runs fine with spork. But the report generated is missing the helper files. But once I stopped spork and run all the specs, then the report includes helper files. Any idea? Anyone have the same issue or know how to fix it?
@fredwu's workaround works for me only if I don't add an add_filter
in simplecov
.
SimpleCov.start "rails" do
add_filter "vendor"
end
When I do it I need to configure config.cache_classes = false
in test.rb
, otherwise I get /coverage. 0 / 0 LOC (0.0%) covered.
.
I don't feel comfortable disabling cache_classes
, is that the correct way to fix it?
An additional observation: If you're using ActiveRecord observers, and activating them from your config/application.rb
, the models they observe will still be skipped, even with @fredwu's fix.
The observed models being skipped seems to only be an issue when I run my specs through Spork.
Works for me:
I followed the "this works" suggestion by fredwu. It didn't work with Spork at first. Then I additionally followed bjnord's suggestion to add :required => false
to the simplecov
line in the Gemfile. That combination seems to have solved the problem for me. Now coverage metrics are being generated BOTH with and without Spork.
I should note though, that in my test.rb, the following is still unchanged, and that seems to not cause a difference. It still works without modification:
config.cache_classes = true
Further update:
Some of my models weren't showing up when running Rspec under Spork with Guard iteratively, but their dependencies were showing up. After trying a few things, it turns out both of the following are needed for simplecov
to deterministically figure out the dependencies on every partial run:
In test.rb
do:
config.cache_classes = false
In your spec, do:
require 'spec_helper'
require '<model>'
I had the same issue with some application_helper lines that were not evaluated as covered code when running guard+spork. When running rspec without guard, coverage was 100% for that helper.
I use a gem that is located under vendor/gems at the moment which is not watched by guard, I excluded it with a filter in the .simplecov file.
I removed the vendor-filter and added :spec_paths => ["spec", "vendor/gems/.../spec"] to the Guardfile, so the gem-tests are now run together with my rails tests. With this configuration, the coverage is correct with AND without guard/spork.
SimpleCov will only pickup if we pass the --no-drb
flag, as in:
bundle exec rspec spec/ --no-drb
Though I'm not sure why...
Here is how I solved the problem
in environment/test.rb config.cache_classes = false
in spec_hepler.rb
Spork.each_run do require 'simplecov' SimpleCov.start do add_filter '/spec/' add_filter '/config/' add_filter '/lib/' add_filter '/vendor/'
add_group 'Controllers', 'app/controllers'
add_group 'Models', 'app/models'
add_group 'Helpers', 'app/helpers'
add_group 'Mailers', 'app/mailers'
add_group 'Views', 'app/views'
end
I am using Sinatra, RSpec, Guard, Spork, and SimpleCov, and I had the same issue where code metrics weren't being generated properly while running with Guard.
Posting here, with a complete spec_helper.rb
file, in case other people try to do something similar with Sinatra.
This was my initial spec_helper, and the tip above where you checked for the DRB env var didn't work for me. This is what my spec_helper.rb
started with. You can see the requiring files in the prefork
block, which is what was the issue.
# Note: Ideas from http://blog.codenursery.com/2011/11/adding-rspec-to-sinatra.html
# and http://www.iamzp.com/blog/2013/04/12/test-driven-development-with-sinatra-rspec-and-guard/
require 'rubygems'
require 'spork'
ENV['RACK_ENV'] = 'test'
def require_files
ruby_files = File.join(File.dirname(__FILE__), '..', 'lib', '**', '*.rb')
Dir[ruby_files].each do |file|
require file
end
require File.join(File.dirname(__FILE__), '..', 'app.rb')
end
Spork.prefork do
require_files
require 'rubygems'
require 'sinatra'
require 'rspec'
require 'rack/test'
# Set up the test env
set :environment, :test
set :run, false
set :raise_errors, true
set :logging, false
def app
@app ||= Sinatra::Application
end
RSpec.configure do |config|
config.include Rack::Test::Methods
end
end
Spork.each_run do
end
I used some of the ideas above to change it to what follows, which generates the correct code coverage metrics:
# Note: Ideas from http://blog.codenursery.com/2011/11/adding-rspec-to-sinatra.html
# and http://www.iamzp.com/blog/2013/04/12/test-driven-development-with-sinatra-rspec-and-guard/
require 'rubygems'
require 'spork'
ENV['RACK_ENV'] = 'test'
def require_files
ruby_files = File.join(File.dirname(__FILE__), '..', 'lib', '**', '*.rb')
Dir[ruby_files].each do |file|
require file
end
require File.join(File.dirname(__FILE__), '..', 'app.rb')
end
Spork.prefork do
require 'rubygems'
require 'sinatra'
require 'rspec'
require 'rack/test'
# Set up the test env
set :environment, :test
set :run, false
set :raise_errors, true
set :logging, false
def app
@app ||= Sinatra::Application
end
end
Spork.each_run do
require 'simplecov'
SimpleCov.start do
add_filter '/spec/'
end
require_files
RSpec.configure do |config|
config.include Rack::Test::Methods
end
end
Closing as stale
tl;dr:
Summary of results:
I'm on an existing Rails project using Mongoid, Devise, RSpec, and Spork (0.9.0.rc5). Our
spec/spec_helper.rb
looks like this:This configuration will run SimpleCov when Spork is not running, and it produces the expected result. When Spork is running, no coverage message is shown.
If I move the
require 'simplecov'; SimpleCov.start 'rails'
lines to the beginning of the Spork.each_run block, as SimpleCov's readme suggests, then when Spork is not running, I get this:If Spork is running, then instead I get no coverage message, as before.