Coverage not working CI server

[jhnvz]

I have both phantomjs and istanbul installed locally and on CircleCI.

teaspoon_env.rb:

config.use_coverage = :default
config.coverage :default do |coverage|
  coverage.reports = ["text", "html"]
  coverage.statements = 90
  coverage.functions  = 90
  coverage.branches   = 90
  coverage.lines      = 90
end

local:

$ which phantomjs #=> /usr/local/bin/phantomjs
$ phantomjs -v #=> 1.9.7

$ node -v #=> v0.10.26
$ npm -v #=> 1.4.3

$ which istanbul #=> /usr/local/share/npm/bin/istanbul
$ istanbul help #=> version:0.2.7

circleci:

$ which phantomjs #=> /usr/local/bin/phantomjs
$ phantomjs -v #=> 1.9.7

$ node -v #=> v0.10.26
$ npm -v #=> 1.4.7

$ which istanbul #=> /home/ubuntu/nvm/v0.10.26/bin/istanbul
$ istanbul help #=> version:0.2.7

I've modified 'teaspoon/runner' to:

module Teaspoon
  class Runner
    def result_from_json(json)
      result = Teaspoon::Result.build_from_json(json)
      p result.inspect if result.type == 'result'
      @failure_count += 1 if result.failing?
      result
    end
  end
end

Locally the coverage attribute is filled with data on circleci I get the following output:

"#<struct Teaspoon::Result type=\"result\", suite=nil, label=nil, status=nil, skipped=nil, link=nil, message=nil, trace=nil, elapsed=\"0.05100\", total=nil, start=nil, level=nil, coverage=nil, original_json=\"{\\\"type\\\":\\\"result\\\",\\\"elapsed\\\":\\\"0.05100\\\",\\\"_teaspoon\\\":true}\\n\">"

Any ideas on what might go wrong?

[jejacks0n]

Thanks for all the version info, that's always useful. So, one question.. which version of rails?

And what I assume is happening.. The instrumentation probably isn't loading or working because assets are getting precompiled. For the coverage to work, it must instrument the javascripts that are being tested, and for that to happen, there's a middleware/sprockets hack in the instrumentation part of teaspoon. So, precompiling assets for tests is probably not the best case if you need coverage reports.

[jhnvz]

Thanks for the quick response! We're running rails 4.0.3. I don't think assets are getting precompiled since the commands we run on our ci server are pretty much the same as locally.

[jhnvz]

Is there I way I can check/debug if instrumentation isn't loading/working correctly?

[jejacks0n]

try:

RAILS_ENV=test bundle exec teaspoon

[jhnvz]

Already tried that. When I run RAILS_ENV=test bundle exec teaspoon locally it also doesn't generate coverage.

[jhnvz]

When your run bundle exec teaspoon (locally) the ENV['RAILS_ENV'] is nil. Rails defaults to the development environment so config/environments/development.rb is loaded.

When your run bundle exec teaspoon on circleci ENV['RAILS_ENV'] is 'test' so config/environments/test.rb is loaded. When you change config.cache_classes to false in for the test environment instrumentation works.

Thanks for pointing me to the right direction.

[jcarlson]

I just spent too long trying to diagnose this same problem. Can anyone possibly explain to me why config.cache_classes = false in my test environment allows coverage to run properly? I don't see the connection. Cucumber emits a warning about class caching being enabled if I do this...