Ruby 1.9.3: Getting test failure from api_test.rb (bbq_test_unit_test.rb)

[afinetooth]

First of all, thanks for bbq! Very excited to work with it more, but having a couple of issues.

I'll submit them separately. The first is minor, but curious to me.

Getting this single failure when I run tests:

---------------------------------------------------------------
# Running tests:

....................F...............

Finished tests in 50.777182s, 0.7090 tests/s, 2.7965 assertions/s.

  1) Failure:
test_api_client(BbqTestUnitTest) [test/unit/bbq_test_unit_test.rb:159]:
Expected /5 tests, 15 assertions, 0 failures, 0 errors/ to match "Run options: \n\n# Running tests:\n\n.....\n\nFinished tests in 0.087016s, 57.4607 tests/s, 206.8585 assertions/s.\n\n5 tests, 18 assertions, 0 failures, 0 errors, 0 skips\n".

36 tests, 142 assertions, 1 failures, 0 errors, 0 skips
---------------------------------------------------------------

It's curious because I count 15 assertions in the method, test_api_client, in the file, test/unit/bbq_test_unit_test.rb, as well as in the file, test/dummy/test/acceptance/api_test.rb, which is generated by it.

Nevertheless, the correct number of assertions seems to be 18.

Spent some time with it, but not enough to understand why.

[paneq]

Which ruby version ?

[afinetooth]

Hi. Ruby version:

ruby-1.9.3-p125

[afinetooth]

Two other details of my environment:

1) Getting this error in first line of test results in console:

[...]/ruby-1.9.3-p125/gems/rake-0.8.7/lib/rake/alt_system.rb:32: Use RbConfig instead of obsolete and deprecated Config.

2) I had bumped the version of rpec-rails in bbq.gemspec:

s.add_development_dependency "rspec-rails", ">= 2.10.1" #, "~> 2.6.0"

in order to avoid six (6) test errors, all with message:

RuntimeError: [...]/ruby-1.9.3-p125@global/gems/bundler-1.1.3/lib/bundler/runtime.rb:31:in `block in setup': You have already activated rspec-core 2.10.1, but your Gemfile requires rspec-core 2.6.4. Using bundle exec may solve this. (Gem::LoadError)

[paneq]

I will look into it in the afternoon.

[afinetooth]

Thanks. No hurry if you have other things to do. I'm going to see if I can replicate with different gemsets.

[afinetooth]

UPDATE:

I eliminated complication (2) above, by doing a fresh clone of bbq and installing a fresh gemset for it.

Then I tested using these rubies:

1.9.3-head 1.9.3-p125 1.9.2-p290

With rubies:

1.9.3-head 1.9.3-p125

I continued getting this error (complication (1) above):

[...]/rake-0.8.7/lib/rake/alt_system.rb:32: Use RbConfig instead of obsolete and deprecated Config.

And I continued getting the one test failure:

1) Failure:
test_api_client(BbqTestUnitTest) [test/unit/bbq_test_unit_test.rb:159]:
Expected /5 tests, 15 assertions, 0 failures, 0 errors/ to match "Run options: \n\n# Running tests:\n\n.....\n\nFinished tests in 0.087016s, 57.4607 tests/s, 206.8585 assertions/s.\n\n5 tests, 18 assertions, 0 failures, 0 errors, 0 skips\n".

However, using ruby:

1.9.2-p290

I get neither the RBConfig error, nor the one test failure.

Instead, I get a perfect test result:

....................................
Finished in 93.176142 seconds.

36 tests, 123 assertions, 0 failures, 0 errors, 0 skips

That said, I don't know why the total number of assertions listed in test results has changed from 142 (see original post) to 123.

But I suppose it's telling that the failure we see with the other rubies, also relates to an incompatible number of assertions. Weird.

(BTW, all of the above has been using latest source, cloned from master twice over the last 3 days. Considering there may have been an update to source, I verified that there have been no commits to bbq since May 18, so the difference in number of assertions is not explained by a code update.)

[mostlyobvious]

Looks like assert_match uses 2 assertions in minitest: https://github.com/seattlerb/minitest/blob/master/lib/minitest/unit.rb#L275

That would fit into 3 additional asserts in that test. Question is what version of minitest 1.9.2 uses and why does it have one assert per assert_match.

[mostlyobvious]

Evidence: https://gist.github.com/2903247

[mostlyobvious]

This is confusing since https://github.com/ruby/ruby/blob/ruby_1_9_2/lib/minitest/unit.rb#L171 does also include 2 assertions though they cheat on output in test/unit wrapper for minitest: https://github.com/ruby/ruby/commit/e501209104a6dbce00fbaec6206c7c6edf3c3e55

And also similar case for not_match: https://github.com/ruby/ruby/blob/ruby_1_9_2/lib/test/unit/assertions.rb#L126

[afinetooth]

Good sleuthing. I see that now.

And yes, it is confusing, still, in that 1.9.3 employs the same cheat to avoid overcounting: https://github.com/ruby/ruby/blob/ruby_1_9_3/lib/test/unit/assertions.rb#L253

And assert_match hasn't (effectively) changed, either: https://github.com/ruby/ruby/blob/ruby_1_9_3/lib/minitest/unit.rb#L279

For good measure, assert_no_match in 1.9.3 (also unchanged): https://github.com/ruby/ruby/blob/ruby_1_9_3/lib/test/unit/assertions.rb#L218

As for which version of minitest 1.9.2 uses, vs 1.9.3, I can at least confirm that I'm not using a gem version, so for my case it's the standard version.

Looking into that now.

[afinetooth]

Not having much luck identifying the correct versions of MiniTest for 1.9.2 vs 1.9.3 (ruby_1_9_3/lib/minitest/unit.rb doesn't indicate version and I'm afraid I don't know enough to know where to look in the project for the dependency info).

In the meantime, however, here is a relevant issue from the MiniTest project:

https://github.com/seattlerb/minitest/pull/2

The pull was rejected. I added a comment pointing to our use case and suggesting reopening.

[zenspider]

The version of minitest is always available via:

% ruby -rubygems -e 'require "minitest/unit"; p MiniTest::Unit::VERSION'
"3.0.0"

I've also backfilled my History.txt file (shipped with the gem, not in ruby's stdlib) to include shipped ruby versions.

This is a brittle test that gains little to test the number of assertions (or even to test the output at all). I'd suggest switching the /15/ to /\d+/ if not dropping testing the output entirely. I test minitest's output thoroughly so you only need to verify that you hook up to minitest correctly.

[afinetooth]

@zenspider Thanks for the info on version and the response.

@pawelpacana IMHO, I think Ryan's point about the test has merit in that the most important test output is the true/false nature of the assertion, rather than the number of assertions; particularly if we don't have control over the number.

The /\d+/ replacement seems like a decent alternative to refactoring the current tests.