This goal here, initially, was to fix issue 282. However, after
using with_backtrace extensively in our custom assertions in
our apps, I've decided an alternate behavior for this helper is
preferable. This switches to the new behavior which, in turn,
fixes issue 282.
First off, I'd like to restate the original goal of with_backtrace
(wbt) which was to use the given caller as the backtrace for any
results produced in the given block. This was useful when your test
assertions were buried in something other than the main test
block (like a shared private method or a mixin method). For our
apps, this most commonly manifests itself in some sort of
assert_custom_behavior shared method that itself makes several
assertions that in concert assert the custom behavior.
Without wbt, if the shared method's assertions produced results
they would use the shared method's backtrace. This produced
incorrect file/line info which caused the results to be ordered
in unexpected ways. For fails/skips we only show the line of the
assertion which makes it difficult to trace back to the actual test
line that originated the result. Finally, wbt's implementation
broke down if you nested wbt calls (ie shared test helper methods
calling other shared test helper methods).
Now, let me describe the new behavior I found myself wanting
while using with_backtrace (wbt). Obviously, the nested bug
needs to be addressed and fixed. However, with shared helpers,
I found myself wanting to see more than just a single line for
fails/skips. I wanted to see the "chain" of the wbt calls -
essentially I wanted a mini backtrace that starts at the test
line and "drills down" through the shared helper methods to the
base assertion line. This way, I can see my test line that
produced the result like normal, but below that I then see the
trace of shared method calls as well so I can debug if needed.
This all needs to continue to produce correct test file/line info
as well.
To accomplish this, I've switched to having wbt push the first line
of its given caller onto a stack for the duration of its given
block. This allows a mini backtrace of nested wbt calls to be
built and then used in the produced results. When setting the wbt
backtrace on results, we tack on the normal line that produced
the result to make sure that is always included (nested at the
bottom of the wbt backtrace. Results now know if a wbt backtrace
is set and fails/skips show their entire wbt backtrace if so.
Finally, to properly "tack on" the normal line that produces the
result, I had to update the backtrace filtering logic to not filter
out included assert/macros lines. These lines are part of assert
but do produce results from shared code. Since they produce the
results, they should be allowed to show up in wbt backtraces.
To illustrate this, given some code like this (I'm temp mod'ing
some Assert system tests):
class SystemTests < Assert::Context
include Assert::Test::TestHelpers
desc "Assert::Test"
subject{ @test }
should have_accessors :an_accessor
should "assert something" do
assert_true false
assert_something
assert_true false
end
private
def assert_something_else
with_backtrace(caller) do
fail "a fail"
end
end
def assert_something
with_backtrace(caller) do
assert_false true
assert_something_else
skip "TODO"
end
end
end
And then running assert test/system/test_tests.rb would produce:
[...]
FAIL: Assert::Test should assert something
Expected true to be false.
/[...]/test/system/test_tests.rb:15:in `block in <class:SystemTests>'
/[...]/test/system/test_tests.rb:29:in `block in assert_something'
assert -t ./test/system/test_tests.rb:13
FAIL: Assert::Test should assert something
a fail
/[...]/test/system/test_tests.rb:15:in `block in <class:SystemTests>'
/[...]/test/system/test_tests.rb:30:in `block in assert_something'
/[...]/test/system/test_tests.rb:23:in `block in assert_something_else'
assert -t ./test/system/test_tests.rb:13
SKIP: Assert::Test should assert something
TODO
/[...]/test/system/test_tests.rb:15:in `block in <class:SystemTests>'
/[...]/test/system/test_tests.rb:31:in `block in assert_something'
assert -t ./test/system/test_tests.rb:13
FAIL: Assert::Test should assert something
Expected false to be true.
/[...]/test/system/test_tests.rb:14:in `block in <class:SystemTests>'
assert -t ./test/system/test_tests.rb:13
FAIL: Assert::Test should respond to methods
NilClass does not have instance method #an_accessor
Expected nil (NilClass) to respond to `an_accessor`.
/[...]/test/system/test_tests.rb:11:in `<class:SystemTests>'
/[...]/lib/assert/macros/methods.rb:108:in `block (3 levels) in _methods_macro_test'
assert -t ./test/system/test_tests.rb:11
FAIL: Assert::Test should respond to methods
NilClass does not have instance method #an_accessor=
Expected nil (NilClass) to respond to `an_accessor=`.
/[...]/test/system/test_tests.rb:11:in `<class:SystemTests>'
/[...]/lib/assert/macros/methods.rb:108:in `block (3 levels) in _methods_macro_test'
assert -t ./test/system/test_tests.rb:11
[...]
This goal here, initially, was to fix issue 282. However, after using
with_backtrace
extensively in our custom assertions in our apps, I've decided an alternate behavior for this helper is preferable. This switches to the new behavior which, in turn, fixes issue 282.First off, I'd like to restate the original goal of
with_backtrace
(wbt) which was to use the given caller as the backtrace for any results produced in the given block. This was useful when your test assertions were buried in something other than the main test block (like a shared private method or a mixin method). For our apps, this most commonly manifests itself in some sort ofassert_custom_behavior
shared method that itself makes several assertions that in concert assert the custom behavior.Without wbt, if the shared method's assertions produced results they would use the shared method's backtrace. This produced incorrect file/line info which caused the results to be ordered in unexpected ways. For fails/skips we only show the line of the assertion which makes it difficult to trace back to the actual test line that originated the result. Finally, wbt's implementation broke down if you nested wbt calls (ie shared test helper methods calling other shared test helper methods).
Now, let me describe the new behavior I found myself wanting while using
with_backtrace
(wbt). Obviously, the nested bug needs to be addressed and fixed. However, with shared helpers, I found myself wanting to see more than just a single line for fails/skips. I wanted to see the "chain" of the wbt calls - essentially I wanted a mini backtrace that starts at the test line and "drills down" through the shared helper methods to the base assertion line. This way, I can see my test line that produced the result like normal, but below that I then see the trace of shared method calls as well so I can debug if needed. This all needs to continue to produce correct test file/line info as well.To accomplish this, I've switched to having wbt push the first line of its given caller onto a stack for the duration of its given block. This allows a mini backtrace of nested wbt calls to be built and then used in the produced results. When setting the wbt backtrace on results, we tack on the normal line that produced the result to make sure that is always included (nested at the bottom of the wbt backtrace. Results now know if a wbt backtrace is set and fails/skips show their entire wbt backtrace if so.
Finally, to properly "tack on" the normal line that produces the result, I had to update the backtrace filtering logic to not filter out included
assert/macros
lines. These lines are part of assert but do produce results from shared code. Since they produce the results, they should be allowed to show up in wbt backtraces.To illustrate this, given some code like this (I'm temp mod'ing some Assert system tests):
And then running
assert test/system/test_tests.rb
would produce:Closes #282.
@jcredding ready for review.