redding / assert

Assertion style testing framework.
https://github.com/redding/assert
MIT License
10 stars 1 forks source link

rework how `with_backtrace` helper works and alters fail traces #284

Closed kellyredding closed 6 years ago

kellyredding commented 6 years ago

This goal here, initially, was to fix issue 282. However, after using with_backtrace extensively in our custom assertions in our apps, I've decided an alternate behavior for this helper is preferable. This switches to the new behavior which, in turn, fixes issue 282.

First off, I'd like to restate the original goal of with_backtrace (wbt) which was to use the given caller as the backtrace for any results produced in the given block. This was useful when your test assertions were buried in something other than the main test block (like a shared private method or a mixin method). For our apps, this most commonly manifests itself in some sort of assert_custom_behavior shared method that itself makes several assertions that in concert assert the custom behavior.

Without wbt, if the shared method's assertions produced results they would use the shared method's backtrace. This produced incorrect file/line info which caused the results to be ordered in unexpected ways. For fails/skips we only show the line of the assertion which makes it difficult to trace back to the actual test line that originated the result. Finally, wbt's implementation broke down if you nested wbt calls (ie shared test helper methods calling other shared test helper methods).

Now, let me describe the new behavior I found myself wanting while using with_backtrace (wbt). Obviously, the nested bug needs to be addressed and fixed. However, with shared helpers, I found myself wanting to see more than just a single line for fails/skips. I wanted to see the "chain" of the wbt calls - essentially I wanted a mini backtrace that starts at the test line and "drills down" through the shared helper methods to the base assertion line. This way, I can see my test line that produced the result like normal, but below that I then see the trace of shared method calls as well so I can debug if needed. This all needs to continue to produce correct test file/line info as well.

To accomplish this, I've switched to having wbt push the first line of its given caller onto a stack for the duration of its given block. This allows a mini backtrace of nested wbt calls to be built and then used in the produced results. When setting the wbt backtrace on results, we tack on the normal line that produced the result to make sure that is always included (nested at the bottom of the wbt backtrace. Results now know if a wbt backtrace is set and fails/skips show their entire wbt backtrace if so.

Finally, to properly "tack on" the normal line that produces the result, I had to update the backtrace filtering logic to not filter out included assert/macros lines. These lines are part of assert but do produce results from shared code. Since they produce the results, they should be allowed to show up in wbt backtraces.

To illustrate this, given some code like this (I'm temp mod'ing some Assert system tests):

  class SystemTests < Assert::Context
    include Assert::Test::TestHelpers

    desc "Assert::Test"
    subject{ @test }

    should have_accessors :an_accessor

    should "assert something" do
      assert_true false
      assert_something
      assert_true false
    end

    private

    def assert_something_else
      with_backtrace(caller) do
        fail "a fail"
      end
    end

    def assert_something
      with_backtrace(caller) do
        assert_false true
        assert_something_else
        skip "TODO"
      end
    end

  end

And then running assert test/system/test_tests.rb would produce:

[...]

FAIL: Assert::Test should assert something
Expected true to be false.
/[...]/test/system/test_tests.rb:15:in `block in <class:SystemTests>'
/[...]/test/system/test_tests.rb:29:in `block in assert_something'
assert -t ./test/system/test_tests.rb:13

FAIL: Assert::Test should assert something
a fail
/[...]/test/system/test_tests.rb:15:in `block in <class:SystemTests>'
/[...]/test/system/test_tests.rb:30:in `block in assert_something'
/[...]/test/system/test_tests.rb:23:in `block in assert_something_else'
assert -t ./test/system/test_tests.rb:13

SKIP: Assert::Test should assert something
TODO
/[...]/test/system/test_tests.rb:15:in `block in <class:SystemTests>'
/[...]/test/system/test_tests.rb:31:in `block in assert_something'
assert -t ./test/system/test_tests.rb:13

FAIL: Assert::Test should assert something
Expected false to be true.
/[...]/test/system/test_tests.rb:14:in `block in <class:SystemTests>'
assert -t ./test/system/test_tests.rb:13

FAIL: Assert::Test should respond to methods
NilClass does not have instance method #an_accessor
Expected nil (NilClass) to respond to `an_accessor`.
/[...]/test/system/test_tests.rb:11:in `<class:SystemTests>'
/[...]/lib/assert/macros/methods.rb:108:in `block (3 levels) in _methods_macro_test'
assert -t ./test/system/test_tests.rb:11

FAIL: Assert::Test should respond to methods
NilClass does not have instance method #an_accessor=
Expected nil (NilClass) to respond to `an_accessor=`.
/[...]/test/system/test_tests.rb:11:in `<class:SystemTests>'
/[...]/lib/assert/macros/methods.rb:108:in `block (3 levels) in _methods_macro_test'
assert -t ./test/system/test_tests.rb:11

[...]

Closes #282.

@jcredding ready for review.