Failure in `assert_compile_fail` crashes test suite

[oneeman]

Any test using assert_compile_fail which should fail instead crashes the test suite.

The stacktrace is along the lines of:

............................................
08:20:34.828 [error] GenEvent handler ExUnit.CLIFormatter installed in #PID<0.864.0> terminating
Last message: {:test_finished, %ExUnit.Test{case: TestHelperTest, name: :"test assert_compile_fail_test", state: {:failed, {:error, %ExUnit.AssertionError{expr: :ex_unit_no_meaningful_value, left: :ex_unit_no_meaningful_value, message: [message: "Expected expression to fail"], right: :ex_unit_no_meaningful_value}, [{CompileAssertion, :assert_compile_fail, 2, [file: 'test/elixir/test_helper.exs', line: 74]}, {TestHelperTest, :"test assert_compile_fail_test", 1, [file: 'test/elixir/test_helper_test.exs', line: 8]}]}}, tags: %{file: "/home/or/oss/elixir/elixir/lib/elixir/test/elixir/test_helper_test.exs", line: 7, test: :"test assert_compile_fail_test"}, time: 133}}
State: %{colors: [enabled: true], failures_counter: 0, invalids_counter: 0, seed: 383820, skipped_counter: 0, tests_counter: 604, trace: false, width: 80}
** (exit) an exception was raised:
    ** (ArgumentError) argument error
        (stdlib) binary.erl:317: :binary.replace/4
        (ex_unit) lib/ex_unit/formatter.ex:174: ExUnit.Formatter.format_banner/2
        (ex_unit) lib/ex_unit/formatter.ex:130: ExUnit.Formatter.format_kind_reason/4
        (ex_unit) lib/ex_unit/formatter.ex:112: ExUnit.Formatter.format_test_failure/5
        (ex_unit) lib/ex_unit/cli_formatter.ex:67: ExUnit.CLIFormatter.handle_event/2
        (stdlib) gen_event.erl:525: :gen_event.server_update/4
        (stdlib) gen_event.erl:507: :gen_event.server_notify/4
        (stdlib) gen_event.erl:248: :gen_event.handle_msg/5
make: *** [test_stdlib] Error 1

The cause of the problem is in format_rescue in test_helper.exs, line 103:

result || flunk(message: "Expected expression to fail") should be result || flunk("Expected expression to fail")

I can do a PR, but wasn't sure whether we should also add some tests for this (test_helper_test.exs?) and what the test strategy would be. I can look into it further.

[josevalim]

We should add a guard clause to flunk that asserts a binary is given and just fix the assertion directly! Good catch btw!

José Valim www.plataformatec.com.br Skype: jv.ptec Founder and Lead Developer

[oneeman]

Right!

Adding the guard, the error no longer crashes the suite and instead points in the right direction:

  1) test demo_flunk_error (Kernel.ErrorsTest)
     test/elixir/kernel/errors_test.exs:86
     ** (FunctionClauseError) no function clause matching in ExUnit.Assertions.flunk/1
     stacktrace:
       test/elixir/test_helper.exs:74: CompileAssertion.assert_compile_fail/2
       test/elixir/kernel/errors_test.exs:87

(added :demo_flunk_error test just to try it out, not keeping it)

Just to clarify: when you say "just fix the assertion directly" you mean the flunk call in format_rescue that was causing the trouble?

[josevalim]

Just to clarify: when you say "just fix the assertion directly" you mean the flunk call in format_rescue that was causing the trouble?

Yes. :)