Update to make with Elixir 1.4.0

[sebastian]

There are two class of changes:

• those producing warnings
• those fixing breakage

As of Elixir 1.4.0 warnings are produced when zero-arity functions are invoked without parenthesis (self vs self()). The first commit updates the references to such functions to silence the warnings.

The more important change is the changes done to the ExCheck.Formatter. ExUnit.Formatters should now be GenServer's handling cast's, rather than GenEvent implementations. The ExCheck delegates most message formatting to the built in CLIFormatter, but forwarded to handle_event function which no longer exists. The config format had also slightly changed (tests_counter had become test_counter).

Thank you!

[coveralls]

Coverage decreased (-22.1%) to 53.684% when pulling d706f53bc729095b0b783b9e541cf6edc1e98448 on sebastian:master into b0e9354184ecbc71bcecaef467c7375fe37b0620 on parroty:master.

[sebastian]

The formatter change seems to have reduced the test coverage dramatically, but I can't quite understand why, as it wasn't explicitly tested in the past either.

@parroty, any advice/thoughts/comments?

Thank you!

[luc-tielen]

To me, this sounds like the IO server isn't properly receiving events anymore related to IO calls (such as :io.format) and the functions at the end of the test suite not being called(/counted?) anymore. This is probably due to changes in the way ExUnit works? In one of your commit messages I also saw handle info is not used anymore (in ExUnit.CLIFormatter) which may be 1 of the reasons..? Keep me posted if you're still having problems.

[parroty]

Thanks for the update. This change works nicely for v1.4.0, but for v1.3.4, some tests seems failing and causing the drop in coverage (this coverage report above is executed against older version of elixirs).

It may need some conditional check in GenEvent/GenServer switch based on the elixir version being used, if possible? (I haven't been able to try, and not sure yet).

The following is the result I locally tried with https://github.com/sebastian/excheck.

➜  excheck git:(master) git remote -v
origin  https://github.com/sebastian/excheck (fetch)
origin  https://github.com/sebastian/excheck (push)

➜  excheck git:(master) elixir -v
Erlang/OTP 19 [erts-8.2] [source] [64-bit] [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]

Elixir 1.3.4
➜  excheck git:(master) mix coveralls

=ERROR REPORT====
** gen_event handler 'Elixir.ExCheck.Formatter' crashed.
** Was installed in <0.192.0>
** Last event was: {suite_started,[{seed,72080},
                                   {max_cases,8},
                                   {include,[]},
                                   {exclude,[]},
                                   {case_load_timeout,60000},
                                   {autorun,false},
                                   {assert_receive_timeout,100},
                                   {timeout,60000},
                                   {refute_receive_timeout,100},
                                   {colors,[]},
                                   {capture_log,false},
                                   {included_applications,[]},
                                   {stacktrace_depth,20},
                                   {trace,false},
                                   {formatters,['Elixir.ExCheck.Formatter']}]}
** When handler state == #{colors => [{enabled,true}],
                           failures_counter => 0,
                           invalids_counter => 0,
                           seed => 72080,
                           skipped_counter => 0,
                           tests_counter => #{},
                           trace => false,
                           width => 80}
** Reason == {'function not exported',
                 [{'Elixir.ExCheck.Formatter',handle_event,
                      [{suite_started,
                           [{seed,72080},
                            {max_cases,8},
                            {include,[]},
                            {exclude,[]},
                            {case_load_timeout,60000},
                            {autorun,false},
                            {assert_receive_timeout,100},
                            {timeout,60000},
                            {refute_receive_timeout,100},
                            {colors,[]},
                            {capture_log,false},
                            {included_applications,[]},
                            {stacktrace_depth,20},
                            {trace,false},
                            {formatters,['Elixir.ExCheck.Formatter']}]},
                       #{colors => [{enabled,true}],
                         failures_counter => 0,
                         invalids_counter => 0,
                         seed => 72080,
                         skipped_counter => 0,
                         tests_counter => #{},
                         trace => false,
                         width => 80}],
                      []},
                  {gen_event,server_update,4,
                      [{file,"gen_event.erl"},{line,533}]},
                  {gen_event,server_notify,4,
                      [{file,"gen_event.erl"},{line,515}]},
                  {gen_event,handle_msg,5,[{file,"gen_event.erl"},{line,256}]},
                  {proc_lib,init_p_do_apply,3,
                      [{file,"proc_lib.erl"},{line,247}]}]}
...
----------------
COV    FILE                                        LINES RELEVANT   MISSED
 88.9% lib/excheck.ex                                 58        9        1
  0.0% lib/excheck/error.ex                            7        0        0
 88.9% lib/excheck/error_agent.ex                     37        9        1
  7.1% lib/excheck/formatter.ex                       43       14       13
  0.0% lib/excheck/generator.ex                       31        0        0
 38.3% lib/excheck/io_server.ex                      178       47       29
100.0% lib/excheck/predicate.ex                       84        1        0
100.0% lib/excheck/sample.ex                          32        6        0
100.0% lib/excheck/statement.ex                       53        9        0
[TOTAL]  53.7%
----------------

excheck git:(master) elixir -v
Erlang/OTP 19 [erts-8.2] [source] [64-bit] [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]

Elixir 1.4.0

excheck git:(master) mix coveralls

Finished in 0.7 seconds
2506 tests, 0 failures

Randomized with seed 702547
----------------
COV    FILE                                        LINES RELEVANT   MISSED
 88.9% lib/excheck.ex                                 58        9        1
  0.0% lib/excheck/error.ex                            7        0        0
100.0% lib/excheck/error_agent.ex                     37        9        0
 71.4% lib/excheck/formatter.ex                       43       14        4
  0.0% lib/excheck/generator.ex                       31        0        0
 55.3% lib/excheck/io_server.ex                      178       47       21
100.0% lib/excheck/predicate.ex                       84        1        0
100.0% lib/excheck/sample.ex                          32        6        0
100.0% lib/excheck/statement.ex                       53        9        0
[TOTAL]  72.6%
----------------

[coveralls]

Coverage decreased (-3.2%) to 72.632% when pulling 526e828aade2a0d97d5972a4ce58e1e906aeaad8 on sebastian:master into b0e9354184ecbc71bcecaef467c7375fe37b0620 on parroty:master.

[coveralls]

Coverage increased (+1.2%) to 77.0% when pulling 1a13bb582e6df9fcdec8dd2073f46d46cac255df on sebastian:master into b0e9354184ecbc71bcecaef467c7375fe37b0620 on parroty:master.

[coveralls]

Coverage increased (+1.2%) to 77.0% when pulling 1a13bb582e6df9fcdec8dd2073f46d46cac255df on sebastian:master into b0e9354184ecbc71bcecaef467c7375fe37b0620 on parroty:master.

[sebastian]

Thank you for the help and feedback @parroty and @luc-tielen.

I updated my pull with a version of the formatter that implements a GenEvent formatter for versions of elixir up to 1.4.0, and a GenServer based one from 1.4.0 onwards.

I also updated the travis file so it tests against Elixir versions 1.3.4 and 1.4.0 as well, as well as Erlang 19.1. Feel free to discard these changes since they are nothing we have discussed before.

Thanks again.

[sebastian]

It seems I was a little too eager when adding Erlang 19.1 as a test target. excoveralls is causing issues when used with Erlang 19.1 and Elixir 1.1.1: https://travis-ci.org/parroty/excheck/jobs/189491361

I am happy to do any of the following:

• delete the last commit from the branch
• revert it
• let you take it from here, making the changes you deem appropriate

Please instruct.

Thank you!

[coveralls]

Coverage increased (+1.2%) to 77.0% when pulling c73b879ecb8dcf7bca5556cdd4f29f07e6d7259b on sebastian:master into b0e9354184ecbc71bcecaef467c7375fe37b0620 on parroty:master.

[coveralls]

Coverage increased (+1.2%) to 77.0% when pulling c73b879ecb8dcf7bca5556cdd4f29f07e6d7259b on sebastian:master into b0e9354184ecbc71bcecaef467c7375fe37b0620 on parroty:master.

[coveralls]

Coverage increased (+1.2%) to 77.0% when pulling c73b879ecb8dcf7bca5556cdd4f29f07e6d7259b on sebastian:master into b0e9354184ecbc71bcecaef467c7375fe37b0620 on parroty:master.

[coveralls]

Coverage increased (+1.2%) to 77.0% when pulling c73b879ecb8dcf7bca5556cdd4f29f07e6d7259b on sebastian:master into b0e9354184ecbc71bcecaef467c7375fe37b0620 on parroty:master.

[coveralls]

Coverage increased (+1.2%) to 77.0% when pulling c73b879ecb8dcf7bca5556cdd4f29f07e6d7259b on sebastian:master into b0e9354184ecbc71bcecaef467c7375fe37b0620 on parroty:master.

[coveralls]

Coverage increased (+1.2%) to 77.0% when pulling c73b879ecb8dcf7bca5556cdd4f29f07e6d7259b on sebastian:master into b0e9354184ecbc71bcecaef467c7375fe37b0620 on parroty:master.

[coveralls]

Coverage increased (+1.2%) to 77.0% when pulling c73b879ecb8dcf7bca5556cdd4f29f07e6d7259b on sebastian:master into b0e9354184ecbc71bcecaef467c7375fe37b0620 on parroty:master.

[luc-tielen]

Sebastian, thank you also for putting the time in to do this. :)

I would delete the last commit (the one pulling Erlang 19 in for Elixir 1.1.1). With regards to the formatter.. it's a bit worrying to me since I get the feeling that ExUnit changes their interface a lot (we had another issue between versions only a few months ago). This results in us having to figure out a 'hack' each time just to stay compatible. For now this fix is OK though.

[sebastian]

@luc-tielen, I have now removed the last commit.

it's a bit worrying to me since I get the feeling that ExUnit changes their interface a lot (we had another issue between versions only a few months ago). This results in us having to figure out a 'hack' each time just to stay compatible.

Yes, that's worrying indeed! These types of hacks are rather annoying, feel unclean, and leave somewhat of a bad taste... oh well.

Thanks again and have a great weekend.

[parroty]

Thanks, I could confirm it works on both versions!