Switch testsuite to common_test, depends on rabbitmq/rabbitmq-common#73

[dumbbell]

All tests were moved from test_util.erl and negative_test_util.erl to unit_SUITE.erl and system_SUITE.erl.

unit_SUITE.erl has only unit tests and doesn't need a running broker.

system_SUITE.erl has system integration tests and takes care of starting and setting up a broker itself.

make checks automatically runs common_test tests. All our specific Makefile targets are not used anymore. The new testsuite just needs targets to start/stop a node and set/clear resource alarms.

In system_SUITE.erl, almost all tests are executed with direct and network connections. Therefore, they are listed in two groups:

• direct_connection_tests
• network_connection_tests

To run the whole testsuite:

make tests

To run only one group:

make ct-system t=network_connection_tests

To run one test in a group:

make ct-system t=network_connection_tests:basic_consume

We also use common_test to handle the tests which are repeated 100 times.

Running tests in parallel doesn't work yet because one of the tests triggers all channels to be killed. This is to be investigated.

The started node stores all its data in a subdirectory or the logs directory and not in /tmp. This allows to keep the database and log files for each run of the testsuite.

erlang.mk is updated at the same time: we don't need to disable any Erlang.mk plugins anymore.

Fixes #39.

[dumbbell]

That's the first time I use Common Test. @essen, do you see something wrong in my use of it?

[dumbbell]

FYI, here is the report produce by Common Test on my laptop: http://www.dumbbell.fr/~dumbbell/rabbitmq-erlang-client-common_test/logs/

[essen]

Github doesn't show me the full diff. I'll fetch and look tomorrow morning.

[essen]

A few comments based on the commit message:

• "make checks" I guess you meant "make check" or "make tests".
• Tests are run in parallel inside one group, so if you make the test that kills all channels as part of its own group it will not conflict with other tests.
• "in a subdirectory or the logs" I'm guessing means "of"

Will be followed with comments based on changes in a few minutes.

[essen]

I'd recommend doing -compile(export_all). in the test suites. It's a pain to add everything manually.

I'm not a fan of running eunit in ct. For example in system_SUITE:

+no_permission(Config) ->
+    ok = eunit:test(
+      {timeout, 60,
+       fun() -> no_permission_test(Config) end}).
+
+no_permission_test(Config) ->
+    ?assertMatch({error, not_allowed}, new_connection(Config)).

This doesn't make much sense to me. You could just write it as

no_permission(Config) ->
    {error, not_allowed} = new_connection(Config).

Also the whole unit_SUITE. Perhaps you did it to get nice ct logs. If that's the case I'd recommend making a lot of small ct test cases, one per "assertMatch" you have. Otherwise, you can just make a module in test with the eunit tests and erlang.mk should pick them up on "make eunit".

[dumbbell]

Thanks @essen for your comments!

I fixed the commit message, used export_all and got rid of eunit:test() in both testsuites.

About the latter, I went with this wrapper because all tests were somewhat EUnit-based, but you're right, all the ?assertSomething macros still work without running EUnit. Now, it's way better, thanks. I kept common_test for unit_SUITE.erl so all tests logs are in a central place and we have a single interface to run specific tests.

[dumbbell]

FYI, Travis reports errors because of rabbitmq/rabbitmq-common#73. The testsuite is working locally.

[essen]

Keeping all logs together is a very good reason.

[michaelklishin]

I have test failures on OS X and Linux that do not exist on stable:

=ERROR REPORT==== 30-Mar-2016::00:42:56 ===
** Generic server <0.16751.0> terminating
** Last message in was {'EXIT',#Port<0.11767>,normal}
** When Server state == {state,amqp_network_connection,
                            {state,#Port<0.11767>,
                                <<"client 127.0.0.1:54408 -> 127.0.0.1:5672">>,
                                10,<0.16754.0>,131072,<0.16750.0>,undefined,
                                false},
                            <0.16753.0>,
                            {amqp_params_network,<<"guest">>,<<"guest">>,
                                <<"/">>,"localhost",5672,0,0,10,infinity,none,
                                [#Fun<amqp_auth_mechanisms.plain.3>,
                                 #Fun<amqp_auth_mechanisms.amqplain.3>],
                                [],[]},
                            0,
                            [{<<"capabilities">>,table,
                              [{<<"publisher_confirms">>,bool,true},
                               {<<"exchange_exchange_bindings">>,bool,true},
                               {<<"basic.nack">>,bool,true},
                               {<<"consumer_cancel_notify">>,bool,true},
                               {<<"connection.blocked">>,bool,true},
                               {<<"consumer_priorities">>,bool,true},
                               {<<"authentication_failure_close">>,bool,true},
                               {<<"per_consumer_qos">>,bool,true},
                               {<<"direct_reply_to">>,bool,true}]},
                             {<<"cluster_name">>,longstr,
                              <<"rabbit@localhost">>},
                             {<<"copyright">>,longstr,
                              <<"Copyright (C) 2007-2016 Pivotal Software, Inc.">>},
                             {<<"information">>,longstr,
                              <<"Licensed under the MPL.  See http://www.rabbitmq.com/">>},
                             {<<"platform">>,longstr,<<"Erlang/OTP">>},
                             {<<"product">>,longstr,<<"RabbitMQ">>},
                             {<<"version">>,longstr,<<"0.0.0">>}],
                            none,false}
** Reason for termination ==
** {function_clause,
       [{amqp_network_connection,handle_message,
            [{'EXIT',#Port<0.11767>,normal},
             {state,#Port<0.11767>,
                 <<"client 127.0.0.1:54408 -> 127.0.0.1:5672">>,10,
                 <0.16754.0>,131072,<0.16750.0>,undefined,false}],
            [{file,"src/amqp_network_connection.erl"},{line,60}]},
        {amqp_gen_connection,callback,3,
            [{file,"src/amqp_gen_connection.erl"},{line,143}]},
        {gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,615}]},
        {gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,681}]},
        {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}

=ERROR REPORT==== 30-Mar-2016::00:42:56 ===
** Generic server <0.16773.0> terminating
** Last message in was {call,bogus_message,none,<0.16760.0>}
** When Server state == {state,1,<0.16764.0>,<0.16772.0>,network,
                               {[],[]},
                               false,<0.16774.0>,none,none,0,true,none,
                               {0,nil},
                               {0,nil},
                               true,false}
** Reason for termination ==
** {badarg,[{amqp_channel,is_connection_method,1,
                          [{file,"src/amqp_channel.erl"},{line,907}]},
            {amqp_channel,check_invalid_method,1,
                          [{file,"src/amqp_channel.erl"},{line,900}]},
            {amqp_channel,handle_method_to_server,6,
                          [{file,"src/amqp_channel.erl"},{line,559}]},
            {gen_server,try_handle_call,4,
                        [{file,"gen_server.erl"},{line,629}]},
            {gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,661}]},
            {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}

=WARNING REPORT==== 30-Mar-2016::00:42:56 ===
Connection (<0.16764.0>) closing: internal error in channel (<0.16773.0>): {badarg,
                                                                            [{amqp_channel,
                                                                              is_connection_method,
                                                                              1,
                                                                              [{file,
                                                                                "src/amqp_channel.erl"},
                                                                               {line,
                                                                                907}]},
                                                                             {amqp_channel,
                                                                              check_invalid_method,
                                                                              1,
                                                                              [{file,
                                                                                "src/amqp_channel.erl"},
                                                                               {line,
                                                                                900}]},
                                                                             {amqp_channel,
                                                                              handle_method_to_server,
                                                                              6,
                                                                              [{file,
                                                                                "src/amqp_channel.erl"},
                                                                               {line,
                                                                                559}]},
                                                                             {gen_server,
                                                                              try_handle_call,
                                                                              4,
                                                                              [{file,
                                                                                "gen_server.erl"},
                                                                               {line,
                                                                                629}]},
                                                                             {gen_server,
                                                                              handle_msg,
                                                                              5,
                                                                              [{file,
                                                                                "gen_server.erl"},
                                                                               {line,
                                                                                661}]},
                                                                             {proc_lib,
                                                                              init_p_do_apply,
                                                                              3,
                                                                              [{file,
                                                                                "proc_lib.erl"},
                                                                               {line,
                                                                                240}]}]}

=ERROR REPORT==== 30-Mar-2016::00:42:56 ===
** Generic server <0.16764.0> terminating
** Last message in was {channel_exit,0,method_field_shortstr_overflow}
** When Server state == {state,amqp_network_connection,
                            {state,#Port<0.11769>,
                                <<"client 127.0.0.1:54409 -> 127.0.0.1:5672">>,
                                10,<0.16767.0>,131072,<0.16763.0>,
                                {internal_error,541,
                                    <<"<0.16773.0>:{badarg,[{amqp_channel,is_connection_method,1,\n                                   [{file,\"src/amqp_channel.erl\"},{line,907}]},\n                     {amqp_channel,check_invalid_method,1,\n                                   [{file,\"src/amqp_channel.erl\"},{line,900}]},\n                     {amqp_channel,handle_method_to_server,6,\n                                   [{file,\"src/amqp_channel.erl\"},{line,559}]},\n                     {gen_server,try_handle_call,4,\n                                 [{file,\"gen_server.erl\"},{line,629}]},\n                     {gen_server,handle_msg,5,\n                                 [{file,\"gen_server.erl\"},{line,661}]},\n                     {proc_lib,init_p_do_apply,3,\n                               [{file,\"proc_lib.erl\"},{line,240}]}]}">>},
                                false},
                            <0.16766.0>,
                            {amqp_params_network,<<"guest">>,<<"guest">>,
                                <<"/">>,"localhost",5672,0,0,10,infinity,none,
                                [#Fun<amqp_auth_mechanisms.plain.3>,
                                 #Fun<amqp_auth_mechanisms.amqplain.3>],
                                [],[]},
                            0,
                            [{<<"capabilities">>,table,
                              [{<<"publisher_confirms">>,bool,true},
                               {<<"exchange_exchange_bindings">>,bool,true},
                               {<<"basic.nack">>,bool,true},
                               {<<"consumer_cancel_notify">>,bool,true},
                               {<<"connection.blocked">>,bool,true},
                               {<<"consumer_priorities">>,bool,true},
                               {<<"authentication_failure_close">>,bool,true},
                               {<<"per_consumer_qos">>,bool,true},
                               {<<"direct_reply_to">>,bool,true}]},
                             {<<"cluster_name">>,longstr,
                              <<"rabbit@localhost">>},
                             {<<"copyright">>,longstr,
                              <<"Copyright (C) 2007-2016 Pivotal Software, Inc.">>},
                             {<<"information">>,longstr,
                              <<"Licensed under the MPL.  See http://www.rabbitmq.com/">>},
                             {<<"platform">>,longstr,<<"Erlang/OTP">>},
                             {<<"product">>,longstr,<<"RabbitMQ">>},
                             {<<"version">>,longstr,<<"0.0.0">>}],
                            none,
                            {closing,internal_error,
                                {'connection.close',541,
                                    <<"<0.16773.0>:{badarg,[{amqp_channel,is_connection_method,1,\n                                   [{file,\"src/amqp_channel.erl\"},{line,907}]},\n                     {amqp_channel,check_invalid_method,1,\n                                   [{file,\"src/amqp_channel.erl\"},{line,900}]},\n                     {amqp_channel,handle_method_to_server,6,\n                                   [{file,\"src/amqp_channel.erl\"},{line,559}]},\n                     {gen_server,try_handle_call,4,\n                                 [{file,\"gen_server.erl\"},{line,629}]},\n                     {gen_server,handle_msg,5,\n                                 [{file,\"gen_server.erl\"},{line,661}]},\n                     {proc_lib,init_p_do_apply,3,\n                               [{file,\"proc_lib.erl\"},{line,240}]}]}">>,
                                    0,0},
                                none}}
** Reason for termination ==
** {channel0_died,method_field_shortstr_overflow}

----------------------------------------------------
2016-03-30 00:42:57.056
+ (/Users/antares/Tools/rabbitmq/generic/sbin/rabbitmqctl delete_user test_user_no_perm) && echo 'COMMAND SUCCESSFUL'
Deleting user "test_user_no_perm" ...
COMMAND SUCCESSFUL

----------------------------------------------------
2016-03-30 00:42:58.284
+ (/Applications/Xcode.app/Contents/Developer/usr/bin/make -C /Users/antares/Development/RabbitMQ/stable_umbrella.git/deps/amqp_client stop-rabbit-on-node stop-node TEST_TMPDIR='/Users/antares/Development/RabbitMQ/stable_umbrella.git/deps/amqp_client/logs/ct_run.ct@urano.2016-03-30_00.41.42/deps.amqp_client.system_SUITE.logs/run.2016-03-30_00.41.42/log_private/') && echo 'COMMAND SUCCESSFUL'
 EXEC   stop-rabbit-on-node
 EXEC   stop-node
COMMAND SUCCESSFUL

Testing deps.amqp_client.system_SUITE: TEST COMPLETE, 648 ok, 39 failed of 687 test cases

Testing deps.amqp_client.unit_SUITE: Starting test, 2 test cases
Testing deps.amqp_client.unit_SUITE: TEST COMPLETE, 2 ok, 0 failed of 2 test cases

[dumbbell]

The 39 failed tests are because of rabbitmq/rabbitmq-common#73. Those tests are near the beginning of the output. Because test cases are executed in parallel, they all try to load time_compat and 39 of them fail.

The error messages you show here are those from test cases expecting a failure. error_logger messages are printed to stdout; I don't know yet if common_test has a knob to change that.

Errors from common_test look like this:

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
system_SUITE:simultaneous_close failed on line 240
Reason: {badmatch,{ok,<0.301.0>}}
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

(...)

Testing deps.amqp_client.system_SUITE: *** FAILED test case 4 ***

(...)

Testing deps.amqp_client.system_SUITE: TEST COMPLETE, 685 ok, 2 failed of 687 test cases

[dumbbell]

@michaelklishin: Could you please confirm that, if you checkout rabbitmq-common-73 in rabbit_common, then the testsuite passes?

[michaelklishin]

@dumbbell with rabbit_common checked out to rabbitmq-common-73, I'm getting

=ERROR REPORT==== 30-Mar-2016::23:23:17 ===
** Generic server <0.19970.0> terminating
** Last message in was {'EXIT',#Port<0.11741>,normal}
** When Server state == {state,amqp_network_connection,
                            {state,#Port<0.11741>,
                                <<"client 127.0.0.1:63371 -> 127.0.0.1:5672">>,
                                10,<0.19973.0>,131072,<0.19969.0>,undefined,
                                false},
                            <0.19972.0>,
                            {amqp_params_network,<<"guest">>,<<"guest">>,
                                <<"/">>,"localhost",5672,0,0,10,infinity,none,
                                [#Fun<amqp_auth_mechanisms.plain.3>,
                                 #Fun<amqp_auth_mechanisms.amqplain.3>],
                                [],[]},
                            0,
                            [{<<"capabilities">>,table,
                              [{<<"publisher_confirms">>,bool,true},
                               {<<"exchange_exchange_bindings">>,bool,true},
                               {<<"basic.nack">>,bool,true},
                               {<<"consumer_cancel_notify">>,bool,true},
                               {<<"connection.blocked">>,bool,true},
                               {<<"consumer_priorities">>,bool,true},
                               {<<"authentication_failure_close">>,bool,true},
                               {<<"per_consumer_qos">>,bool,true},
                               {<<"direct_reply_to">>,bool,true}]},
                             {<<"cluster_name">>,longstr,
                              <<"rabbit@localhost">>},
                             {<<"copyright">>,longstr,
                              <<"Copyright (C) 2007-2016 Pivotal Software, Inc.">>},
                             {<<"information">>,longstr,
                              <<"Licensed under the MPL.  See http://www.rabbitmq.com/">>},
                             {<<"platform">>,longstr,<<"Erlang/OTP">>},
                             {<<"product">>,longstr,<<"RabbitMQ">>},
                             {<<"version">>,longstr,<<"0.0.0">>}],
                            none,false}
** Reason for termination ==
** {function_clause,
       [{amqp_network_connection,handle_message,
            [{'EXIT',#Port<0.11741>,normal},
             {state,#Port<0.11741>,
                 <<"client 127.0.0.1:63371 -> 127.0.0.1:5672">>,10,
                 <0.19973.0>,131072,<0.19969.0>,undefined,false}],
            [{file,"src/amqp_network_connection.erl"},{line,60}]},
        {amqp_gen_connection,callback,3,
            [{file,"src/amqp_gen_connection.erl"},{line,143}]},
        {gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,615}]},
        {gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,681}]},
        {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}

=ERROR REPORT==== 30-Mar-2016::23:23:17 ===
** Generic server <0.19992.0> terminating
** Last message in was {call,bogus_message,none,<0.19979.0>}
** When Server state == {state,1,<0.19983.0>,<0.19991.0>,network,
                               {[],[]},
                               false,<0.19993.0>,none,none,0,true,none,
                               {0,nil},
                               {0,nil},
                               true,false}
** Reason for termination ==
** {badarg,[{amqp_channel,is_connection_method,1,
                          [{file,"src/amqp_channel.erl"},{line,907}]},
            {amqp_channel,check_invalid_method,1,
                          [{file,"src/amqp_channel.erl"},{line,900}]},
            {amqp_channel,handle_method_to_server,6,
                          [{file,"src/amqp_channel.erl"},{line,559}]},
            {gen_server,try_handle_call,4,
                        [{file,"gen_server.erl"},{line,629}]},
            {gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,661}]},
            {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}

=WARNING REPORT==== 30-Mar-2016::23:23:17 ===
Connection (<0.19983.0>) closing: internal error in channel (<0.19992.0>): {badarg,
                                                                            [{amqp_channel,
                                                                              is_connection_method,
                                                                              1,
                                                                              [{file,
                                                                                "src/amqp_channel.erl"},
                                                                               {line,
                                                                                907}]},
                                                                             {amqp_channel,
                                                                              check_invalid_method,
                                                                              1,
                                                                              [{file,
                                                                                "src/amqp_channel.erl"},
                                                                               {line,
                                                                                900}]},
                                                                             {amqp_channel,
                                                                              handle_method_to_server,
                                                                              6,
                                                                              [{file,
                                                                                "src/amqp_channel.erl"},
                                                                               {line,
                                                                                559}]},
                                                                             {gen_server,
                                                                              try_handle_call,
                                                                              4,
                                                                              [{file,
                                                                                "gen_server.erl"},
                                                                               {line,
                                                                                629}]},
                                                                             {gen_server,
                                                                              handle_msg,
                                                                              5,
                                                                              [{file,
                                                                                "gen_server.erl"},
                                                                               {line,
                                                                                661}]},
                                                                             {proc_lib,
                                                                              init_p_do_apply,
                                                                              3,
                                                                              [{file,
                                                                                "proc_lib.erl"},
                                                                               {line,
                                                                                240}]}]}

=ERROR REPORT==== 30-Mar-2016::23:23:17 ===
** Generic server <0.19983.0> terminating
** Last message in was {channel_exit,0,method_field_shortstr_overflow}
** When Server state == {state,amqp_network_connection,
                            {state,#Port<0.11743>,
                                <<"client 127.0.0.1:63372 -> 127.0.0.1:5672">>,
                                10,<0.19986.0>,131072,<0.19982.0>,
                                {internal_error,541,
                                    <<"<0.19992.0>:{badarg,[{amqp_channel,is_connection_method,1,\n                                   [{file,\"src/amqp_channel.erl\"},{line,907}]},\n                     {amqp_channel,check_invalid_method,1,\n                                   [{file,\"src/amqp_channel.erl\"},{line,900}]},\n                     {amqp_channel,handle_method_to_server,6,\n                                   [{file,\"src/amqp_channel.erl\"},{line,559}]},\n                     {gen_server,try_handle_call,4,\n                                 [{file,\"gen_server.erl\"},{line,629}]},\n                     {gen_server,handle_msg,5,\n                                 [{file,\"gen_server.erl\"},{line,661}]},\n                     {proc_lib,init_p_do_apply,3,\n                               [{file,\"proc_lib.erl\"},{line,240}]}]}">>},
                                false},
                            <0.19985.0>,
                            {amqp_params_network,<<"guest">>,<<"guest">>,
                                <<"/">>,"localhost",5672,0,0,10,infinity,none,
                                [#Fun<amqp_auth_mechanisms.plain.3>,
                                 #Fun<amqp_auth_mechanisms.amqplain.3>],
                                [],[]},
                            0,
                            [{<<"capabilities">>,table,
                              [{<<"publisher_confirms">>,bool,true},
                               {<<"exchange_exchange_bindings">>,bool,true},
                               {<<"basic.nack">>,bool,true},
                               {<<"consumer_cancel_notify">>,bool,true},
                               {<<"connection.blocked">>,bool,true},
                               {<<"consumer_priorities">>,bool,true},
                               {<<"authentication_failure_close">>,bool,true},
                               {<<"per_consumer_qos">>,bool,true},
                               {<<"direct_reply_to">>,bool,true}]},
                             {<<"cluster_name">>,longstr,
                              <<"rabbit@localhost">>},
                             {<<"copyright">>,longstr,
                              <<"Copyright (C) 2007-2016 Pivotal Software, Inc.">>},
                             {<<"information">>,longstr,
                              <<"Licensed under the MPL.  See http://www.rabbitmq.com/">>},
                             {<<"platform">>,longstr,<<"Erlang/OTP">>},
                             {<<"product">>,longstr,<<"RabbitMQ">>},
                             {<<"version">>,longstr,<<"0.0.0">>}],
                            none,
                            {closing,internal_error,
                                {'connection.close',541,
                                    <<"<0.19992.0>:{badarg,[{amqp_channel,is_connection_method,1,\n                                   [{file,\"src/amqp_channel.erl\"},{line,907}]},\n                     {amqp_channel,check_invalid_method,1,\n                                   [{file,\"src/amqp_channel.erl\"},{line,900}]},\n                     {amqp_channel,handle_method_to_server,6,\n                                   [{file,\"src/amqp_channel.erl\"},{line,559}]},\n                     {gen_server,try_handle_call,4,\n                                 [{file,\"gen_server.erl\"},{line,629}]},\n                     {gen_server,handle_msg,5,\n                                 [{file,\"gen_server.erl\"},{line,661}]},\n                     {proc_lib,init_p_do_apply,3,\n                               [{file,\"proc_lib.erl\"},{line,240}]}]}">>,
                                    0,0},
                                none}}
** Reason for termination ==
** {channel0_died,method_field_shortstr_overflow}

----------------------------------------------------
2016-03-30 23:23:18.462
+ ("/Users/antares/Tools/rabbitmq/generic/sbin/rabbitmqctl" delete_user test_user_no_perm) && echo COMMAND SUCCESSFUL
Deleting user "test_user_no_perm" ...
COMMAND SUCCESSFUL

----------------------------------------------------
2016-03-30 23:23:19.674
+ ("/Applications/Xcode.app/Contents/Developer/usr/bin/make" -C /Users/antares/Development/RabbitMQ/stable_umbrella.git/deps/amqp_client stop-rabbit-on-node stop-node TEST_TMPDIR='/Users/antares/Development/RabbitMQ/stable_umbrella.git/deps/amqp_client/logs/ct_run.ct@urano.2016-03-30_23.22.09/deps.amqp_client.system_SUITE.logs/run.2016-03-30_23.22.09/log_private/') && echo COMMAND SUCCESSFUL
 EXEC   stop-rabbit-on-node
 EXEC   stop-node
COMMAND SUCCESSFUL

Testing deps.amqp_client.system_SUITE: TEST COMPLETE, 687 ok, 0 failed of 687 test cases

Testing deps.amqp_client.unit_SUITE: Starting test, 2 test cases
Testing deps.amqp_client.unit_SUITE: TEST COMPLETE, 2 ok, 0 failed of 2 test cases

[dumbbell]

So the tests pass, cool :-) Thank you!

[michaelklishin]

@dumbbell yeah but it's hard to tell from all the SASL reports noise (some of which simply reveals expected failures) :(

[dumbbell]

Yes, the output is very noisy...

The important line is at the end:

Testing deps.amqp_client.system_SUITE: TEST COMPLETE, 687 ok, 0 failed of 687 test cases
                                                              ^^^^^^^^

[essen]

In Cowboy I have code that hides all the expected failures, maybe this could apply here also: https://github.com/ninenines/ct_helper/blob/master/src/ct_helper_error_h.erl

I don't think I have updated it for 18+ yet (there was some changes in how errors are presented to error_handler).