Closed dumbbell closed 8 years ago
That's the first time I use Common Test. @essen, do you see something wrong in my use of it?
FYI, here is the report produce by Common Test on my laptop: http://www.dumbbell.fr/~dumbbell/rabbitmq-erlang-client-common_test/logs/
Github doesn't show me the full diff. I'll fetch and look tomorrow morning.
A few comments based on the commit message:
logs
" I'm guessing means "of"Will be followed with comments based on changes in a few minutes.
I'd recommend doing -compile(export_all).
in the test suites. It's a pain to add everything manually.
I'm not a fan of running eunit in ct. For example in system_SUITE:
+no_permission(Config) ->
+ ok = eunit:test(
+ {timeout, 60,
+ fun() -> no_permission_test(Config) end}).
+
+no_permission_test(Config) ->
+ ?assertMatch({error, not_allowed}, new_connection(Config)).
This doesn't make much sense to me. You could just write it as
no_permission(Config) ->
{error, not_allowed} = new_connection(Config).
Also the whole unit_SUITE. Perhaps you did it to get nice ct logs. If that's the case I'd recommend making a lot of small ct test cases, one per "assertMatch" you have. Otherwise, you can just make a module in test with the eunit tests and erlang.mk should pick them up on "make eunit".
Thanks @essen for your comments!
I fixed the commit message, used export_all
and got rid of eunit:test()
in both testsuites.
About the latter, I went with this wrapper because all tests were somewhat EUnit-based, but you're right, all the ?assertSomething
macros still work without running EUnit. Now, it's way better, thanks. I kept common_test for unit_SUITE.erl
so all tests logs are in a central place and we have a single interface to run specific tests.
FYI, Travis reports errors because of rabbitmq/rabbitmq-common#73. The testsuite is working locally.
Keeping all logs together is a very good reason.
I have test failures on OS X and Linux that do not exist on stable:
=ERROR REPORT==== 30-Mar-2016::00:42:56 ===
** Generic server <0.16751.0> terminating
** Last message in was {'EXIT',#Port<0.11767>,normal}
** When Server state == {state,amqp_network_connection,
{state,#Port<0.11767>,
<<"client 127.0.0.1:54408 -> 127.0.0.1:5672">>,
10,<0.16754.0>,131072,<0.16750.0>,undefined,
false},
<0.16753.0>,
{amqp_params_network,<<"guest">>,<<"guest">>,
<<"/">>,"localhost",5672,0,0,10,infinity,none,
[#Fun<amqp_auth_mechanisms.plain.3>,
#Fun<amqp_auth_mechanisms.amqplain.3>],
[],[]},
0,
[{<<"capabilities">>,table,
[{<<"publisher_confirms">>,bool,true},
{<<"exchange_exchange_bindings">>,bool,true},
{<<"basic.nack">>,bool,true},
{<<"consumer_cancel_notify">>,bool,true},
{<<"connection.blocked">>,bool,true},
{<<"consumer_priorities">>,bool,true},
{<<"authentication_failure_close">>,bool,true},
{<<"per_consumer_qos">>,bool,true},
{<<"direct_reply_to">>,bool,true}]},
{<<"cluster_name">>,longstr,
<<"rabbit@localhost">>},
{<<"copyright">>,longstr,
<<"Copyright (C) 2007-2016 Pivotal Software, Inc.">>},
{<<"information">>,longstr,
<<"Licensed under the MPL. See http://www.rabbitmq.com/">>},
{<<"platform">>,longstr,<<"Erlang/OTP">>},
{<<"product">>,longstr,<<"RabbitMQ">>},
{<<"version">>,longstr,<<"0.0.0">>}],
none,false}
** Reason for termination ==
** {function_clause,
[{amqp_network_connection,handle_message,
[{'EXIT',#Port<0.11767>,normal},
{state,#Port<0.11767>,
<<"client 127.0.0.1:54408 -> 127.0.0.1:5672">>,10,
<0.16754.0>,131072,<0.16750.0>,undefined,false}],
[{file,"src/amqp_network_connection.erl"},{line,60}]},
{amqp_gen_connection,callback,3,
[{file,"src/amqp_gen_connection.erl"},{line,143}]},
{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,615}]},
{gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,681}]},
{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}
=ERROR REPORT==== 30-Mar-2016::00:42:56 ===
** Generic server <0.16773.0> terminating
** Last message in was {call,bogus_message,none,<0.16760.0>}
** When Server state == {state,1,<0.16764.0>,<0.16772.0>,network,
{[],[]},
false,<0.16774.0>,none,none,0,true,none,
{0,nil},
{0,nil},
true,false}
** Reason for termination ==
** {badarg,[{amqp_channel,is_connection_method,1,
[{file,"src/amqp_channel.erl"},{line,907}]},
{amqp_channel,check_invalid_method,1,
[{file,"src/amqp_channel.erl"},{line,900}]},
{amqp_channel,handle_method_to_server,6,
[{file,"src/amqp_channel.erl"},{line,559}]},
{gen_server,try_handle_call,4,
[{file,"gen_server.erl"},{line,629}]},
{gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,661}]},
{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}
=WARNING REPORT==== 30-Mar-2016::00:42:56 ===
Connection (<0.16764.0>) closing: internal error in channel (<0.16773.0>): {badarg,
[{amqp_channel,
is_connection_method,
1,
[{file,
"src/amqp_channel.erl"},
{line,
907}]},
{amqp_channel,
check_invalid_method,
1,
[{file,
"src/amqp_channel.erl"},
{line,
900}]},
{amqp_channel,
handle_method_to_server,
6,
[{file,
"src/amqp_channel.erl"},
{line,
559}]},
{gen_server,
try_handle_call,
4,
[{file,
"gen_server.erl"},
{line,
629}]},
{gen_server,
handle_msg,
5,
[{file,
"gen_server.erl"},
{line,
661}]},
{proc_lib,
init_p_do_apply,
3,
[{file,
"proc_lib.erl"},
{line,
240}]}]}
=ERROR REPORT==== 30-Mar-2016::00:42:56 ===
** Generic server <0.16764.0> terminating
** Last message in was {channel_exit,0,method_field_shortstr_overflow}
** When Server state == {state,amqp_network_connection,
{state,#Port<0.11769>,
<<"client 127.0.0.1:54409 -> 127.0.0.1:5672">>,
10,<0.16767.0>,131072,<0.16763.0>,
{internal_error,541,
<<"<0.16773.0>:{badarg,[{amqp_channel,is_connection_method,1,\n [{file,\"src/amqp_channel.erl\"},{line,907}]},\n {amqp_channel,check_invalid_method,1,\n [{file,\"src/amqp_channel.erl\"},{line,900}]},\n {amqp_channel,handle_method_to_server,6,\n [{file,\"src/amqp_channel.erl\"},{line,559}]},\n {gen_server,try_handle_call,4,\n [{file,\"gen_server.erl\"},{line,629}]},\n {gen_server,handle_msg,5,\n [{file,\"gen_server.erl\"},{line,661}]},\n {proc_lib,init_p_do_apply,3,\n [{file,\"proc_lib.erl\"},{line,240}]}]}">>},
false},
<0.16766.0>,
{amqp_params_network,<<"guest">>,<<"guest">>,
<<"/">>,"localhost",5672,0,0,10,infinity,none,
[#Fun<amqp_auth_mechanisms.plain.3>,
#Fun<amqp_auth_mechanisms.amqplain.3>],
[],[]},
0,
[{<<"capabilities">>,table,
[{<<"publisher_confirms">>,bool,true},
{<<"exchange_exchange_bindings">>,bool,true},
{<<"basic.nack">>,bool,true},
{<<"consumer_cancel_notify">>,bool,true},
{<<"connection.blocked">>,bool,true},
{<<"consumer_priorities">>,bool,true},
{<<"authentication_failure_close">>,bool,true},
{<<"per_consumer_qos">>,bool,true},
{<<"direct_reply_to">>,bool,true}]},
{<<"cluster_name">>,longstr,
<<"rabbit@localhost">>},
{<<"copyright">>,longstr,
<<"Copyright (C) 2007-2016 Pivotal Software, Inc.">>},
{<<"information">>,longstr,
<<"Licensed under the MPL. See http://www.rabbitmq.com/">>},
{<<"platform">>,longstr,<<"Erlang/OTP">>},
{<<"product">>,longstr,<<"RabbitMQ">>},
{<<"version">>,longstr,<<"0.0.0">>}],
none,
{closing,internal_error,
{'connection.close',541,
<<"<0.16773.0>:{badarg,[{amqp_channel,is_connection_method,1,\n [{file,\"src/amqp_channel.erl\"},{line,907}]},\n {amqp_channel,check_invalid_method,1,\n [{file,\"src/amqp_channel.erl\"},{line,900}]},\n {amqp_channel,handle_method_to_server,6,\n [{file,\"src/amqp_channel.erl\"},{line,559}]},\n {gen_server,try_handle_call,4,\n [{file,\"gen_server.erl\"},{line,629}]},\n {gen_server,handle_msg,5,\n [{file,\"gen_server.erl\"},{line,661}]},\n {proc_lib,init_p_do_apply,3,\n [{file,\"proc_lib.erl\"},{line,240}]}]}">>,
0,0},
none}}
** Reason for termination ==
** {channel0_died,method_field_shortstr_overflow}
----------------------------------------------------
2016-03-30 00:42:57.056
+ (/Users/antares/Tools/rabbitmq/generic/sbin/rabbitmqctl delete_user test_user_no_perm) && echo 'COMMAND SUCCESSFUL'
Deleting user "test_user_no_perm" ...
COMMAND SUCCESSFUL
----------------------------------------------------
2016-03-30 00:42:58.284
+ (/Applications/Xcode.app/Contents/Developer/usr/bin/make -C /Users/antares/Development/RabbitMQ/stable_umbrella.git/deps/amqp_client stop-rabbit-on-node stop-node TEST_TMPDIR='/Users/antares/Development/RabbitMQ/stable_umbrella.git/deps/amqp_client/logs/ct_run.ct@urano.2016-03-30_00.41.42/deps.amqp_client.system_SUITE.logs/run.2016-03-30_00.41.42/log_private/') && echo 'COMMAND SUCCESSFUL'
EXEC stop-rabbit-on-node
EXEC stop-node
COMMAND SUCCESSFUL
Testing deps.amqp_client.system_SUITE: TEST COMPLETE, 648 ok, 39 failed of 687 test cases
Testing deps.amqp_client.unit_SUITE: Starting test, 2 test cases
Testing deps.amqp_client.unit_SUITE: TEST COMPLETE, 2 ok, 0 failed of 2 test cases
The 39 failed tests are because of rabbitmq/rabbitmq-common#73. Those tests are near the beginning of the output. Because test cases are executed in parallel, they all try to load time_compat
and 39 of them fail.
The error messages you show here are those from test cases expecting a failure. error_logger
messages are printed to stdout; I don't know yet if common_test has a knob to change that.
Errors from common_test look like this:
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
system_SUITE:simultaneous_close failed on line 240
Reason: {badmatch,{ok,<0.301.0>}}
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(...)
Testing deps.amqp_client.system_SUITE: *** FAILED test case 4 ***
(...)
Testing deps.amqp_client.system_SUITE: TEST COMPLETE, 685 ok, 2 failed of 687 test cases
@michaelklishin: Could you please confirm that, if you checkout rabbitmq-common-73
in rabbit_common
, then the testsuite passes?
@dumbbell with rabbit_common checked out to rabbitmq-common-73
, I'm getting
=ERROR REPORT==== 30-Mar-2016::23:23:17 ===
** Generic server <0.19970.0> terminating
** Last message in was {'EXIT',#Port<0.11741>,normal}
** When Server state == {state,amqp_network_connection,
{state,#Port<0.11741>,
<<"client 127.0.0.1:63371 -> 127.0.0.1:5672">>,
10,<0.19973.0>,131072,<0.19969.0>,undefined,
false},
<0.19972.0>,
{amqp_params_network,<<"guest">>,<<"guest">>,
<<"/">>,"localhost",5672,0,0,10,infinity,none,
[#Fun<amqp_auth_mechanisms.plain.3>,
#Fun<amqp_auth_mechanisms.amqplain.3>],
[],[]},
0,
[{<<"capabilities">>,table,
[{<<"publisher_confirms">>,bool,true},
{<<"exchange_exchange_bindings">>,bool,true},
{<<"basic.nack">>,bool,true},
{<<"consumer_cancel_notify">>,bool,true},
{<<"connection.blocked">>,bool,true},
{<<"consumer_priorities">>,bool,true},
{<<"authentication_failure_close">>,bool,true},
{<<"per_consumer_qos">>,bool,true},
{<<"direct_reply_to">>,bool,true}]},
{<<"cluster_name">>,longstr,
<<"rabbit@localhost">>},
{<<"copyright">>,longstr,
<<"Copyright (C) 2007-2016 Pivotal Software, Inc.">>},
{<<"information">>,longstr,
<<"Licensed under the MPL. See http://www.rabbitmq.com/">>},
{<<"platform">>,longstr,<<"Erlang/OTP">>},
{<<"product">>,longstr,<<"RabbitMQ">>},
{<<"version">>,longstr,<<"0.0.0">>}],
none,false}
** Reason for termination ==
** {function_clause,
[{amqp_network_connection,handle_message,
[{'EXIT',#Port<0.11741>,normal},
{state,#Port<0.11741>,
<<"client 127.0.0.1:63371 -> 127.0.0.1:5672">>,10,
<0.19973.0>,131072,<0.19969.0>,undefined,false}],
[{file,"src/amqp_network_connection.erl"},{line,60}]},
{amqp_gen_connection,callback,3,
[{file,"src/amqp_gen_connection.erl"},{line,143}]},
{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,615}]},
{gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,681}]},
{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}
=ERROR REPORT==== 30-Mar-2016::23:23:17 ===
** Generic server <0.19992.0> terminating
** Last message in was {call,bogus_message,none,<0.19979.0>}
** When Server state == {state,1,<0.19983.0>,<0.19991.0>,network,
{[],[]},
false,<0.19993.0>,none,none,0,true,none,
{0,nil},
{0,nil},
true,false}
** Reason for termination ==
** {badarg,[{amqp_channel,is_connection_method,1,
[{file,"src/amqp_channel.erl"},{line,907}]},
{amqp_channel,check_invalid_method,1,
[{file,"src/amqp_channel.erl"},{line,900}]},
{amqp_channel,handle_method_to_server,6,
[{file,"src/amqp_channel.erl"},{line,559}]},
{gen_server,try_handle_call,4,
[{file,"gen_server.erl"},{line,629}]},
{gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,661}]},
{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}
=WARNING REPORT==== 30-Mar-2016::23:23:17 ===
Connection (<0.19983.0>) closing: internal error in channel (<0.19992.0>): {badarg,
[{amqp_channel,
is_connection_method,
1,
[{file,
"src/amqp_channel.erl"},
{line,
907}]},
{amqp_channel,
check_invalid_method,
1,
[{file,
"src/amqp_channel.erl"},
{line,
900}]},
{amqp_channel,
handle_method_to_server,
6,
[{file,
"src/amqp_channel.erl"},
{line,
559}]},
{gen_server,
try_handle_call,
4,
[{file,
"gen_server.erl"},
{line,
629}]},
{gen_server,
handle_msg,
5,
[{file,
"gen_server.erl"},
{line,
661}]},
{proc_lib,
init_p_do_apply,
3,
[{file,
"proc_lib.erl"},
{line,
240}]}]}
=ERROR REPORT==== 30-Mar-2016::23:23:17 ===
** Generic server <0.19983.0> terminating
** Last message in was {channel_exit,0,method_field_shortstr_overflow}
** When Server state == {state,amqp_network_connection,
{state,#Port<0.11743>,
<<"client 127.0.0.1:63372 -> 127.0.0.1:5672">>,
10,<0.19986.0>,131072,<0.19982.0>,
{internal_error,541,
<<"<0.19992.0>:{badarg,[{amqp_channel,is_connection_method,1,\n [{file,\"src/amqp_channel.erl\"},{line,907}]},\n {amqp_channel,check_invalid_method,1,\n [{file,\"src/amqp_channel.erl\"},{line,900}]},\n {amqp_channel,handle_method_to_server,6,\n [{file,\"src/amqp_channel.erl\"},{line,559}]},\n {gen_server,try_handle_call,4,\n [{file,\"gen_server.erl\"},{line,629}]},\n {gen_server,handle_msg,5,\n [{file,\"gen_server.erl\"},{line,661}]},\n {proc_lib,init_p_do_apply,3,\n [{file,\"proc_lib.erl\"},{line,240}]}]}">>},
false},
<0.19985.0>,
{amqp_params_network,<<"guest">>,<<"guest">>,
<<"/">>,"localhost",5672,0,0,10,infinity,none,
[#Fun<amqp_auth_mechanisms.plain.3>,
#Fun<amqp_auth_mechanisms.amqplain.3>],
[],[]},
0,
[{<<"capabilities">>,table,
[{<<"publisher_confirms">>,bool,true},
{<<"exchange_exchange_bindings">>,bool,true},
{<<"basic.nack">>,bool,true},
{<<"consumer_cancel_notify">>,bool,true},
{<<"connection.blocked">>,bool,true},
{<<"consumer_priorities">>,bool,true},
{<<"authentication_failure_close">>,bool,true},
{<<"per_consumer_qos">>,bool,true},
{<<"direct_reply_to">>,bool,true}]},
{<<"cluster_name">>,longstr,
<<"rabbit@localhost">>},
{<<"copyright">>,longstr,
<<"Copyright (C) 2007-2016 Pivotal Software, Inc.">>},
{<<"information">>,longstr,
<<"Licensed under the MPL. See http://www.rabbitmq.com/">>},
{<<"platform">>,longstr,<<"Erlang/OTP">>},
{<<"product">>,longstr,<<"RabbitMQ">>},
{<<"version">>,longstr,<<"0.0.0">>}],
none,
{closing,internal_error,
{'connection.close',541,
<<"<0.19992.0>:{badarg,[{amqp_channel,is_connection_method,1,\n [{file,\"src/amqp_channel.erl\"},{line,907}]},\n {amqp_channel,check_invalid_method,1,\n [{file,\"src/amqp_channel.erl\"},{line,900}]},\n {amqp_channel,handle_method_to_server,6,\n [{file,\"src/amqp_channel.erl\"},{line,559}]},\n {gen_server,try_handle_call,4,\n [{file,\"gen_server.erl\"},{line,629}]},\n {gen_server,handle_msg,5,\n [{file,\"gen_server.erl\"},{line,661}]},\n {proc_lib,init_p_do_apply,3,\n [{file,\"proc_lib.erl\"},{line,240}]}]}">>,
0,0},
none}}
** Reason for termination ==
** {channel0_died,method_field_shortstr_overflow}
----------------------------------------------------
2016-03-30 23:23:18.462
+ ("/Users/antares/Tools/rabbitmq/generic/sbin/rabbitmqctl" delete_user test_user_no_perm) && echo COMMAND SUCCESSFUL
Deleting user "test_user_no_perm" ...
COMMAND SUCCESSFUL
----------------------------------------------------
2016-03-30 23:23:19.674
+ ("/Applications/Xcode.app/Contents/Developer/usr/bin/make" -C /Users/antares/Development/RabbitMQ/stable_umbrella.git/deps/amqp_client stop-rabbit-on-node stop-node TEST_TMPDIR='/Users/antares/Development/RabbitMQ/stable_umbrella.git/deps/amqp_client/logs/ct_run.ct@urano.2016-03-30_23.22.09/deps.amqp_client.system_SUITE.logs/run.2016-03-30_23.22.09/log_private/') && echo COMMAND SUCCESSFUL
EXEC stop-rabbit-on-node
EXEC stop-node
COMMAND SUCCESSFUL
Testing deps.amqp_client.system_SUITE: TEST COMPLETE, 687 ok, 0 failed of 687 test cases
Testing deps.amqp_client.unit_SUITE: Starting test, 2 test cases
Testing deps.amqp_client.unit_SUITE: TEST COMPLETE, 2 ok, 0 failed of 2 test cases
So the tests pass, cool :-) Thank you!
@dumbbell yeah but it's hard to tell from all the SASL reports noise (some of which simply reveals expected failures) :(
Yes, the output is very noisy...
The important line is at the end:
Testing deps.amqp_client.system_SUITE: TEST COMPLETE, 687 ok, 0 failed of 687 test cases
^^^^^^^^
In Cowboy I have code that hides all the expected failures, maybe this could apply here also: https://github.com/ninenines/ct_helper/blob/master/src/ct_helper_error_h.erl
I don't think I have updated it for 18+ yet (there was some changes in how errors are presented to error_handler).
All tests were moved from
test_util.erl
andnegative_test_util.erl
tounit_SUITE.erl
andsystem_SUITE.erl
.unit_SUITE.erl
has only unit tests and doesn't need a running broker.system_SUITE.erl
has system integration tests and takes care of starting and setting up a broker itself.make checks
automatically runs common_test tests. All our specific Makefile targets are not used anymore. The new testsuite just needs targets to start/stop a node and set/clear resource alarms.In
system_SUITE.erl
, almost all tests are executed with direct and network connections. Therefore, they are listed in two groups:To run the whole testsuite:
To run only one group:
To run one test in a group:
We also use common_test to handle the tests which are repeated 100 times.
Running tests in parallel doesn't work yet because one of the tests triggers all channels to be killed. This is to be investigated.
The started node stores all its data in a subdirectory or the
logs
directory and not in/tmp
. This allows to keep the database and log files for each run of the testsuite.erlang.mk
is updated at the same time: we don't need to disable any Erlang.mk plugins anymore.Fixes #39.