VerneMQ server generates a lot of errors and no results when testing fan_in.bdl

[syberkitten]

VMQ server is running locally on my Mac Book Pro, El-Capitan, compiled from source.

I'm getting both system errors and user errors from the default BDL test fan_in.bdl: System:

2:59:45.965 [error] [mzb_director8_0@127.0.0.1] <0.266.0> [ "pool2" ] Received DOWN from worker <0.1938.0> with reason {{badmatch,{error,einval}},[{gen_emqtt,active_once,2,[{file,"src/gen_emqtt.erl"},{line,604}]},{gen_emqtt,connecting,2,[{file,"src/gen_emqtt.erl"},{line,180}]},{gen_fsm,handle_msg,8,[{file,"gen_fsm.erl"},{line,483}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}
--
22:59:45.967 [error] [mzb_director8_0@127.0.0.1] <0.266.0> [ "pool2" ] Received DOWN from worker <0.1931.0> with reason {{badmatch,{error,einval}},[{gen_emqtt,active_once,2,[{file,"src/gen_emqtt.erl"},{line,604}]},{gen_emqtt,connecting,2,[{file,"src/gen_emqtt.erl"},{line,180}]},{gen_fsm,handle_msg,8,[{file,"gen_fsm.erl"},{line,483}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}
22:59:45.967 [error] [mzb_director8_0@127.0.0.1] <0.266.0> [ "pool2" ] Received DOWN from worker <0.1939.0> with reason {{badmatch,{error,einval}},[{gen_emqtt,active_once,2,[{file,"src/gen_emqtt.erl"},{line,604}]},{gen_emqtt,connecting,2,[{file,"src/gen_emqtt.erl"},{line,180}]},{gen_fsm,handle_msg,8,[{file,"gen_fsm.erl"},{line,483}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}
22:59:45.967 [error] [mzb_director8_0@127.0.0.1] <0.266.0> [ "pool2" ] Received DOWN from worker <0.1937.0> with reason {{badmatch,{error,einval}},[{gen_emqtt,active_once,2,[{file,"src/gen_emqtt.erl"},{line,604}]},{gen_emqtt,connecting,2,[{file,"src/gen_emqtt.erl"},{line,180}]},{gen_fsm,handle_msg,8,[{file,"gen_fsm.erl"},{line,483}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}

User Errors:

22:59:45.937 [error] [Undefined] <0.1932.0> gen_fsm <0.1932.0> in state connecting terminated with reason: no match of right hand value {error,einval} in gen_emqtt:active_once/2 line 604
--
22:59:45.937 [error] [Undefined] <0.1941.0> gen_fsm <0.1941.0> in state connecting terminated with reason: no match of right hand value {error,einval} in gen_emqtt:active_once/2 line 604
22:59:45.937 [error] [Undefined] <0.1942.0> gen_fsm <0.1942.0> in state connecting terminated with reason: no match of right hand value {error,einval} in gen_emqtt:active_once/2 line 604
22:59:45.937 [error] [Undefined] <0.1940.0> gen_fsm <0.1940.0> in state connecting terminated with reason: no match of right hand value {error,einval} in gen_emqtt:active_once/2 line 604
22:59:45.963 [error] [Undefined] <0.1941.0> CRASH REPORT Process <0.1941.0> with 1 neighbours exited with reason: no match of right hand value {error,einval} in gen_emqtt:active_once/2 line 604 in gen_fsm:terminate/7 line 602
22:59:45.963 [error] [Undefined] <0.1932.0> CRASH REPORT Process <0.1932.0> with 1 neighbours exited with reason: no match of right hand value {error,einval} in gen_emqtt:active_once/2 line 604 in gen_fsm:terminate/7 line 602
22:59:45.964 [error] [Undefined] <0.1942.0> CRASH REPORT Process <0.1942.0> with 1 neighbours exited with reason: no match of right hand value {error,einval} in gen_emqtt:active_once/2 line 604 in gen_fsm:terminate/7 line 602
22:59:45.965 [error] [Undefined] <0.1940.0> CRASH REPORT Process <0.1940.0> with 1 neighbours exited with reason: no match of right hand value {error,einval} in gen_emqtt:active_once/2 line 604 in gen_fsm:terminate/7 line 602
22:59:50.048 [error] [Undefined] <0.1963.0> gen_fsm <0.1963.0> in state connecting terminated with reason: no match of right hand value {error,einval} in gen_emqtt:active_once/2 line 604
22:59:50.048 [error] [Undefined] <0.1963.0> CRASH REPORT Process <0.1963.0> with 1 neighbours exited with reason: no match of right han

[ioolkos]

Hi @syberkitten, this is not VerneMQ crashing, but the test environment. It runs into some sort of network troubles indicated by EINVAL errors, which can probably be caused by a couple of things. Is your 'localhost' resolving ok?

[syberkitten]

Hello @ioolkos Yes, my localhost is ok. I'm also getting now more errors from mzbench itself, even when testing a remote server

23:50:25.992 [error] [mzb_director11_0@127.0.0.1] <0.266.0> Worker <0.467.0> on 'mzb_director11_0@127.0.0.1' has crashed: undef
Stacktrace: [{crypto,rand_bytes,[150],[]},{mzbl_stdlib,random_binary,4,[{file,"/private/tmp/bench_mzbench_api_Liams-MacBook-Pro_1500_578967_295800/deployment_code/node/_build/default/deps/mzbench_language/src/mzbl_stdlib.erl"},{line,79}]},{mzbl_interpreter,'-eval/4-fun-0-',4,[{file,"/private/tmp/bench_mzbench_api_Liams-MacBook-Pro_1500_578967_295800/deployment_code/node/_build/default/deps/mzbench_language/src/mzbl_interpreter.erl"},{line,19}]},{lists,foldl,3,[{file,"lists.erl"},{line,1263}]},{mzbl_interpreter,eval_function,6,[{file,"/private/tmp/bench_mzbench_api_Liams-MacBook-Pro_1500_578967_295800/deployment_code/node/_build/default/deps/mzbench_language/src/mzbl_interpreter.erl"},{line,33}]},{mzbl_interpreter,'-eval/4-fun-0-',4,[{file,"/private/tmp/bench_mzbench_api_Liams-MacBook-Pro_1500_578967_295800/deployment_code/node/_build/default/deps/mzbench_language/src/mzbl_interpreter.erl"},{line,19}]},{lists,foldl,3,[{file,"lists.erl"},{line,1263}]},{mzbl_loop,k_times,5,[{file,"/private/tmp/bench_mzbench_api_Liams-MacBook-Pro_1500_578967_295800/deployment_code/node/_build/default/deps/mzbench_language/src/mzbl_loop.erl"},{line,316}]}]

00:09:13.354 [error] [mzb_director13_0@127.0.0.1] <0.266.0> Worker <0.1455.0> on 'mzb_director13_0@127.0.0.1' has crashed: undef Stacktrace: [{crypto,rand_bytes,[150],[]},{mzbl_stdlib,random_binary,4,[{file,"/private/tmp/bench_mzbench_api_Liams-MacBook-Pro_1500_578967_295800/deployment_code/node/_build/default/deps/mzbench_language/src/mzbl_stdlib.erl"},{line,79}]},{mzbl_interpreter,'-eval/4-fun-0-',4,[{file,"/private/tmp/bench_mzbench_api_Liams-MacBook-Pro_1500_578967_295800/deployment_code/node/_build/default/deps/mzbench_language/src/mzbl_interpreter.erl"},{line,19}]},{lists,foldl,3,[{file,"lists.erl"},{line,1263}]},{mzbl_interpreter,eval_function,6,[{file,"/private/tmp/bench_mzbench_api_Liams-MacBook-Pro_1500_578967_295800/deployment_code/node/_build/default/deps/mzbench_language/src/mzbl_interpreter.erl"},{line,33}]},{mzbl_interpreter,'-eval/4-fun-0-',4,[{file,"/private/tmp/bench_mzbench_api_Liams-MacBook-Pro_1500_578967_295800/deployment_code/node/_build/default/deps/mzbench_language/src/mzbl_interpreter.erl"},{line,19}]},{lists,foldl,3,[{file,"lists.erl"},{line,1263}]},{mzbl_loop,k_times,5,[{file,"/private/tmp/bench_mzbench_api_Liams-MacBook-Pro_1500_578967_295800/deployment_code/node/_build/default/deps/mzbench_language/src/mzbl_loop.erl"},{line,316}]}]
--
00:09:13.354 [error] [mzb_director13_0@127.0.0.1] <0.266.0> Worker <0.449.0> on 'mzb_director13_0@127.0.0.1' has crashed: undef Stacktrace: [{crypto,rand_bytes,[150],[]},{mzbl_stdlib,random_binary,4,[{file,"/private/tmp/bench_mzbench_api_Liams-MacBook-Pro_1500_578967_295800/deployment_code/node/_build/default/deps/mzbench_language/src/mzbl_stdlib.erl"},{line,79}]},{mzbl_interpreter,'-eval/4-fun-0-',4,[{file,"/private/tmp/bench_mzbench_api_Liams-MacBook-Pro_1500_578967_295800/deployment_code/node/_build/default/deps/mzbench_language/src/mzbl_interpreter.erl"},{line,19}]},{lists,foldl,3,[{file,"lists.erl"},{line,1263}]},{mzbl_interpreter,eval_function,6,[{file,"/private/tmp/bench_mzbench_api_Liams-MacBook-Pro_1500_578967_295800/deployment_code/node/_build/default/deps/mzbench_language/src/mzbl_interpreter.erl"},{line,33}]},{mzbl_interpreter,'-eval/4-fun-0-',4,[{file,"/private/tmp/bench_mzbench_api_Liams-MacBook-Pro_1500_578967_295800/deployment_code/node/_build/default/deps/mzbench_language/src/mzbl_interpreter.erl"},{line,19}]},{lists,foldl,3,[{file,"lists.erl"},{line,1263}]},{mzbl_loop,k_times,5,[{file,"/private/tmp/bench_mzbench_api_Liams-MacBook-Pro_1500_578967_295800/deployment_code/node/_build/default/deps/mzbench_language/src/mzbl_loop.erl"},{line,316}]}]

Happened both in Erlang/OTP 20/19 Could this be related to an open files limit of 10000 ? I will try to reduce the number of workers and test again.

[ioolkos]

crypto:rand_bytes/1 was removed in OTP 20, in favour of crypto:strong_rand_bytes/1. That immediately makes MZBench (the underlying Machinezone MZBench framework) incompatible. I guess I always used it on OTP 18.

And yes, 10k file descriptors is low, can you configure this for a higher value or run a lower scenario?

Many thanks for your ongoing effort & learnings for OS X!

[syberkitten]

@ioolkos thanks for the feedback, I've switched to Erlang/OTP 18.3 using Kerl, and the benchmark is working. 👍

[ioolkos]

Great! One additional thing to note is that vmq_mzbench is asynchronous, that's why sometimes you have to play around with wait times, especially for subscribers.

[ioolkos]

I guess I can close this issue.