search

emqx / emqx

The most scalable open-source MQTT broker for IoT, IIoT, and connected vehicles
https://www.emqx.com/
Other
13.97k stars 2.23k forks source link

EMQX Not starting properly after System Update #3114

Closed NoelzeN closed 4 years ago

NoelzeN commented 4 years ago

System: CentOS 7 emqx Version: emqx-3.2.1-1.el7.x86_64 After a System Update I can't start emqx anymore. I have 2 nodes running on 2 more or less identical servers (Both CentOS 7). On Server 1, I performed the System Update and started emqx again, no issue at all. On Server 2, I performed the System Update and since then I can't start emqx anymore. The failing command seems to be this:

/usr/lib/emqx/erts-10.3.5.1/bin/beam.smp -spp true -e 256000 -Q 1024000 -P 2048000 -A 32 -zdbbl 8192 -- -root /usr/lib/emqx -progname usr/bin/emqx -- -home /var/lib/emqx -- -boot /usr/lib/emqx/releases/v3.2.1/emqx -mode embedded -boot_var ERTS_LIB_DIR /usr/lib/emqx/erts-10.3.5.1/../lib -mnesia dir "/var/lib/emqx/mnesia/emqx1@XXX.XXX.XXX.XXX" -config /var/lib/emqx/configs/app.2019.12.16.14.34.12.config -kernel net_ticktime 60 -setcookie [RANDOMCOOKIE] -ssl_dist_optfile /etc/emqx/ssl_dist.conf -proto_dist inet_tls -name emqx1@XXX.XXX.XXX.XXX -vm_args /var/lib/emqx/configs/vm.2019.12.16.14.34.12.args -- console

When I run it manually I get the following error:

Environment variable BINDIR is not set

I tried echo $BINDIR as well as root user and emqx user but on both Systems (The working one as well as the not working one) I get an empty result. Any hint what the issue might be? Unfortunately I'm not familiar with Erlang at all. The Logfiles are not very helpful.

[root@vmi150543 emqx]# tail run_erl.log run_erl [8509] Mon Dec 16 14:50:48 2019 Args before exec of shell: run_erl [8509] Mon Dec 16 14:50:48 2019 argv[0] = sh run_erl [8509] Mon Dec 16 14:50:48 2019 argv[1] = -c run_erl [8509] Mon Dec 16 14:50:48 2019 argv[2] = exec "/usr/bin/emqx" "console"

[root@vmi150543 emqx]# tail erlang.log.1 LOGGING STARTED Mon Dec 16 14:50:48 CET 2019 Exec: /usr/lib/emqx/erts-10.3.5.1/bin/erlexec -boot /usr/lib/emqx/releases/v3.2.1/emqx -mode embedded -boot_var ERTS_LIB_DIR /usr/lib/emqx/erts-10.3.5.1/../lib -mnesia dir "/var/lib/emqx/mnesia/emqx1@XXX.XXX.XXX.XXX" -config /var/lib/emqx/configs/app.2019.12.16.14.50.49.config -args_file /var/lib/emqx/configs/vm.2019.12.16.14.50.49.args -vm_args /var/lib/emqx/configs/vm.2019.12.16.14.50.49.args -- console Root: /usr/lib/emqx /usr/lib/emqx Starting emqx on node emqx1@XXX.XXX.XXX.XXX

[root@vmi150543 emqx]# tail erlang.log.1 LOGGING STARTED Mon Dec 16 14:50:48 CET 2019 Exec: /usr/lib/emqx/erts-10.3.5.1/bin/erlexec -boot /usr/lib/emqx/releases/v3.2.1/emqx -mode embedded -boot_var ERTS_LIB_DIR /usr/lib/emqx/erts-10.3.5.1/../lib -mnesia dir "/var/lib/emqx/mnesia/emqx1@XXX.XXX.XXX.XXX" -config /var/lib/emqx/configs/app.2019.12.16.14.50.49.config -args_file /var/lib/emqx/configs/vm.2019.12.16.14.50.49.args -vm_args /var/lib/emqx/configs/vm.2019.12.16.14.50.49.args -- console Root: /usr/lib/emqx /usr/lib/emqx Starting emqx on node emqx1@XXX.XXX.XXX.XXX [root@vmi150543 emqx]# ll total 32 -rw-r--r-- 1 emqx emqx 14414 Dec 16 14:50 emqx.log.1 -rw-r--r-- 1 emqx emqx 18 Dec 16 14:50 emqx.log.idx -rw-r--r-- 1 emqx emqx 13 Dec 16 14:50 emqx.log.siz -rw-r--r-- 1 emqx emqx 553 Dec 16 14:50 erlang.log.1 -rw-r--r-- 1 emqx emqx 254 Dec 16 14:50 run_erl.log [root@vmi150543 emqx]# tail emqx.log.1 {child_type,worker}] 2019-12-16 14:50:56.078 [info] supervisor: {local,mnesia_sup} started: [{pid,<0.1399.0>}, {id,mnesia_kernel_sup}, {mfargs,{mnesia_kernel_sup,start,[]}}, {restart_type,permanent}, {shutdown,infinity}, {child_type,supervisor}] 2019-12-16 14:50:56.078 [info] application: mnesia started_at: 'emqx1@XXX.XXX.XXX.XXX' [root@vmi150543 emqx]# tail -1000 emqx.log.1 2019-12-16 14:50:55.974 [info] application: kernel started_at: 'emqx1@XXX.XXX.XXX.XXX' 2019-12-16 14:50:55.974 [info] application: stdlib started_at: 'emqx1@XXX.XXX.XXX.XXX' 2019-12-16 14:50:55.975 [info] application: syntax_tools started_at: 'emqx1@XXX.XXX.XXX.XXX' 2019-12-16 14:50:55.975 [info] application: compiler started_at: 'emqx1@XXX.XXX.XXX.XXX' 2019-12-16 14:50:55.975 [info] supervisor: {local,runtime_tools_sup} started: [{pid,<0.1314.0>}, {id,ttb_autostart}, {mfargs,{ttb_autostart,start_link,[]}}, {restart_type,temporary}, {shutdown,3000}, {child_type,worker}] 2019-12-16 14:50:55.976 [info] application: runtime_tools started_at: 'emqx1@XXX.XXX.XXX.XXX' 2019-12-16 14:50:55.976 [info] application: getopt started_at: 'emqx1@XXX.XXX.XXX.XXX' 2019-12-16 14:50:55.976 [info] application: cuttlefish started_at: 'emqx1@XXX.XXX.XXX.XXX' 2019-12-16 14:50:55.976 [info] application: crypto started_at: 'emqx1@XXX.XXX.XXX.XXX' 2019-12-16 14:50:55.977 [info] application: cowlib started_at: 'emqx1@XXX.XXX.XXX.XXX' 2019-12-16 14:50:55.977 [info] application: asn1 started_at: 'emqx1@XXX.XXX.XXX.XXX' 2019-12-16 14:50:55.977 [info] application: public_key started_at: 'emqx1@XXX.XXX.XXX.XXX' 2019-12-16 14:50:55.977 [info] supervisor: {local,ssl_admin_sup} started: [{pid,<0.1326.0>}, {id,ssl_pem_cache}, {mfargs,{ssl_pem_cache,start_link,[[]]}}, {restart_type,permanent}, {shutdown,4000}, {child_type,worker}] 2019-12-16 14:50:55.978 [info] supervisor: {local,ssl_admin_sup} started: [{pid,<0.1327.0>}, {id,ssl_manager}, {mfargs,{ssl_manager,start_link,[[]]}}, {restart_type,permanent}, {shutdown,4000}, {child_type,worker}] 2019-12-16 14:50:55.978 [info] supervisor: {local,ssl_sup} started: [{pid,<0.1325.0>}, {id,ssl_admin_sup}, {mfargs,{ssl_admin_sup,start_link,[]}}, {restart_type,permanent}, {shutdown,4000}, {child_type,supervisor}] 2019-12-16 14:50:55.978 [info] supervisor: {local,ssl_connection_sup} started: [{pid,<0.1329.0>}, {id,tls_connection}, {mfargs,{tls_connection_sup,start_link,[]}}, {restart_type,permanent}, {shutdown,4000}, {child_type,supervisor}] 2019-12-16 14:50:55.979 [info] supervisor: {local,ssl_connection_sup} started: [{pid,<0.1330.0>}, {id,tls_socket}, {mfargs,{ssl_listen_tracker_sup,start_link,[]}}, {restart_type,permanent}, {shutdown,4000}, {child_type,supervisor}] 2019-12-16 14:50:55.979 [info] supervisor: {local,ssl_connection_sup} started: [{pid,<0.1331.0>}, {id,dtls_connection}, {mfargs,{dtls_connection_sup,start_link,[]}}, {restart_type,permanent}, {shutdown,4000}, {child_type,supervisor}] 2019-12-16 14:50:55.979 [info] supervisor: {local,ssl_connection_sup} started: [{pid,<0.1332.0>}, {id,dtls_listener}, {mfargs,{dtls_listener_sup,start_link,[]}}, {restart_type,permanent}, {shutdown,4000}, {child_type,supervisor}] 2019-12-16 14:50:55.980 [info] supervisor: {local,ssl_sup} started: [{pid,<0.1328.0>}, {id,ssl_connection_sup}, {mfargs,{ssl_connection_sup,start_link,[]}}, {restart_type,permanent}, {shutdown,4000}, {child_type,supervisor}] 2019-12-16 14:50:55.980 [info] application: ssl started_at: 'emqx1@XXX.XXX.XXX.XXX' 2019-12-16 14:50:55.981 [info] supervisor: {local,ranch_sup} started: [{pid,<0.1337.0>}, {id,ranch_server}, {mfargs,{ranch_server,start_link,[]}}, {restart_type,permanent}, {shutdown,5000}, {child_type,worker}] 2019-12-16 14:50:55.981 [info] application: ranch started_at: 'emqx1@XXX.XXX.XXX.XXX' 2019-12-16 14:50:55.981 [info] application: jsx started_at: 'emqx1@XXX.XXX.XXX.XXX' 2019-12-16 14:50:55.981 [info] supervisor: {local,gproc_sup} started: [{pid,<0.1343.0>}, {id,gproc}, {mfargs,{gproc,start_link,[]}}, {restart_type,permanent}, {shutdown,2000}, {child_type,worker}] 2019-12-16 14:50:55.982 [info] supervisor: {local,gproc_sup} started: [{pid,<0.1344.0>}, {id,gproc_monitor}, {mfargs,{gproc_monitor,start_link,[]}}, {restart_type,permanent}, {shutdown,2000}, {child_type,worker}] 2019-12-16 14:50:55.982 [info] supervisor: {local,gproc_sup} started: [{pid,<0.1345.0>}, {id,gproc_bcast}, {mfargs,{gproc_bcast,start_link,[]}}, {restart_type,permanent}, {shutdown,2000}, {child_type,worker}] 2019-12-16 14:50:55.982 [info] supervisor: {local,gproc_sup} started: [{pid,<0.1346.0>}, {id,gproc_pool}, {mfargs,{gproc_pool,start_link,[]}}, {restart_type,permanent}, {shutdown,2000}, {child_type,worker}] 2019-12-16 14:50:55.983 [info] application: gproc started_at: 'emqx1@XXX.XXX.XXX.XXX' 2019-12-16 14:50:55.983 [info] event=server_setup_successfully driver=tcp socket="#Port<0.8>" 2019-12-16 14:50:55.983 [info] supervisor: {local,gen_rpc_sup} started: [{pid,<0.1351.0>}, {id,gen_rpc_server_tcp}, {mfargs,{gen_rpc_server,start_link,[tcp]}}, {restart_type,permanent}, {shutdown,5000}, {child_type,worker}] 2019-12-16 14:50:55.984 [info] supervisor: {local,gen_rpc_sup} started: [{pid,<0.1352.0>}, {id,gen_rpc_acceptor_sup}, {mfargs,{gen_rpc_acceptor_sup,start_link,[]}}, {restart_type,permanent}, {shutdown,5000}, {child_type,supervisor}] 2019-12-16 14:50:55.984 [info] event=start 2019-12-16 14:50:55.984 [info] supervisor: {local,gen_rpc_sup} started: [{pid,<0.1353.0>}, {id,gen_rpc_dispatcher}, {mfargs,{gen_rpc_dispatcher,start_link,[]}}, {restart_type,permanent}, {shutdown,5000}, {child_type,worker}] 2019-12-16 14:50:55.984 [info] supervisor: {local,gen_rpc_sup} started: [{pid,<0.1354.0>}, {id,gen_rpc_client_sup}, {mfargs,{gen_rpc_client_sup,start_link,[]}}, {restart_type,permanent}, {shutdown,5000}, {child_type,supervisor}] 2019-12-16 14:50:55.984 [info] application: gen_rpc started_at: 'emqx1@XXX.XXX.XXX.XXX' 2019-12-16 14:50:55.985 [info] supervisor: {local,esockd_sup} started: [{pid,<0.1359.0>}, {id,rate_limiter}, {mfargs,{esockd_rate_limiter,start_link,[]}}, {restart_type,permanent}, {shutdown,5000}, {child_type,worker}] 2019-12-16 14:50:55.985 [info] supervisor: {local,esockd_sup} started: [{pid,<0.1360.0>}, {id,esockd_server}, {mfargs,{esockd_server,start_link,[]}}, {restart_type,permanent}, {shutdown,5000}, {child_type,worker}] 2019-12-16 14:50:55.985 [info] application: esockd started_at: 'emqx1@XXX.XXX.XXX.XXX' 2019-12-16 14:50:55.986 [info] supervisor: {local,cowboy_sup} started: [{pid,<0.1365.0>}, {id,cowboy_clock}, {mfargs,{cowboy_clock,start_link,[]}}, {restart_type,permanent}, {shutdown,5000}, {child_type,worker}] 2019-12-16 14:50:55.986 [info] application: cowboy started_at: 'emqx1@XXX.XXX.XXX.XXX' 2019-12-16 14:50:55.986 [info] application: replayq started_at: 'emqx1@XXX.XXX.XXX.XXX' 2019-12-16 14:50:55.987 [info] supervisor: {local,sasl_safe_sup} started: [{pid,<0.1372.0>}, {id,alarm_handler}, {mfargs,{alarm_handler,start_link,[]}}, {restart_type,permanent}, {shutdown,2000}, {child_type,worker}] 2019-12-16 14:50:55.987 [info] supervisor: {local,sasl_sup} started: [{pid,<0.1371.0>}, {id,sasl_safe_sup}, {mfargs,{supervisor,start_link, [{local,sasl_safe_sup},sasl,safe]}}, {restart_type,permanent}, {shutdown,infinity}, {child_type,supervisor}] 2019-12-16 14:50:55.993 [info] supervisor: {local,sasl_sup} started: [{pid,<0.1373.0>}, {id,release_handler}, {mfargs,{release_handler,start_link,[]}}, {restart_type,permanent}, {shutdown,2000}, {child_type,worker}] 2019-12-16 14:50:55.993 [info] application: sasl started_at: 'emqx1@XXX.XXX.XXX.XXX' 2019-12-16 14:50:56.000 [info] supervisor: {local,os_mon_sup} started: [{pid,<0.1379.0>}, {id,disksup}, {mfargs,{disksup,start_link,[]}}, {restart_type,permanent}, {shutdown,2000}, {child_type,worker}] 2019-12-16 14:50:56.000 [info] supervisor: {local,os_mon_sup} started: [{pid,<0.1380.0>}, {id,memsup}, {mfargs,{memsup,start_link,[]}}, {restart_type,permanent}, {shutdown,2000}, {child_type,worker}] 2019-12-16 14:50:56.001 [info] supervisor: {local,os_mon_sup} started: [{pid,<0.1382.0>}, {id,cpu_sup}, {mfargs,{cpu_sup,start_link,[]}}, {restart_type,permanent}, {shutdown,2000}, {child_type,worker}] 2019-12-16 14:50:56.001 [info] application: os_mon started_at: 'emqx1@XXX.XXX.XXX.XXX' 2019-12-16 14:50:56.017 [notice] alarm_handler: {set,{system_memory_high_watermark,[]}} 2019-12-16 14:50:56.018 [info] supervisor: {local,kernel_safe_sup} started: [{pid,<0.1388.0>}, {id,dets_sup}, {mfargs,{dets_sup,start_link,[]}}, {restart_type,permanent}, {shutdown,1000}, {child_type,supervisor}] 2019-12-16 14:50:56.018 [info] supervisor: {local,kernel_safe_sup} started: [{pid,<0.1389.0>}, {id,dets}, {mfargs,{dets_server,start_link,[]}}, {restart_type,permanent}, {shutdown,2000}, {child_type,worker}] 2019-12-16 14:50:56.021 [info] supervisor: {local,mnesia_sup} started: [{pid,<0.1397.0>}, {id,mnesia_event}, {mfargs,{mnesia_sup,start_event,[]}}, {restart_type,permanent}, {shutdown,30000}, {child_type,worker}] 2019-12-16 14:50:56.021 [info] supervisor: {local,mnesia_sup} started: [{pid,<0.1398.0>}, {id,mnesia_ext_sup}, {mfargs,{mnesia_ext_sup,start,[]}}, {restart_type,permanent}, {shutdown,infinity}, {child_type,supervisor}] 2019-12-16 14:50:56.022 [info] supervisor: {local,mnesia_kernel_sup} started: [{pid,<0.1400.0>}, {id,mnesia_monitor}, {mfargs,{mnesia_monitor,start,[]}}, {restart_type,permanent}, {shutdown,3000}, {child_type,worker}] 2019-12-16 14:50:56.023 [info] supervisor: {local,mnesia_kernel_sup} started: [{pid,<0.1401.0>}, {id,mnesia_subscr}, {mfargs,{mnesia_subscr,start,[]}}, {restart_type,permanent}, {shutdown,3000}, {child_type,worker}] 2019-12-16 14:50:56.023 [info] supervisor: {local,mnesia_kernel_sup} started: [{pid,<0.1402.0>}, {id,mnesia_locker}, {mfargs,{mnesia_locker,start,[]}}, {restart_type,permanent}, {shutdown,3000}, {child_type,worker}] 2019-12-16 14:50:56.024 [info] supervisor: {local,mnesia_kernel_sup} started: [{pid,<0.1403.0>}, {id,mnesia_recover}, {mfargs,{mnesia_recover,start,[]}}, {restart_type,permanent}, {shutdown,180000}, {child_type,worker}] 2019-12-16 14:50:56.032 [info] supervisor: {local,inet_gethost_native_sup} started: [{pid,<0.1414.0>},{mfa,{inet_gethost_native,init,[[]]}}] 2019-12-16 14:50:56.032 [info] supervisor: {local,kernel_safe_sup} started: [{pid,<0.1413.0>}, {id,inet_gethost_native_sup}, {mfargs,{inet_gethost_native,start_link,[]}}, {restart_type,temporary}, {shutdown,1000}, {child_type,worker}] 2019-12-16 14:50:56.076 [info] supervisor: {local,mnesia_kernel_sup} started: [{pid,<0.1404.0>}, {id,mnesia_tm}, {mfargs,{mnesia_tm,start,[]}}, {restart_type,permanent}, {shutdown,30000}, {child_type,worker}] 2019-12-16 14:50:56.076 [info] supervisor: {local,mnesia_kernel_sup} started: [{pid,<0.1426.0>}, {id,mnesia_checkpoint_sup}, {mfargs,{mnesia_checkpoint_sup,start,[]}}, {restart_type,permanent}, {shutdown,infinity}, {child_type,supervisor}] 2019-12-16 14:50:56.076 [info] supervisor: {local,mnesia_kernel_sup} started: [{pid,<0.1427.0>}, {id,mnesia_controller}, {mfargs,{mnesia_controller,start,[]}}, {restart_type,permanent}, {shutdown,3000}, {child_type,worker}] 2019-12-16 14:50:56.077 [info] supervisor: {local,mnesia_kernel_sup} started: [{pid,<0.1428.0>}, {id,mnesia_late_loader}, {mfargs,{mnesia_late_loader,start,[]}}, {restart_type,permanent}, {shutdown,3000}, {child_type,worker}] 2019-12-16 14:50:56.078 [info] supervisor: {local,mnesia_sup} started: [{pid,<0.1399.0>}, {id,mnesia_kernel_sup}, {mfargs,{mnesia_kernel_sup,start,[]}}, {restart_type,permanent}, {shutdown,infinity}, {child_type,supervisor}] 2019-12-16 14:50:56.078 [info] application: mnesia started_at: 'emqx1@XXX.XXX.XXX.XXX'

turtleDeng commented 4 years ago

Try using

emqx console
NoelzeN commented 4 years ago

When running emqx console immediately I get

[root@vmi150543 nrehwald]# emqx console
Exec: /usr/lib/emqx/erts-10.3.5.1/bin/erlexec -boot /usr/lib/emqx/releases/v3.2.1/emqx -mode embedded -boot_var ERTS_LIB_DIR /usr/lib/emqx/erts-10.3.5.1/../lib -mnesia dir "/var/lib/emqx/mnesia/emqx1@XXX.XXX.XXX.XXX" -config /var/lib/emqx/configs/app.2019.12.17.09.17.17.config -args_file /var/lib/emqx/configs/vm.2019.12.17.09.17.17.args -vm_args /var/lib/emqx/configs/vm.2019.12.17.09.17.17.args -- console
Root: /usr/lib/emqx
/usr/lib/emqx
Erlang/OTP 21 [erts-10.3.5.1] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:32] [hipe]

Starting emqx on node emqx1@XXX.XXX.XXX.XXX

Then after a minute or two I get

[os_mon] memory supervisor port (memsup): Erlang has closed
[os_mon] cpu supervisor port (cpu_sup): Erlang has closed
{"Kernel pid terminated",application_controller,"{application_start_failure,emqx,{bad_return,{{emqx_app,start,[normal,[]]},{'EXIT',{{badmatch,{error,{shutdown,{failed_to_start_child,emqx_router_sup,{shutdown,{failed_to_start_child,helper,{{badmatch,{error,{not_active_local,emqx_routing_node}}},[{emqx_router_helper,init,1,[{file,\"/tmp/emqx/v3.2.1/_checkouts/emqx/src/emqx_router_helper.erl\"},{line,95}]},{gen_server,init_it,2,[{file,\"gen_server.erl\"},{line,374}]},{gen_server,init_it,6,[{file,\"gen_server.erl\"},{line,342}]},{proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,249}]}]}}}}}}},[{emqx_app,start,2,[{file,\"/tmp/emqx/v3.2.1/_checkouts/emqx/src/emqx_app.erl\"},{line,32}]},{application_master,start_it_old,4,[{file,\"application_master.erl\"},{line,277}]}]}}}}}"}
Kernel pid terminated (application_controller) ({application_start_failure,emqx,{bad_return,{{emqx_app,start,[normal,[]]},{'EXIT',{{badmatch,{error,{shutdown,{failed_to_start_child,emqx_router_sup,{sh

Crash dump is being written to: /var/log/emqx/crash.dump...done
You have new mail in /var/spool/mail/nrehwald

If I run ps aux during the time when the command line shows "Starting emqx..." for a minute or so, I get the following:

[root@vmi150543 emqx]# ps aux | grep emqx
root     10338  0.0  0.0 191780  2348 pts/0    S+   09:17   0:00 su - emqx -c /usr/bin/emqx console
emqx     10412  1.1  1.6 2316872 96560 ?       Ssl  09:17   0:01 /usr/lib/emqx/erts-10.3.5.1/bin/beam.smp -spp true -e 256000 -Q 1024000 -P 2048000 -A 32 -zdbbl 8192 -- -root /usr/lib/emqx -progname usr/bin/emqx -- -home /var/lib/emqx -- -boot /usr/lib/emqx/releases/v3.2.1/emqx -mode embedded -boot_var ERTS_LIB_DIR /usr/lib/emqx/erts-10.3.5.1/../lib -mnesia dir "/var/lib/emqx/mnesia/emqx1@XXX.XXX.XXX.XXX" -config /var/lib/emqx/configs/app.2019.12.17.09.17.17.config -kernel net_ticktime 60 -setcookie [SOMERANDOMCOOKIE] -ssl_dist_optfile /etc/emqx/ssl_dist.conf -proto_dist inet_tls -name emqx1@XXX.XXX.XXX.XXX -vm_args /var/lib/emqx/configs/vm.2019.12.17.09.17.17.args -- console
emqx     10582  0.0  0.0  11628   340 ?        S    09:17   0:00 /usr/lib/emqx/erts-10.3.5.1/bin/epmd -daemon
emqx     11092  0.0  0.0   4356   524 ?        Ss   09:17   0:00 erl_child_setup 1024
emqx     11111  0.0  0.0 113192  1208 ?        Ss   09:17   0:00 sh -s disksup
emqx     11114  0.0  0.0   4348   596 ?        Ss   09:17   0:00 /usr/lib/emqx/lib/os_mon-2.4.7/priv/bin/memsup
emqx     11115  0.0  0.0   4348   352 ?        Ss   09:17   0:00 /usr/lib/emqx/lib/os_mon-2.4.7/priv/bin/cpu_sup
emqx     11116  0.0  0.0  11592   452 ?        Ss   09:17   0:00 inet_gethost 4
emqx     11117  0.0  0.0  11592   432 ?        S    09:17   0:00 inet_gethost 4
root     11515  0.0  0.0 112716   964 pts/1    S+   09:19   0:00 grep --color=auto emqx

[root@vmi150543 emqx]# netstat -tulpn | grep 5369
tcp        0      0 0.0.0.0:5369            0.0.0.0:*               LISTEN      10412/beam.smp      
[root@vmi150543 emqx]# netstat -tulpn | grep 6369
tcp        0      0 0.0.0.0:6369            0.0.0.0:*               LISTEN      10412/beam.smp

After I get the error message in console, I get this:

[root@vmi150543 emqx]# ps aux | grep emqx
emqx     10582  0.0  0.0  11628   340 ?        S    09:17   0:00 /usr/lib/emqx/erts-10.3.5.1/bin/epmd -daemon
root     11568  0.0  0.0 112716   964 pts/1    S+   09:20   0:00 grep --color=auto emqx

Content of crash.dump file

turtleDeng commented 4 years ago

Try using

rm -rf /var/lib/emqx/mnesia/*
emqx console
NoelzeN commented 4 years ago

This solved the issue, now I can start emqx again. Thanks a lot. Our of curiosity, how did this cause the issue? I had multiple folders in there and now I only have one, was that related to the issue?