Rory-Z
commented
1 year ago Looks like same to https://github.com/emqx/emqx/issues/11071 ？
Closed webfrank closed 1 year ago
Rory-Z
commented
1 year ago Looks like same to https://github.com/emqx/emqx/issues/11071 ？
webfrank
commented
1 year ago Hi, not same error. I try to get the log.
Running 5.0.14 exit with error once core get upgraded to 5.0.26:
2023-06-16T12:19:42.182978+00:00 [warning] last_version: 0, line: 112, mfa: mria_lb:handle_call/3, msg: Different Mria version on the core node, my_version: 0, node: 'emqx@emqx-core-0.emqx-headless.emqx.svc.cluster.local', server_version: 1
--
2023-06-16T12:19:42.256286+00:00 [error] crasher: initial call: emqx_connection:init/4, pid: <0.21480.112>, registered_name: [], error: {{case_clause,{badrpc,{'EXIT',{undef,[{mria_lib,dirty_wrapper,[mnesia,dirty_write,emqx_channel_registry,[{channel,<<"MzExMTgxNDQyMzc3MzQ4MTE4NjA3MTIxMDE4Nzg2OTM4ODI">>,<0.21480.112>}]],[]},{gen_rpc_acceptor,call_middleman,3,[{file,"gen_rpc_acceptor.erl"},{line,287}]}]}}}},[{mria_lib,call_backend_rw_dirty,4,[{file,"mria_lib.erl"},{line,195}]},{emqx_cm,register_channel,3,[{file,"emqx_cm.erl"},{line,170}]},{emqx_cm,'-open_session/3-fun-0-',5,[{file,"emqx_cm.erl"},{line,275}]},{emqx_cm_locker,trans,3,[{file,"emqx_cm_locker.erl"},{line,51}]},{emqx_channel,process_connect,2,[{file,"emqx_channel.erl"},{line,616}]},{emqx_connection,with_channel,3,[{file,"emqx_connection.erl"},{line,790}]},{emqx_connection,process_msg,2,[{file,"emqx_connection.erl"},{line,466}]},{emqx_connection,process_msg,2,[{file,"emqx_connection.erl"},{line,472}]},{emqx_connection,handle_recv,3,[{file,"emqx_connection.erl"},{line,428}]},{proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,236}]}]}, ancestors: [<0.2302.0>,<0.2301.0>,esockd_sup,<0.1922.0>], message_queue_len: 0, messages: [], links: [<0.2302.0>], dictionary: [{guid,{1686917979316060,61711969178600,1}},{'$logger_metadata,#{clientid => <<"MzExMTgxNDQyMzc3MzQ4MTE4NjA3MTIxMDE4Nzg2OTM4ODI">>,peername => "192.168.70.121:9730"}},{recv_pkt,1},{incoming_bytes,59}], trap_exit: false, status: running, heap_size: 2586, stack_size: 29, reductions: 5349; neighbours:
2023-06-16T12:19:42.257791+00:00 [error] supervisor: 'esockd_connection_sup - <0.2302.0>', errorContext: connection_crashed, reason: {{case_clause,{badrpc,{'EXIT',{undef,[{mria_lib,dirty_wrapper,[mnesia,dirty_write,emqx_channel_registry,[{channel,<<"MzExMTgxNDQyMzc3MzQ4MTE4NjA3MTIxMDE4Nzg2OTM4ODI">>,<0.21480.112>}]],[]},{gen_rpc_acceptor,call_middleman,3,[{file,"gen_rpc_acceptor.erl"},{line,287}]}]}}}},[{mria_lib,call_backend_rw_dirty,4,[{file,"mria_lib.erl"},{line,195}]},{emqx_cm,register_channel,3,[{file,"emqx_cm.erl"},{line,170}]},{emqx_cm,'-open_session/3-fun-0-',5,[{file,"emqx_cm.erl"},{line,275}]},{emqx_cm_locker,trans,3,[{file,"emqx_cm_locker.erl"},{line,51}]},{emqx_channel,process_connect,2,[{file,"emqx_channel.erl"},{line,616}]},{emqx_connection,with_channel,3,[{file,"emqx_connection.erl"},{line,790}]},{emqx_connection,process_msg,2,[{file,"emqx_connection.erl"},{line,466}]},{emqx_connection,process_msg,2,[{file,"emqx_connection.erl"},{line,472}]},{emqx_connection,handle_recv,3,[{file,"emqx_connection.erl"},{line,428}]},{proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,236}]}]}, offender: [{pid,<0.21480.112>},{name,connection},{mfargs,{emqx_connection,start_link,[#{enable_authn => true,limiter => #{bytes_in => #{capacity => 1099511627776,initial => 0,rate => infinity},client => #{bytes_in => #{capacity => 1099511627776,divisible => false,failure_strategy => force,initial => 0,low_watermark => 0,max_retry_time => 10000,rate => infinity},connection => #{capacity => 1099511627776,divisible => false,failure_strategy => force,initial => 0,low_watermark => 0,max_retry_time => 10000,rate => infinity},message_in => #{capacity => 1099511627776,divisible => false,failure_strategy => force,initial => 0,low_watermark => 0,max_retry_time => 10000,rate => infinity},message_routing => #{capacity => 1099511627776,divisible => false,failure_strategy => force,initial => 0,low_watermark => 0,max_retry_time => 10000,rate => infinity}},connection => #{capacity => 1000,initial => 0,rate => 100.0}
,message_in => #{capacity => 1099511627776,initial => 0,rate => infinity},message_routing => #{capacity => 1099511627776,initial => 0,rate => infinity}},listener => {tcp,default},zone => default}]}}]
2023-06-16T12:19:42.261741+00:00 [error] Generic server emqx_broker_1 terminating. Reason: {{case_clause,{badrpc,{'EXIT',{undef,[{mria_lib,dirty_wrapper,[mnesia,dirty_delete_object,emqx_route,[{route,<<"keycloak/events/events">>,'emqx@192.168.194.242'}]],[]},{gen_rpc_acceptor,call_middleman,3,[{file,"gen_rpc_acceptor.erl"},{line,287}]}]}}}},[{mria_lib,call_backend_rw_dirty,4,[{file,"mria_lib.erl"},{line,195}]},{emqx_broker,handle_cast,2,[{file,"emqx_broker.erl"},{line,512}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,695}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,771}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}. Last message: {'$gen_cast',{unsubscribed,<<"keycloak/events/events">>}}. State: #{id => 1,pool => broker_pool}.
2023-06-16T12:19:42.262119+00:00 [error] crasher: initial call: emqx_broker:init/1, pid: <0.2215.0>, registered_name: emqx_broker_1, error: {{case_clause,{badrpc,{'EXIT',{undef,[{mria_lib,dirty_wrapper,[mnesia,dirty_delete_object,emqx_route,[{route,<<"keycloak/events/events">>,'emqx@192.168.194.242'}]],[]},{gen_rpc_acceptor,call_middleman,3,[{file,"gen_rpc_acceptor.erl"},{line,287}]}]}}}},[{mria_lib,call_backend_rw_dirty,4,[{file,"mria_lib.erl"},{line,195}]},{emqx_broker,handle_cast,2,[{file,"emqx_broker.erl"},{line,512}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,695}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,771}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}, ancestors: [<0.2214.0>,emqx_broker_sup,emqx_sup,<0.2135.0>], message_queue_len: 0, messages: [], links: [<0.2214.0>], dictionary: [], trap_exit: false, status: running, heap_size: 6772, stack_size: 29, reductions: 76064; neighbours:
2023-06-16T12:19:42.262592+00:00 [error] Supervisor: {<0.2214.0>,emqx_pool_sup}. Context: child_terminated. Reason: {{case_clause,{badrpc,{'EXIT',{undef,[{mria_lib,dirty_wrapper,[mnesia,dirty_delete_object,emqx_route,[{route,<<"keycloak/events/events">>,'emqx@192.168.194.242'}]],[]},{gen_rpc_acceptor,call_middleman,3,[{file,"gen_rpc_acceptor.erl"},{line,287}]}]}}}},[{mria_lib,call_backend_rw_dirty,4,[{file,"mria_lib.erl"},{line,195}]},{emqx_broker,handle_cast,2,[{file,"emqx_broker.erl"},{line,512}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,695}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,771}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}. Offender: id={emqx_broker,1},pid=<0.2215.0>.
2023-06-16T12:19:42.261624+00:00 [error] exception: error, line: 120, mfa: emqx_pool:handle_cast/2, msg: async_submit_error, reason: {case_clause,{badrpc,{'EXIT',{undef,[{mria_lib,dirty_wrapper,[mnesia,dirty_delete_object,emqx_channel_registry,[{channel,<<"mosq/3AVBUCduKfnSXcWZG3">>,<0.19354.112>}]],[]},{gen_rpc_acceptor,call_middleman,3,[{file,"gen_rpc_acceptor.erl"},{line,287}]}]}}}}, stacktrace: [{mria_lib,call_backend_rw_dirty,4,[{file,"mria_lib.erl"},{line,195}]},{emqx_cm,do_unregister_channel,1,[{file,"emqx_cm.erl"},{line,182}]},{lists,foreach,2,[{file,"lists.erl"},{line,1342}]},{emqx_pool,handle_cast,2,[{file,"emqx_pool.erl"},{line,112}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,695}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,771}]},{proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,236}]}]
2023-06-16T12:19:42.282781+00:00 [error] crasher: initial call: emqx_connection:init/4, pid: <0.21482.112>, registered_name: [], error: {{case_clause,{badrpc,{'EXIT',{undef,[{mria_lib,dirty_wrapper,[mnesia,dirty_write,emqx_channel_registry,[{channel,<<"mosq/gizn07c8G99dnT5daJ">>,<0.21482.112>}]],[]},{gen_rpc_acceptor,call_middleman,3,[{file,"gen_rpc_acceptor.erl"},{line,287}]}]}}}},[{mria_lib,call_backend_rw_dirty,4,[{file,"mria_lib.erl"},{line,195}]},{emqx_cm,register_channel,3,[{file,"emqx_cm.erl"},{line,170}]},{emqx_cm,'-open_session/3-fun-0-',5,[{file,"emqx_cm.erl"},{line,275}]},{emqx_cm_locker,trans,3,[{file,"emqx_cm_locker.erl"},{line,51}]},{emqx_channel,process_connect,2,[{file,"emqx_channel.erl"},{line,616}]},{emqx_connection,with_channel,3,[{file,"emqx_connection.erl"},{line,790}]},{emqx_connection,process_msg,2,[{file,"emqx_connection.erl"},{line,466}]},{emqx_connection,process_msg,2,[{file,"emqx_connection.erl"},{line,472}]},{emqx_connection,handle_recv,3,[{file,"emqx_connection.erl"},{line,428}]},{proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,236}]}]}, ancestors: [<0.2302.0>,<0.2301.0>,esockd_sup,<0.1922.0>], message_queue_len: 0, messages: [], links: [<0.2302.0>], dictionary: [{guid,{1686917979935135,61711969178602,0}},{'$logger_metadata,#{clientid => <<"mosq/gizn07c8G99dnT5daJ">>,peername => "192.168.232.152:16678"}},{recv_pkt,1},{incoming_bytes,111}], trap_exit: false, status: running, heap_size: 2586, stack_size: 29, reductions: 5217; neighbours:
2023-06-16T12:19:42.283587+00:00 [error] supervisor: 'esockd_connection_sup - <0.2302.0>', errorContext: connection_crashed, reason: {{case_clause,{badrpc,{'EXIT',{undef,[{mria_lib,dirty_wrapper,[mnesia,dirty_write,emqx_channel_registry,[{channel,<<"mosq/gizn07c8G99dnT5daJ">>,<0.21482.112>}]],[]},{gen_rpc_acceptor,call_middleman,3,[{file,"gen_rpc_acceptor.erl"},{line,287}]}]}}}},[{mria_lib,call_backend_rw_dirty,4,[{file,"mria_lib.erl"},{line,195}]},{emqx_cm,register_channel,3,[{file,"emqx_cm.erl"},{line,170}]},{emqx_cm,'-open_session/3-fun-0-',5,[{file,"emqx_cm.erl"},{line,275}]},{emqx_cm_locker,trans,3,[{file,"emqx_cm_locker.erl"},{line,51}]},{emqx_channel,process_connect,2,[{file,"emqx_channel.erl"},{line,616}]},{emqx_connection,with_channel,3,[{file,"emqx_connection.erl"},{line,790}]},{emqx_connection,process_msg,2,[{file,"emqx_connection.erl"},{line,466}]},{emqx_connection,process_msg,2,[{file,"emqx_connection.erl"},{line,472}]},{emqx_connection,handle_recv,3,[{file,"emqx_connection.erl"},{line,428}]},{proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,236}]}]}, offender: [{pid,<0.21482.112>},{name,connection},{mfargs,{emqx_connection,start_link,[#{enable_authn => true,limiter => #{bytes_in => #{capacity => 1099511627776,initial => 0,rate => infinity},client => #{bytes_in => #{capacity => 1099511627776,divisible => false,failure_strategy => force,initial => 0,low_watermark => 0,max_retry_time => 10000,rate => infinity},connection => #{capacity => 1099511627776,divisible => false,failure_strategy => force,initial => 0,low_watermark => 0,max_retry_time => 10000,rate => infinity},message_in => #{capacity => 1099511627776,divisible => false,failure_strategy => force,initial => 0,low_watermark => 0,max_retry_time => 10000,rate => infinity},message_routing => #{capacity => 1099511627776,divisible => false,failure_strategy => force,initial => 0,low_watermark => 0,max_retry_time => 10000,rate => infinity}},connection => #{capacity => 1000,initial => 0,rate => 100.0},message_in => #{capacit
y => 1099511627776,initial => 0,rate => infinity},message_routing => #{capacity => 1099511627776,initial => 0,rate => infinity}},listener => {tcp,default},zone => default}]}}]
2023-06-16T12:19:42.289682+00:00 [error] crasher: initial call: emqx_connection:init/4, pid: <0.21454.112>, registered_name: [], error: {{case_clause,{badrpc,{'EXIT',{undef,[{mria_lib,dirty_wrapper,[mnesia,dirty_write,emqx_channel_registry,[{channel,<<"MzExMTgxNDQwODk2NzU4NTc0NzU4NDk3MTc0MTc5MDIwODA">>,<0.21454.112>}]],[]},{gen_rpc_acceptor,call_middleman,3,[{file,"gen_rpc_acceptor.erl"},{line,287}]}]}}}},[{mria_lib,call_backend_rw_dirty,4,[{file,"mria_lib.erl"},{line,195}]},{emqx_cm,register_channel,3,[{file,"emqx_cm.erl"},{line,170}]},{emqx_cm,'-open_session/3-fun-0-',5,[{file,"emqx_cm.erl"},{line,275}]},{emqx_cm_locker,trans,3,[{file,"emqx_cm_locker.erl"},{line,51}]},{emqx_channel,process_connect,2,[{file,"emqx_channel.erl"},{line,616}]},{emqx_connection,with_channel,3,[{file,"emqx_connection.erl"},{line,790}]},{emqx_connection,process_msg,2,[{file,"emqx_connection.erl"},{line,466}]},{emqx_connection,process_msg,2,[{file,"emqx_connection.erl"},{line,472}]},{emqx_connection,handle_recv,3,[{file,"emqx_connection.erl"},{line,428}]},{proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,236}]}]}, ancestors: [<0.2302.0>,<0.2301.0>,esockd_sup,<0.1922.0>], message_queue_len: 0, messages: [], links: [<0.2302.0>], dictionary: [{guid,{1686917971290119,61711969178574,1}},{'$logger_metadata,#{clientid => <<"MzExMTgxNDQwODk2NzU4NTc0NzU4NDk3MTc0MTc5MDIwODA">>,peername => "192.168.188.21:49135"}},{recv_pkt,1},{incoming_bytes,91}], trap_exit: false, status: running, heap_size: 6772, stack_size: 29, reductions: 5447; neighbours:
2023-06-16T12:19:42.290499+00:00 [error] supervisor: 'esockd_connection_sup - <0.2302.0>', errorContext: connection_crashed, reason: {{case_clause,{badrpc,{'EXIT',{undef,[{mria_lib,dirty_wrapper,[mnesia,dirty_write,emqx_channel_registry,[{channel,<<"MzExMTgxNDQwODk2NzU4NTc0NzU4NDk3MTc0MTc5MDIwODA">>,<0.21454.112>}]],[]},{gen_rpc_acceptor,call_middleman,3,[{file,"gen_rpc_acceptor.erl"},{line,287}]}]}}}},[{mria_lib,call_backend_rw_dirty,4,[{file,"mria_lib.erl"},{line,195}]},{emqx_cm,register_channel,3,[{file,"emqx_cm.erl"},{line,170}]},{emqx_cm,'-open_session/3-fun-0-',5,[{file,"emqx_cm.erl"},{line,275}]},{emqx_cm_locker,trans,3,[{file,"emqx_cm_locker.erl"},{line,51}]},{emqx_channel,process_connect,2,[{file,"emqx_channel.erl"},{line,616}]},{emqx_connection,with_channel,3,[{file,"emqx_connection.erl"},{line,790}]},{emqx_connection,process_msg,2,[{file,"emqx_connection.erl"},{line,466}]},{emqx_connection,process_msg,2,[{file,"emqx_connection.erl"},{line,472}]},{emqx_connection,handle_recv,3,[{file,"emqx_connection.erl"},{line,428}]},{proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,236}]}]}, offender: [{pid,<0.21454.112>},{name,connection},{mfargs,{emqx_connection,start_link,[#{enable_authn => true,limiter => #{bytes_in => #{capacity => 1099511627776,initial => 0,rate => infinity},client => #{bytes_in => #{capacity => 1099511627776,divisible => false,failure_strategy => force,initial => 0,low_watermark => 0,max_retry_time => 10000,rate => infinity},connection => #{capacity => 1099511627776,divisible => false,failure_strategy => force,initial => 0,low_watermark => 0,max_retry_time => 10000,rate => infinity},message_in => #{capacity => 1099511627776,divisible => false,failure_strategy => force,initial => 0,low_watermark => 0,max_retry_time => 10000,rate => infinity},message_routing => #{capacity => 1099511627776,divisible => false,failure_strategy => force,initial => 0,low_watermark => 0,max_retry_time => 10000,rate => infinity}},connection => #{capacity => 1000,initial => 0,rate => 100.0}
,message_in => #{capacity => 1099511627776,initial => 0,rate => infinity},message_routing => #{capacity => 1099511627776,initial => 0,rate => infinity}},listener => {tcp,default},zone => default}]}}]
2023-06-16T12:19:42.296656+00:00 [error] exception: error, line: 120, mfa: emqx_pool:handle_cast/2, msg: async_submit_error, reason: {case_clause,{badrpc,{'EXIT',{undef,[{mria_lib,dirty_wrapper,[mnesia,dirty_delete_object,emqx_channel_registry,[{channel,<<"camel-53499728384901236">>,<0.19183.112>}]],[]},{gen_rpc_acceptor,call_middleman,3,[{file,"gen_rpc_acceptor.erl"},{line,287}]}]}}}}, stacktrace: [{mria_lib,call_backend_rw_dirty,4,[{file,"mria_lib.erl"},{line,195}]},{emqx_cm,do_unregister_channel,1,[{file,"emqx_cm.erl"},{line,182}]},{lists,foreach,2,[{file,"lists.erl"},{line,1342}]},{emqx_pool,handle_cast,2,[{file,"emqx_pool.erl"},{line,112}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,695}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,771}]},{proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,236}]}]
2023-06-16T12:19:42.297340+00:00 [error] exception: error, line: 120, mfa: emqx_pool:handle_cast/2, msg: async_submit_error, reason: {case_clause,{badrpc,{'EXIT',{undef,[{mria_lib,dirty_wrapper,[mnesia,dirty_delete_object,emqx_channel_registry,[{channel,<<"camel-paho354026589242">>,<0.19154.112>}]],[]},{gen_rpc_acceptor,call_middleman,3,[{file,"gen_rpc_acceptor.erl"},{line,287}]}]}}}}, stacktrace: [{mria_lib,call_backend_rw_dirty,4,[{file,"mria_lib.erl"},{line,195}]},{emqx_cm,do_unregister_channel,1,[{file,"emqx_cm.erl"},{line,182}]},{lists,foreach,2,[{file,"lists.erl"},{line,1342}]},{emqx_pool,handle_cast,2,[{file,"emqx_pool.erl"},{line,112}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,695}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,771}]},{proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,236}]}]
2023-06-16T12:19:42.297793+00:00 [error] crasher: initial call: emqx_connection:init/4, pid: <0.21459.112>, registered_name: [], error: {{case_clause,{badrpc,{'EXIT',{undef,[{mria_lib,dirty_wrapper,[mnesia,dirty_write,emqx_channel_registry,[{channel,<<"camel-paho354026589242">>,<0.21459.112>}]],[]},{gen_rpc_acceptor,call_middleman,3,[{file,"gen_rpc_acceptor.erl"},{line,287}]}]}}}},[{mria_lib,call_backend_rw_dirty,4,[{file,"mria_lib.erl"},{line,195}]},{emqx_cm,register_channel,3,[{file,"emqx_cm.erl"},{line,170}]},{emqx_cm,'-open_session/3-fun-0-',5,[{file,"emqx_cm.erl"},{line,275}]},{emqx_cm_locker,trans,3,[{file,"emqx_cm_locker.erl"},{line,51}]},{emqx_channel,process_connect,2,[{file,"emqx_channel.erl"},{line,616}]},{emqx_connection,with_channel,3,[{file,"emqx_connection.erl"},{line,790}]},{emqx_connection,process_msg,2,[{file,"emqx_connection.erl"},{line,466}]},{emqx_connection,process_msg,2,[{file,"emqx_connection.erl"},{line,472}]},{emqx_connection,handle_recv,3,[{file,"emqx_connection.erl"},{line,428}]},{proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,236}]}]}, ancestors: [<0.2302.0>,<0.2301.0>,esockd_sup,<0.1922.0>], message_queue_len: 0, messages: [], links: [<0.2302.0>], dictionary: [{guid,{1686917974139322,61711969178579,0}},{'$logger_metadata,#{clientid => <<"camel-paho354026589242">>,peername => "192.168.249.85:26806"}},{recv_pkt,1},{incoming_bytes,58}], trap_exit: false, status: running, heap_size: 4185, stack_size: 29, reductions: 4624; neighbours:
2023-06-16T12:19:42.298562+00:00 [error] supervisor: 'esockd_connection_sup - <0.2302.0>', errorContext: connection_crashed, reason: {{case_clause,{badrpc,{'EXIT',{undef,[{mria_lib,dirty_wrapper,[mnesia,dirty_write,emqx_channel_registry,[{channel,<<"camel-paho354026589242">>,<0.21459.112>}]],[]},{gen_rpc_acceptor,call_middleman,3,[{file,"gen_rpc_acceptor.erl"},{line,287}]}]}}}},[{mria_lib,call_backend_rw_dirty,4,[{file,"mria_lib.erl"},{line,195}]},{emqx_cm,register_channel,3,[{file,"emqx_cm.erl"},{line,170}]},{emqx_cm,'-open_session/3-fun-0-',5,[{file,"emqx_cm.erl"},{line,275}]},{emqx_cm_locker,trans,3,[{file,"emqx_cm_locker.erl"},{line,51}]},{emqx_channel,process_connect,2,[{file,"emqx_channel.erl"},{line,616}]},{emqx_connection,with_channel,3,[{file,"emqx_connection.erl"},{line,790}]},{emqx_connection,process_msg,2,[{file,"emqx_connection.erl"},{line,466}]},{emqx_connection,process_msg,2,[{file,"emqx_connection.erl"},{line,472}]},{emqx_connection,handle_recv,3,[{file,"emqx_connection.erl"},{line,428}]},{proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,236}]}]}, offender: [{pid,<0.21459.112>},{name,connection},{mfargs,{emqx_connection,start_link,[#{enable_authn => true,limiter => #{bytes_in => #{capacity => 1099511627776,initial => 0,rate => infinity},client => #{bytes_in => #{capacity => 1099511627776,divisible => false,failure_strategy => force,initial => 0,low_watermark => 0,max_retry_time => 10000,rate => infinity},connection => #{capacity => 1099511627776,divisible => false,failure_strategy => force,initial => 0,low_watermark => 0,max_retry_time => 10000,rate => infinity},message_in => #{capacity => 1099511627776,divisible => false,failure_strategy => force,initial => 0,low_watermark => 0,max_retry_time => 10000,rate => infinity},message_routing => #{capacity => 1099511627776,divisible => false,failure_strategy => force,initial => 0,low_watermark => 0,max_retry_time => 10000,rate => infinity}},connection => #{capacity => 1000,initial => 0,rate => 100.0},message_in => #{capacity
=> 1099511627776,initial => 0,rate => infinity},message_routing => #{capacity => 1099511627776,initial => 0,rate => infinity}},listener => {tcp,default},zone => default}]}}]
2023-06-16T12:19:42.299805+00:00 [error] crasher: initial call: emqx_connection:init/4, pid: <0.21483.112>, registered_name: [], error: {{case_clause,{badrpc,{'EXIT',{undef,[{mria_lib,dirty_wrapper,[mnesia,dirty_write,emqx_channel_registry,[{channel,<<"mosq/ezAMdMX0CyOuWDuod9">>,<0.21483.112>}]],[]},{gen_rpc_acceptor,call_middleman,3,[{file,"gen_rpc_acceptor.erl"},{line,287}]}]}}}},[{mria_lib,call_backend_rw_dirty,4,[{file,"mria_lib.erl"},{line,195}]},{emqx_cm,register_channel,3,[{file,"emqx_cm.erl"},{line,170}]},{emqx_cm,'-open_session/3-fun-0-',5,[{file,"emqx_cm.erl"},{line,275}]},{emqx_cm_locker,trans,3,[{file,"emqx_cm_locker.erl"},{line,51}]},{emqx_channel,process_connect,2,[{file,"emqx_channel.erl"},{line,616}]},{emqx_connection,with_channel,3,[{file,"emqx_connection.erl"},{line,790}]},{emqx_connection,process_msg,2,[{file,"emqx_connection.erl"},{line,466}]},{emqx_connection,process_msg,2,[{file,"emqx_connection.erl"},{line,472}]},{emqx_connection,handle_recv,3,[{file,"emqx_connection.erl"},{line,428}]},{proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,236}]}]}, ancestors: [<0.2302.0>,<0.2301.0>,esockd_sup,<0.1922.0>], message_queue_len: 0, messages: [], links: [<0.2302.0>], dictionary: [{guid,{1686917979935984,61711969178603,0}},{'$logger_metadata,#{clientid => <<"mosq/ezAMdMX0CyOuWDuod9">>,peername => "192.168.184.187:59648"}},{recv_pkt,1},{incoming_bytes,111}], trap_exit: false, status: running, heap_size: 1598, stack_size: 29, reductions: 5169; neighbours:
2023-06-16T12:19:42.300572+00:00 [error] supervisor: 'esockd_connection_sup - <0.2302.0>', errorContext: connection_crashed, reason: {{case_clause,{badrpc,{'EXIT',{undef,[{mria_lib,dirty_wrapper,[mnesia,dirty_write,emqx_channel_registry,[{channel,<<"mosq/ezAMdMX0CyOuWDuod9">>,<0.21483.112>}]],[]},{gen_rpc_acceptor,call_middleman,3,[{file,"gen_rpc_acceptor.erl"},{line,287}]}]}}}},[{mria_lib,call_backend_rw_dirty,4,[{file,"mria_lib.erl"},{line,195}]},{emqx_cm,register_channel,3,[{file,"emqx_cm.erl"},{line,170}]},{emqx_cm,'-open_session/3-fun-0-',5,[{file,"emqx_cm.erl"},{line,275}]},{emqx_cm_locker,trans,3,[{file,"emqx_cm_locker.erl"},{line,51}]},{emqx_channel,process_connect,2,[{file,"emqx_channel.erl"},{line,616}]},{emqx_connection,with_channel,3,[{file,"emqx_connection.erl"},{line,790}]},{emqx_connection,process_msg,2,[{file,"emqx_connection.erl"},{line,466}]},{emqx_connection,process_msg,2,[{file,"emqx_connection.erl"},{line,472}]},{emqx_connection,handle_recv,3,[{file,"emqx_connection.erl"},{line,428}]},{proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,236}]}]}, offender: [{pid,<0.21483.112>},{name,connection},{mfargs,{emqx_connection,start_link,[#{enable_authn => true,limiter => #{bytes_in => #{capacity => 1099511627776,initial => 0,rate => infinity},client => #{bytes_in => #{capacity => 1099511627776,divisible => false,failure_strategy => force,initial => 0,low_watermark => 0,max_retry_time => 10000,rate => infinity},connection => #{capacity => 1099511627776,divisible => false,failure_strategy => force,initial => 0,low_watermark => 0,max_retry_time => 10000,rate => infinity},message_in => #{capacity => 1099511627776,divisible => false,failure_strategy => force,initial => 0,low_watermark => 0,max_retry_time => 10000,rate => infinity},message_routing => #{capacity => 1099511627776,divisible => false,failure_strategy => force,initial => 0,low_watermark => 0,max_retry_time => 10000,rate => infinity}},connection => #{capacity => 1000,initial => 0,rate => 100.0},message_in => #{capacit
y => 1099511627776,initial => 0,rate => infinity},message_routing => #{capacity => 1099511627776,initial => 0,rate => infinity}},listener => {tcp,default},zone => default}]}}]
2023-06-16T12:19:42.823846+00:00 [error] connector: <<"emqx_authn_http:2">>, line: 328, mfa: emqx_connector_http:on_query/3, msg: http connector do request, received error response, request: {"/authenticate",[{<<"accept">>,<<"application/json">>},{<<"cache-control">>,<<"no-cache">>},{<<"connection">>,<<"keep-alive">>},{<<"content-type">>,<<"application/json">>},{<<"keep-alive">>,<<"timeout=30, max=1000">>}],<<"{\"username\":\"brenntag\",\"password\":\"Hf7vxM1GcvPQ28VqJ3fyXaXsAiwDd4v6MZTTYK8lJ4uaJYPZO7Kn_0khm8V2QZxa\"}">>}, status_code: 404
2023-06-16T12:19:42.824846+00:00 [error] id: <<"emqx_authn_http:2">>, line: 584, mfa: emqx_resource_worker:handle_query_result/4, msg: send_error, reason: #{headers => [{<<"content-type">>,<<"application/octet-stream">>},{<<"date">>,<<"Fri, 16 Jun 2023 12:19:42 GMT">>},{<<"content-length">>,<<"0">>}],status_code => 404}
2023-06-16T12:19:42.837750+00:00 [error] Generic server emqx_router_helper terminating. Reason: {{case_clause,{badrpc,{'EXIT',{undef,[{mria_lib,dirty_wrapper,[mnesia,dirty_delete,emqx_routing_node,['emqx@emqx-core-0.emqx-headless.emqx.svc.cluster.local']],[]},{gen_rpc_acceptor,call_middleman,3,[{file,"gen_rpc_acceptor.erl"},{line,287}]}]}}}},[{mria_lib,call_backend_rw_dirty,4,[{file,"mria_lib.erl"},{line,195}]},{emqx_router_helper,handle_info,2,[{file,"emqx_router_helper.erl"},{line,156}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,695}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,771}]},{proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,236}]}]}. Last message: {membership,{node,down,'emqx@emqx-core-0.emqx-headless.emqx.svc.cluster.local'}}. State: #{nodes => ['emqx@192.168.194.242','emqx@192.168.182.84','emqx@192.168.127.73']}.
2023-06-16T12:19:42.841894+00:00 [error] exception: error, line: 120, mfa: emqx_pool:handle_cast/2, msg: async_submit_error, reason: {case_clause,{badrpc,{'EXIT',{undef,[{mria_lib,dirty_wrapper,[mnesia,dirty_delete_object,emqx_channel_registry,[{channel,<<"mosq/DBqCNXgevmcViTmUHJ">>,<0.21308.112>}]],[]},{gen_rpc_acceptor,call_middleman,3,[{file,"gen_rpc_acceptor.erl"},{line,287}]}]}}}}, stacktrace: [{mria_lib,call_backend_rw_dirty,4,[{file,"mria_lib.erl"},{line,195}]},{emqx_cm,do_unregister_channel,1,[{file,"emqx_cm.erl"},{line,182}]},{lists,foreach,2,[{file,"lists.erl"},{line,1342}]},{emqx_pool,handle_cast,2,[{file,"emqx_pool.erl"},{line,112}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,695}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,771}]},{proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,236}]}]
2023-06-16T12:19:42.839872+00:00 [error] crasher: initial call: emqx_router_helper:init/1, pid: <0.2206.0>, registered_name: emqx_router_helper, error: {{case_clause,{badrpc,{'EXIT',{undef,[{mria_lib,dirty_wrapper,[mnesia,dirty_delete,emqx_routing_node,['emqx@emqx-core-0.emqx-headless.emqx.svc.cluster.local']],[]},{gen_rpc_acceptor,call_middleman,3,[{file,"gen_rpc_acceptor.erl"},{line,287}]}]}}}},[{mria_lib,call_backend_rw_dirty,4,[{file,"mria_lib.erl"},{line,195}]},{emqx_router_helper,handle_info,2,[{file,"emqx_router_helper.erl"},{line,156}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,695}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,771}]},{proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,236}]}]}, ancestors: [emqx_router_sup,emqx_sup,<0.2135.0>], message_queue_len: 1, messages: [{membership,{node,up,'emqx@emqx-core-0.emqx-headless.emqx.svc.cluster.local'}}], links: [<0.2205.0>], dictionary: [{rand_seed,{#{bits => 58,jump => #Fun<rand.3.92093067>,next => #Fun<rand.0.92093067>,type => exsss,uniform => #Fun<rand.1.92093067>,uniform_n => #Fun<rand.2.92093067>},[130499561858135813\|99243898912219086]}}], trap_exit: true, status: running, heap_size: 6772, stack_size: 29, reductions: 15369; neighbours:
2023-06-16T12:19:42.845157+00:00 [error] Supervisor: {local,emqx_router_sup}. Context: child_terminated. Reason: {{case_clause,{badrpc,{'EXIT',{undef,[{mria_lib,dirty_wrapper,[mnesia,dirty_delete,emqx_routing_node,['emqx@emqx-core-0.emqx-headless.emqx.svc.cluster.local']],[]},{gen_rpc_acceptor,call_middleman,3,[{file,"gen_rpc_acceptor.erl"},{line,287}]}]}}}},[{mria_lib,call_backend_rw_dirty,4,[{file,"mria_lib.erl"},{line,195}]},{emqx_router_helper,handle_info,2,[{file,"emqx_router_helper.erl"},{line,156}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,695}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,771}]},{proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,236}]}]}. Offender: id=helper,pid=<0.2206.0>.
2023-06-16T12:19:42.845505+00:00 [error] Supervisor: {local,emqx_router_sup}. Context: shutdown. Reason: reached_max_restart_intensity. Offender: id=helper,pid=<0.2206.0>.
2023-06-16T12:19:42.846439+00:00 [error] Supervisor: {local,emqx_sup}. Context: child_terminated. Reason: shutdown. Offender: id=emqx_router_sup,pid=<0.2205.0>.
2023-06-16T12:19:42.846608+00:00 [error] Supervisor: {local,emqx_sup}. Context: shutdown. Reason: reached_max_restart_intensity. Offender: id=emqx_router_sup,pid=<0.2205.0>.
Listener ssl:default on 0.0.0.0:8883 stopped.
Listener ssl:vapp on 0.0.0.0:8884 stopped.
Listener tcp:default on 0.0.0.0:1883 stopped.
Listener ws:Websocket on 0.0.0.0:8080 stopped.
2023-06-16T12:19:42.966537+00:00 [error] State machine 'emqx_resource_manager_emqx_authz_http:4:5' terminating. Reason: {noproc,{gen_server,call,[emqx_alarm,{deactivate_alarm,<<"emqx_authz_http:4">>,no_details,<<>>}]}}. Stack: [{gen_server,call,2,[{file,"gen_server.erl"},{line,239}]},{emqx_resource_manager,terminate,3,[{file,"emqx_resource_manager.erl"},{line,313}]},{gen_statem,terminate,7,[{file,"gen_statem.erl"},{line,2365}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]. Last event: {{call,{<0.21522.112>,[alias\|#Ref<0.673987955.3244621825.88064>]}},{remove,true}}. State: {connected,{data,<<"emqx_authz_http:4">>,<<"emqx_authz_http:4:5">>,<<"emqx_authz">>,emqx_connector_http,async_if_possible,sync,#{base_path_templete => [{str,<<"authorize">>}],base_query_template => [],base_url => #{host => "emqx.iotbuilder.svc.cluster.local",path => "/",port => 4195,scheme => http},body => #{action => <<"${action}">>,topic => <<"${topic}">>,username => <<"${username}">>},body_template => [{tuple,[{value,action},{tmpl,[{var,{var,<<"action">>}}]}]},{tuple,[{value,topic},{tmpl,[{var,{var,<<"topic">>}}]}]},{tuple,[{value,username},{tmpl,[{var,{var,<<"username">>}}]}]}],connect_timeout => 5000,enable => true,enable_pipelining => 100,headers => [{<<"accept">>,<<"application/json">>},{<<"cache-control">>,<<"no-cache">>},{<<"connection">>,<<"keep-alive">>},{<<"content-type">>,<<"application/json">>},{<<"keep-alive">>,<<"timeout=30, max=1000">>}],method => post,pool_size => 8,pool_type => random,request_timeout => 5000,ssl => #{ciphers => [],depth => 10,enable => false,reuse_sessions => true,secure_renegotiate => true,user_lookup_fun => {fun emqx_tls_psk:lookup/3,undefined},verify => verify_peer,versions => ['tlsv1.3','tlsv1.2','tlsv1.1',tlsv1]},type => http,url => <<"http://emqx.iotbuilder.svc.cluster.local:4195/authorize">>},#{start_after_created => false},connected,#{base_path => "/",connect_timeout => 5000,host => "emqx.iotbuilder.svc.cluster.local",pool_name => 'emqx_authz_http:4:5
',port => 4195,request => undefined},undefined}}. Client <0.21522.112> stacktrace: [{gen,do_call,4,[{file,"gen.erl"},{line,233}]},{gen_statem,'-call_clean/4-fun-0-',5,[{file,"gen_statem.erl"},{line,725}]}].
2023-06-16T12:19:42.967515+00:00 [error] crasher: initial call: emqx_resource_manager:init/1, pid: <0.2770.0>, registered_name: 'emqx_resource_manager_emqx_authz_http:4:5', exit: {{noproc,{gen_server,call,[emqx_alarm,{deactivate_alarm,<<"emqx_authz_http:4">>,no_details,<<>>}]}},[{gen_server,call,2,[{file,"gen_server.erl"},{line,239}]},{emqx_resource_manager,terminate,3,[{file,"emqx_resource_manager.erl"},{line,313}]},{gen_statem,terminate,7,[{file,"gen_statem.erl"},{line,2365}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}, ancestors: [emqx_resource_manager_sup,emqx_resource_sup,<0.2633.0>], message_queue_len: 0, messages: [], links: [<0.2636.0>], dictionary: [], trap_exit: true, status: running, heap_size: 17731, stack_size: 29, reductions: 433017; neighbours:
2023-06-16T12:19:42.968175+00:00 [error] Supervisor: {local,emqx_resource_manager_sup}. Context: child_terminated. Reason: {noproc,{gen_server,call,[emqx_alarm,{deactivate_alarm,<<"emqx_authz_http:4">>,no_details,<<>>}]}}. Offender: id=emqx_resource_manager,pid=<0.2770.0>.
Stop listener http:dashboard on :18083 successfully.
[os_mon] memory supervisor port (memsup): Erlang has closed
[os_mon] cpu supervisor port (cpu_sup): Erlang has closed
{"Kernel pid terminated",application_controller,"{application_terminated,emqx,shutdown}"}
Kernel pid terminated (application_controller) ({application_terminated,emqx,shutdown})
Crash dump is being written to: log/erl_crash.dump...doneNew 5.0.26 does not start:
MQX_RPC__PORT_DISCOVERY [rpc.port_discovery]: manual
--
EMQX_NODE__DB_ROLE [node.role]: replicant
EMQX_NODE__COOKIE [node.cookie]: ******
EMQX_NODE__NAME [node.name]: emqx@192.168.113.194
2023-06-16T12:20:08.196488+00:00 [error] Mria(Membership): Failed to ping emqx@192.168.127.73
2023-06-16T12:20:13.206475+00:00 [error] Mria(Membership): Failed to ping emqx@192.168.182.84
2023-06-16T12:20:18.215509+00:00 [error] Mria(Membership): Failed to ping emqx@192.168.194.242
2023-06-16T12:20:23.235442+00:00 [error] Mria(Membership): Failed to ping emqx@192.168.127.73
2023-06-16T12:20:28.244571+00:00 [error] Mria(Membership): Failed to ping emqx@192.168.182.84
2023-06-16T12:20:33.256060+00:00 [error] Mria(Membership): Failed to ping emqx@192.168.194.242
2023-06-16T12:20:38.257525+00:00 [error] State machine '$mria_meta_shard' terminating. Reason: {timeout,{gen_server,call,[mria_lb,{probe,'emqx@emqx-core-0.emqx-headless.emqx.svc.cluster.local','$mria_meta_shard'}]}}. Stack: [{gen_server,call,2,[{file,"gen_server.erl"},{line,239}]},{mria_rlog,subscribe,4,[{file,"mria_rlog.erl"},{line,167}]},{mria_rlog_replica,try_connect,3,[{file,"mria_rlog_replica.erl"},{line,395}]},{mria_rlog_replica,handle_reconnect,1,[{file,"mria_rlog_replica.erl"},{line,341}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1205}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]. Last event: {state_timeout,reconnect}. State: {disconnected,{d,'$mria_meta_shard',<0.1979.0>,undefined,undefined,undefined,0,undefined,undefined,false}}.
2023-06-16T12:20:38.257971+00:00 [error] crasher: initial call: mria_rlog_replica:init/1, pid: <0.1980.0>, registered_name: '$mria_meta_shard', exit: {{timeout,{gen_server,call,[mria_lb,{probe,'emqx@emqx-core-0.emqx-headless.emqx.svc.cluster.local','$mria_meta_shard'}]}},[{gen_server,call,2,[{file,"gen_server.erl"},{line,239}]},{mria_rlog,subscribe,4,[{file,"mria_rlog.erl"},{line,167}]},{mria_rlog_replica,try_connect,3,[{file,"mria_rlog_replica.erl"},{line,395}]},{mria_rlog_replica,handle_reconnect,1,[{file,"mria_rlog_replica.erl"},{line,341}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1205}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}, ancestors: [<0.1979.0>,mria_shards_sup,mria_rlog_sup,mria_sup,<0.1898.0>], message_queue_len: 0, messages: [], links: [<0.1979.0>], dictionary: [{rand_seed,{#{bits => 58,jump => #Fun<rand.3.92093067>,next => #Fun<rand.0.92093067>,type => exsss,uniform => #Fun<rand.1.92093067>,uniform_n => #Fun<rand.2.92093067>},[174577254786938427\|159900066860853580]}},{'$logger_metadata,#{domain => [mria,rlog,replica],shard => '$mria_meta_shard'}}], trap_exit: true, status: running, heap_size: 1598, stack_size: 29, reductions: 12282; neighbours:
2023-06-16T12:20:38.258593+00:00 [error] Supervisor: {<0.1979.0>,mria_replicant_shard_sup}. Context: child_terminated. Reason: {timeout,{gen_server,call,[mria_lb,{probe,'emqx@emqx-core-0.emqx-headless.emqx.svc.cluster.local','$mria_meta_shard'}]}}. Offender: id=replica,pid=<0.1980.0>.
2023-06-16T12:20:38.258770+00:00 [error] Supervisor: {<0.1979.0>,mria_replicant_shard_sup}. Context: shutdown. Reason: reached_max_restart_intensity. Offender: id=replica,pid=<0.1980.0>.
2023-06-16T12:20:38.258977+00:00 [error] Supervisor: {local,mria_shards_sup}. Context: child_terminated. Reason: shutdown. Offender: id='$mria_meta_shard',pid=<0.1979.0>.
2023-06-16T12:20:38.304534+00:00 [error] Mria(Membership): Failed to ping emqx@192.168.127.73
2023-06-16T12:20:43.313461+00:00 [error] Mria(Membership): Failed to ping emqx@192.168.182.84
2023-06-16T12:20:48.322408+00:00 [error] Mria(Membership): Failed to ping emqx@192.168.194.242
2023-06-16T12:20:53.333453+00:00 [error] Mria(Membership): Failed to ping emqx@192.168.127.73
2023-06-16T12:20:58.341422+00:00 [error] Mria(Membership): Failed to ping emqx@192.168.182.84
2023-06-16T12:21:03.356689+00:00 [error] Mria(Membership): Failed to ping emqx@192.168.194.242
2023-06-16T12:21:08.358129+00:00 [error] State machine '$mria_meta_shard' terminating. Reason: {timeout,{gen_server,call,[mria_lb,{probe,'emqx@emqx-core-0.emqx-headless.emqx.svc.cluster.local','$mria_meta_shard'}]}}. Stack: [{gen_server,call,2,[{file,"gen_server.erl"},{line,239}]},{mria_rlog,subscribe,4,[{file,"mria_rlog.erl"},{line,167}]},{mria_rlog_replica,try_connect,3,[{file,"mria_rlog_replica.erl"},{line,395}]},{mria_rlog_replica,handle_reconnect,1,[{file,"mria_rlog_replica.erl"},{line,341}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1205}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]. Last event: {state_timeout,reconnect}. State: {disconnected,{d,'$mria_meta_shard',<0.2064.0>,undefined,undefined,undefined,0,undefined,undefined,false}}.
2023-06-16T12:21:08.358671+00:00 [error] crasher: initial call: mria_rlog_replica:init/1, pid: <0.2065.0>, registered_name: '$mria_meta_shard', exit: {{timeout,{gen_server,call,[mria_lb,{probe,'emqx@emqx-core-0.emqx-headless.emqx.svc.cluster.local','$mria_meta_shard'}]}},[{gen_server,call,2,[{file,"gen_server.erl"},{line,239}]},{mria_rlog,subscribe,4,[{file,"mria_rlog.erl"},{line,167}]},{mria_rlog_replica,try_connect,3,[{file,"mria_rlog_replica.erl"},{line,395}]},{mria_rlog_replica,handle_reconnect,1,[{file,"mria_rlog_replica.erl"},{line,341}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1205}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}, ancestors: [<0.2064.0>,mria_shards_sup,mria_rlog_sup,mria_sup,<0.1898.0>], message_queue_len: 0, messages: [], links: [<0.2064.0>], dictionary: [{rand_seed,{#{bits => 58,jump => #Fun<rand.3.92093067>,next => #Fun<rand.0.92093067>,type => exsss,uniform => #Fun<rand.1.92093067>,uniform_n => #Fun<rand.2.92093067>},[105177099337860761\|134422031376837027]}},{'$logger_metadata,#{domain => [mria,rlog,replica],shard => '$mria_meta_shard'}}], trap_exit: true, status: running, heap_size: 10958, stack_size: 29, reductions: 12427; neighbours:
2023-06-16T12:21:08.359287+00:00 [error] Supervisor: {<0.2064.0>,mria_replicant_shard_sup}. Context: child_terminated. Reason: {timeout,{gen_server,call,[mria_lb,{probe,'emqx@emqx-core-0.emqx-headless.emqx.svc.cluster.local','$mria_meta_shard'}]}}. Offender: id=replica,pid=<0.2065.0>.
2023-06-16T12:21:08.359466+00:00 [error] Supervisor: {<0.2064.0>,mria_replicant_shard_sup}. Context: shutdown. Reason: reached_max_restart_intensity. Offender: id=replica,pid=<0.2065.0>.
2023-06-16T12:21:08.359650+00:00 [error] Supervisor: {local,mria_shards_sup}. Context: child_terminated. Reason: shutdown. Offender: id='$mria_meta_shard',pid=<0.2064.0>.
2023-06-16T12:21:08.387456+00:00 [error] Mria(Membership): Failed to ping emqx@192.168.127.73
2023-06-16T12:21:13.396476+00:00 [error] Mria(Membership): Failed to ping emqx@192.168.182.84
2023-06-16T12:21:18.408489+00:00 [error] Mria(Membership): Failed to ping emqx@192.168.194.242
2023-06-16T12:21:23.422469+00:00 [error] Mria(Membership): Failed to ping emqx@192.168.127.73
2023-06-16T12:21:28.430487+00:00 [error] Mria(Membership): Failed to ping emqx@192.168.182.84
2023-06-16T12:21:33.439416+00:00 [error] Mria(Membership): Failed to ping emqx@192.168.194.242
@ieQu1 Could you please take a look, I see "Different Mria version on the core node", so Is the jump from 5.0.14 to 5.0.26 too big ? @zmstone I remember you mentioned this risk to me, right?
mariusstaicu
commented
1 year ago I jumped from 5.0.17 to 5.0.26 today with no problems.
ieQu1
commented
1 year ago Hello all,
Mria (our embedded DB) sometimes has incompatible changes in the protocol. In order to prevent data corruption, replicant nodes will refuse to connect to the core node that doesn't run the compatible version.
Because of that, rolling upgrade of EMQX cluster should be done in batches. Let's assume we have N cores and M replicants. Stages of upgrade should look like this:
When this procedure is used, the replicants will always have at least one core node that runs a compatible version of the code.
webfrank
commented
1 year ago Hi, i supposed was something like this but once the core get upgraded why the new replicant does not connect? I agree the old replicants get disconnected but not the new one, same version as the core.
Another point is how the operator handle this?
Il giorno sab 17 giu 2023 alle 11:24 ieQu1 @.***> ha scritto:
Hello all,
Mria (our embedded DB) sometimes has incompatible changes in the protocol. In order to prevent data corruption, replicant nodes will refuse to connect to the core node that doesn't run the compatible version.
Because of that, rolling upgrade of EMQX cluster should be done in batches. Let's assume we have N cores and M replicants. Stages of upgrade should look like this:
- Upgrade 1 core node of N
- Upgrade M/N replicants
- Repeat until all nodes are upgraded
When this procedure is used, the replicants will always have at least one core node that runs a compatible version of the code.
— Reply to this email directly, view it on GitHub https://github.com/emqx/emqx-operator/issues/809#issuecomment-1595689569, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABDYWGHM4HVQI5NTBI3OATDXLVZV7ANCNFSM6AAAAAAZI2BGWY . You are receiving this because you authored the thread.Message ID: @.***>
Rory-Z
commented
1 year ago Hi, @ieQu1 As far as I see, EMQX Operator didn't do this, but we are doing this part of the work.
In our current plan, we will first update the core node by rolling, and then update the replication node after all the core nodes are updated, which is also the plan suggested by @zmstone.
So is your proposed solution like this:
Do I understand it right?
ieQu1
commented
1 year ago Yes, this is correct.
Rory-Z
commented
1 year ago Yes, this is correct.
In k8s, I usually manage core nodes as a set of Pods. In the first step above, if I create a new set of core node Pods to join the cluster, it may cause the number of core nodes to be multiplied by 2. Will this have any impact, especially if I cooperate with node evacuation to transfer MQTT connections?
Describe the bug Not able to upgrade EMQX Replicant version
To Reproduce Deploy a cluster with EMQX Operator 2.1.0, upgrade Operator to 2.1.3, change EMQX version from 5.0.14 to 5.0.26. The core get upgraded, the replicants sticks in mria cannot ping the nodes.
Expected behavior A rolling upgrade of all nodes
Anything else we need to know?:
Environment details::