rabbitmq / rabbitmq-management

RabbitMQ Management UI and HTTP API
https://www.rabbitmq.com/management.html
Other
370 stars 165 forks source link

`badarg` errors in metrics_collector processes in the log #411

Closed laurentluce closed 6 years ago

laurentluce commented 7 years ago

RMQ 3.6.10 Erlang 19.3.4

=ERROR REPORT==== 4-Jun-2017::01:45:28 ===

** Generic server channel_exchange_metrics_metrics_collector terminating
** Last message in was collect_metrics
** When Server state == {state,channel_exchange_metrics,5000,
                               {[{60,20}],[{5,5}],[{60,5}]},
                               none,
                               #Fun<rabbit_mgmt_metrics_collector.0.6133563>,
                               #Fun<rabbit_mgmt_metrics_collector.1.6133563>,
                               {dict,0,16,16,8,80,48,
                                     {[],[],[],[],[],[],[],[],[],[],[],[],[],
                                      [],[],[]},
                                     {{[],[],[],[],[],[],[],[],[],[],[],[],[],
                                       [],[],[]}}}}
** Reason for termination ==
** {badarg,
       [{ets,safe_fixtable,[channel_exchange_metrics,true],[]},
        {ets,foldl,3,[{file,"ets.erl"},{line,571}]},

        {rabbit_mgmt_metrics_collector,aggregate_metrics,2,
            [{file,"src/rabbit_mgmt_metrics_collector.erl"},{line,158}]},
        {rabbit_mgmt_metrics_collector,handle_info,2,
            [{file,"src/rabbit_mgmt_metrics_collector.erl"},{line,102}]},
        {gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,601}]},
        {gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,667}]},

        {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}

=ERROR REPORT==== 4-Jun-2017::01:45:28 ===

** Generic server node_persister_metrics_metrics_collector terminating
** Last message in was collect_metrics
** When Server state == {state,node_persister_metrics,5000,
                               {[{60,20}],[{5,5}],[{60,5}]},
                               none,
                               #Fun<rabbit_mgmt_metrics_collector.0.6133563>,
                               #Fun<rabbit_mgmt_metrics_collector.1.6133563>,
                               {dict,0,16,16,8,80,48,
                                     {[],[],[],[],[],[],[],[],[],[],[],[],[],
                                      [],[],[]},
                                     {{[],[],[],[],[],[],[],[],[],[],[],[],[],
                                       [],[],[]}}}}
** Reason for termination ==
** {badarg,
       [{ets,safe_fixtable,[node_persister_metrics,true],[]},
        {ets,foldl,3,[{file,"ets.erl"},{line,571}]},

        {rabbit_mgmt_metrics_collector,aggregate_metrics,2,
            [{file,"src/rabbit_mgmt_metrics_collector.erl"},{line,158}]},
        {rabbit_mgmt_metrics_collector,handle_info,2,
            [{file,"src/rabbit_mgmt_metrics_collector.erl"},{line,102}]},
        {gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,601}]},
        {gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,667}]},

        {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}

=ERROR REPORT==== 4-Jun-2017::01:45:28 ===

** Generic server node_coarse_metrics_metrics_collector terminating
** Last message in was collect_metrics
** When Server state == {state,node_coarse_metrics,5000,
                               {[{60,20}],[{5,5}],[{60,5}]},
                               none,
                               #Fun<rabbit_mgmt_metrics_collector.0.6133563>,
                               #Fun<rabbit_mgmt_metrics_collector.1.6133563>,
                               {dict,0,16,16,8,80,48,
                                     {[],[],[],[],[],[],[],[],[],[],[],[],[],
                                      [],[],[]},
                                     {{[],[],[],[],[],[],[],[],[],[],[],[],[],
                                       [],[],[]}}}}
** Reason for termination ==
** {badarg,
       [{ets,safe_fixtable,[node_coarse_metrics,true],[]},
        {ets,foldl,3,[{file,"ets.erl"},{line,571}]},

        {rabbit_mgmt_metrics_collector,aggregate_metrics,2,
            [{file,"src/rabbit_mgmt_metrics_collector.erl"},{line,158}]},
        {rabbit_mgmt_metrics_collector,handle_info,2,
            [{file,"src/rabbit_mgmt_metrics_collector.erl"},{line,102}]},
        {gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,601}]},
        {gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,667}]},

        {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}

=ERROR REPORT==== 4-Jun-2017::01:45:28 ===

** Generic server channel_queue_exchange_metrics_metrics_collector terminating
** Last message in was collect_metrics
** When Server state == {state,channel_queue_exchange_metrics,5000,
                               {[{60,20}],[{5,5}],[{60,5}]},
                               none,
                               #Fun<rabbit_mgmt_metrics_collector.0.6133563>,
                               #Fun<rabbit_mgmt_metrics_collector.1.6133563>,
                               {dict,0,16,16,8,80,48,
                                     {[],[],[],[],[],[],[],[],[],[],[],[],[],
                                      [],[],[]},
                                     {{[],[],[],[],[],[],[],[],[],[],[],[],[],
                                       [],[],[]}}}}
** Reason for termination ==
** {badarg,
       [{ets,safe_fixtable,[channel_queue_exchange_metrics,true],[]},
        {ets,foldl,3,[{file,"ets.erl"},{line,571}]},

        {rabbit_mgmt_metrics_collector,aggregate_metrics,2,
            [{file,"src/rabbit_mgmt_metrics_collector.erl"},{line,158}]},
        {rabbit_mgmt_metrics_collector,handle_info,2,
            [{file,"src/rabbit_mgmt_metrics_collector.erl"},{line,102}]},
        {gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,601}]},
        {gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,667}]},

        {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}

=ERROR REPORT==== 4-Jun-2017::01:45:28 ===

** Generic server channel_queue_metrics_metrics_collector terminating
** Last message in was collect_metrics
** When Server state == {state,channel_queue_metrics,5000,
                               {[{60,20}],[{5,5}],[{60,5}]},
                               none,
                               #Fun<rabbit_mgmt_metrics_collector.0.6133563>,
                               #Fun<rabbit_mgmt_metrics_collector.1.6133563>,
                               {dict,0,16,16,8,80,48,
                                     {[],[],[],[],[],[],[],[],[],[],[],[],[],
                                      [],[],[]},
                                     {{[],[],[],[],[],[],[],[],[],[],[],[],[],
                                       [],[],[]}}}}
** Reason for termination ==
** {badarg,
       [{ets,safe_fixtable,[channel_queue_metrics,true],[]},
        {ets,foldl,3,[{file,"ets.erl"},{line,571}]},

        {rabbit_mgmt_metrics_collector,aggregate_metrics,2,
            [{file,"src/rabbit_mgmt_metrics_collector.erl"},{line,158}]},
        {rabbit_mgmt_metrics_collector,handle_info,2,
            [{file,"src/rabbit_mgmt_metrics_collector.erl"},{line,102}]},
        {gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,601}]},
        {gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,667}]},

        {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}

=ERROR REPORT==== 4-Jun-2017::01:45:28 ===

** Generic server node_node_metrics_metrics_collector terminating
** Last message in was collect_metrics
** When Server state == {state,node_node_metrics,5000,
                               {[{60,20}],[{5,5}],[{60,5}]},
                               none,
                               #Fun<rabbit_mgmt_metrics_collector.0.6133563>,
                               #Fun<rabbit_mgmt_metrics_collector.1.6133563>,
                               {dict,0,16,16,8,80,48,
                                     {[],[],[],[],[],[],[],[],[],[],[],[],[],
                                      [],[],[]},
                                     {{[],[],[],[],[],[],[],[],[],[],[],[],[],
                                       [],[],[]}}}}
** Reason for termination ==
** {badarg,
       [{ets,safe_fixtable,[node_node_metrics,true],[]},
        {ets,foldl,3,[{file,"ets.erl"},{line,571}]},

        {rabbit_mgmt_metrics_collector,aggregate_metrics,2,
            [{file,"src/rabbit_mgmt_metrics_collector.erl"},{line,158}]},
        {rabbit_mgmt_metrics_collector,handle_info,2,
            [{file,"src/rabbit_mgmt_metrics_collector.erl"},{line,102}]},
        {gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,601}]},
        {gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,667}]},

        {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}
$ sudo rabbitmqctl report
Reporting server status on {{2017,6,5},{13,15,40}}

Status of node 'rabbit@ip-10-0-90-245'
[{pid,3166},
 {running_applications,
     [{autocluster,"RabbitMQ Automatic Clustering Plugin","0.6.1"},
      {rabbit,"RabbitMQ","3.6.10"},
      {mnesia,"MNESIA  CXC 138 12","4.14.3"},
      {rabbitmq_aws,
          "A light-weight, relatively unopinionated AWS API client for RabbitMQ plugins",
          "0.1.2"},
      {rabbitmq_web_stomp,"Rabbit WEB-STOMP - WebSockets to Stomp adapter",
          "3.6.10"},
      {rabbitmq_stomp,"RabbitMQ STOMP plugin","3.6.10"},
      {sockjs,"SockJS","0.3.4"},
      {rabbitmq_management,"RabbitMQ Management Console","3.6.10"},
      {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.6.10"},
      {cowboy,"Small, fast, modular HTTP server.","1.0.4"},
      {rabbitmq_management_agent,"RabbitMQ Management Agent","3.6.10"},
      {inets,"INETS  CXC 138 49","6.3.9"},
      {amqp_client,"RabbitMQ AMQP Client","3.6.10"},
      {cowlib,"Support library for manipulating Web protocols.","1.0.2"},
      {os_mon,"CPO  CXC 138 46","2.4.2"},
      {rabbit_common,
          "Modules shared by rabbitmq-server and rabbitmq-erlang-client",
          "3.6.10"},
      {ranch,"Socket acceptor pool for TCP protocols.","1.3.0"},
      {ssl,"Erlang/OTP SSL application","8.1.3"},
      {public_key,"Public key infrastructure","1.4"},
      {asn1,"The Erlang ASN1 compiler version 4.0.4","4.0.4"},
      {compiler,"ERTS  CXC 138 10","7.0.4"},
      {syntax_tools,"Syntax tools","2.1.1"},
      {xmerl,"XML parser","1.3.13"},
      {crypto,"CRYPTO","3.7.4"},
      {sasl,"SASL  CXC 138 11","3.0.3"},
      {stdlib,"ERTS  CXC 138 10","3.3"},
      {kernel,"ERTS  CXC 138 10","5.2"}]},
 {os,{unix,linux}},
 {erlang_version,
     "Erlang/OTP 19 [erts-8.3.3] [source] [64-bit] [async-threads:64] [hipe] [kernel-poll:true]\n"},
 {memory,
     [{total,75075808},
      {connection_readers,0},
      {connection_writers,0},
      {connection_channels,0},
      {connection_other,1112184},
      {queue_procs,1178224},
      {queue_slave_procs,583960},
      {plugins,3976344},
      {other_proc,27786304},
      {mnesia,759960},
      {metrics,106928},
      {mgmt_db,326448},
      {msg_index,72712},
      {other_ets,2787376},
      {binary,2224104},
      {code,25345587},
      {atom,1041593},
      {other_system,7878324}]},
 {alarms,[]},
 {listeners,[{clustering,25672,"::"},{amqp,5672,"::"}]},
 {vm_memory_high_watermark,0.75},
 {vm_memory_limit,1445895168},
 {disk_free_limit,50000000},
 {disk_free,6009643008},
 {file_descriptors,
     [{total_limit,65436},
      {total_used,13},
      {sockets_limit,58890},
      {sockets_used,11}]},
 {processes,[{limit,1048576},{used,660}]},
 {run_queue,0},
 {uptime,107090},
 {kernel,{net_ticktime,60}}]

Cluster status of node 'rabbit@ip-10-0-90-245'
[{nodes,[{disc,['rabbit@ip-10-0-26-214','rabbit@ip-10-0-90-245']}]},
 {running_nodes,['rabbit@ip-10-0-26-214','rabbit@ip-10-0-90-245']},
 {cluster_name,<<"rabbit@ip-10-0-90-177.ec2.internal">>},
 {partitions,[]},
 {alarms,[{'rabbit@ip-10-0-26-214',[]},{'rabbit@ip-10-0-90-245',[]}]}]

Application environment of node 'rabbit@ip-10-0-90-245'
[{amqp_client,[{prefer_ipv6,false},{ssl_options,[]}]},
 {asn1,[]},
 {autocluster,
     [{autocluster_failure,ignore},
      {autocluster_log_level,info},
      {aws_autoscaling,true},
      {backend,aws},
      {cleanup_warn_only,false},
      {cluster_cleanup,true}]},
 {compiler,[]},
 {cowboy,[]},
 {cowlib,[]},
 {crypto,[]},
 {inets,[]},
 {kernel,
     [{error_logger,tty},
      {inet_default_connect_options,[{nodelay,true}]},
      {inet_dist_listen_max,25672},
      {inet_dist_listen_min,25672}]},
 {mnesia,
     [{dir,"/var/lib/rabbitmq/mnesia/rabbit@ip-10-0-90-245"},
      {dump_log_write_threshold,256}]},
 {os_mon,
     [{start_cpu_sup,false},
      {start_disksup,false},
      {start_memsup,false},
      {start_os_sup,false}]},
 {public_key,[]},
 {rabbit,
     [{auth_backends,[rabbit_auth_backend_internal]},
      {auth_mechanisms,['PLAIN','AMQPLAIN']},
      {background_gc_enabled,false},
      {background_gc_target_interval,60000},
      {backing_queue_module,rabbit_priority_queue},
      {channel_max,0},
      {channel_operation_timeout,15000},
      {cluster_keepalive_interval,10000},
      {cluster_nodes,{[],disc}},
      {cluster_partition_handling,autoheal},
      {collect_statistics,coarse},
      {collect_statistics_interval,60000},
      {config_entry_decoder,
          [{cipher,aes_cbc256},
           {hash,sha512},
           {iterations,1000},
           {passphrase,undefined}]},
      {credit_flow_default_credit,{400,200}},
      {default_permissions,[<<".*">>,<<".*">>,<<".*">>]},
      {default_user,<<"guest">>},
      {default_user_tags,[administrator]},
      {default_vhost,<<"/">>},
      {delegate_count,16},
      {disk_free_limit,50000000},
      {disk_monitor_failure_retries,10},
      {disk_monitor_failure_retry_interval,120000},
      {enabled_plugins_file,"/etc/rabbitmq/enabled_plugins"},
      {error_logger,{file,"/var/log/rabbitmq/rabbit@ip-10-0-90-245.log"}},
      {fhc_read_buffering,false},
      {fhc_write_buffering,true},
      {frame_max,131072},
      {halt_on_upgrade_failure,true},
      {handshake_timeout,10000},
      {heartbeat,60},
      {hipe_compile,false},
      {hipe_modules,
          [rabbit_reader,rabbit_channel,gen_server2,rabbit_exchange,
           rabbit_command_assembler,rabbit_framing_amqp_0_9_1,rabbit_basic,
           rabbit_event,lists,queue,priority_queue,rabbit_router,rabbit_trace,
           rabbit_misc,rabbit_binary_parser,rabbit_exchange_type_direct,
           rabbit_guid,rabbit_net,rabbit_amqqueue_process,
           rabbit_variable_queue,rabbit_binary_generator,rabbit_writer,
           delegate,gb_sets,lqueue,sets,orddict,rabbit_amqqueue,
           rabbit_limiter,gb_trees,rabbit_queue_index,
           rabbit_exchange_decorator,gen,dict,ordsets,file_handle_cache,
           rabbit_msg_store,array,rabbit_msg_store_ets_index,rabbit_msg_file,
           rabbit_exchange_type_fanout,rabbit_exchange_type_topic,mnesia,
           mnesia_lib,rpc,mnesia_tm,qlc,sofs,proplists,credit_flow,pmon,
           ssl_connection,tls_connection,ssl_record,tls_record,gen_fsm,ssl]},
      {lazy_queue_explicit_gc_run_operation_threshold,1000},
      {log_levels,[{connection,info},{autocluster,info}]},
      {loopback_users,[<<"guest">>]},
      {memory_monitor_interval,2500},
      {mirroring_flow_control,true},
      {mirroring_sync_batch_size,4096},
      {mnesia_table_loading_retry_limit,10},
      {mnesia_table_loading_retry_timeout,30000},
      {msg_store_credit_disc_bound,{4000,800}},
      {msg_store_file_size_limit,16777216},
      {msg_store_index_module,rabbit_msg_store_ets_index},
      {msg_store_io_batch_size,4096},
      {num_ssl_acceptors,1},
      {num_tcp_acceptors,10},
      {password_hashing_module,rabbit_password_hashing_sha256},
      {plugins_dir,
          "/usr/lib/rabbitmq/plugins:/usr/lib/rabbitmq/lib/rabbitmq_server-3.6.10/plugins"},
      {plugins_expand_dir,
          "/var/lib/rabbitmq/mnesia/rabbit@ip-10-0-90-245-plugins-expand"},
      {queue_explicit_gc_run_operation_threshold,1000},
      {queue_index_embed_msgs_below,4096},
      {queue_index_max_journal_entries,32768},
      {reverse_dns_lookups,false},
      {sasl_error_logger,
          {file,"/var/log/rabbitmq/rabbit@ip-10-0-90-245-sasl.log"}},
      {server_properties,[]},
      {ssl_allow_poodle_attack,false},
      {ssl_apps,[asn1,crypto,public_key,ssl]},
      {ssl_cert_login_from,distinguished_name},
      {ssl_handshake_timeout,5000},
      {ssl_listeners,[]},
      {ssl_options,[]},
      {tcp_listen_options,
          [{backlog,128},
           {nodelay,true},
           {linger,{true,0}},
           {exit_on_close,false}]},
      {tcp_listeners,[5672]},
      {trace_vhosts,[]},
      {vm_memory_high_watermark,0.75},
      {vm_memory_high_watermark_paging_ratio,0.9}]},
 {rabbit_common,[]},
 {rabbitmq_aws,[]},
 {rabbitmq_management,
     [{cors_allow_origins,[]},
      {cors_max_age,1800},
      {http_log_dir,none},
      {listener,
          [{port,15672},
           {ssl,false},
           {ssl_opts,
               [{cacertfile,"/etc/rabbitmq/ssl/cacert.pem"},
                {certfile,"/etc/rabbitmq/ssl/server.pem"},
                {keyfile,"/etc/rabbitmq/ssl/server.key"}]}]},
      {load_definitions,none},
      {management_db_cache_multiplier,5},
      {process_stats_gc_timeout,300000},
      {rates_mode,none},
      {sample_retention_policies,
          [{global,[{60,5}]},{basic,[{60,20}]},{detailed,[{5,5}]}]},
      {stats_event_max_backlog,250}]},
 {rabbitmq_management_agent,
     [{rates_mode,basic},
      {sample_retention_policies,
          [{global,[{605,5},{3660,60},{29400,600},{86400,1800}]},
           {basic,[{605,5},{3600,60}]},
           {detailed,[{605,5}]}]}]},
 {rabbitmq_stomp,
     [{default_user,[{login,<<"guest">>},{passcode,<<"guest">>}]},
      {default_vhost,<<"/">>},
      {hide_server_info,false},
      {implicit_connect,false},
      {num_ssl_acceptors,1},
      {num_tcp_acceptors,10},
      {ssl_cert_login,false},
      {ssl_listeners,[]},
      {tcp_listen_options,[{backlog,128},{nodelay,true}]},
      {tcp_listeners,[61613]},
      {trailing_lf,true}]},
 {rabbitmq_web_dispatch,[]},
 {rabbitmq_web_stomp,
     [{cowboy_opts,[]},
      {num_ssl_acceptors,1},
      {num_tcp_acceptors,10},
      {port,15671},
      {sockjs_opts,[]},
      {ssl_config,
          [{port,15674},
           {backlog,1024},
           {recbuf,32768},
           {sndbuf,32768},
           {cacertfile,"/etc/rabbitmq/ssl/cacert.pem"},
           {certfile,"/etc/rabbitmq/ssl/server.pem"},
           {keyfile,"/etc/rabbitmq/ssl/server.key"}]},
      {tcp_config,[]},
      {use_http_auth,true},
      {ws_frame,text}]},
 {ranch,[]},
 {sasl,[{errlog_type,error},{sasl_error_logger,false}]},
 {sockjs,[]},
 {ssl,[{protocol_version,['tlsv1.2','tlsv1.1',tlsv1]}]},
 {stdlib,[]},
 {syntax_tools,[]},
 {xmerl,[]}]

Status of node 'rabbit@ip-10-0-26-214'
[{pid,3185},
 {running_applications,
     [{autocluster,"RabbitMQ Automatic Clustering Plugin","0.6.1"},
      {rabbit,"RabbitMQ","3.6.10"},
      {mnesia,"MNESIA  CXC 138 12","4.14.3"},
      {rabbitmq_aws,
          "A light-weight, relatively unopinionated AWS API client for RabbitMQ plugins",
          "0.1.2"},
      {rabbitmq_web_stomp,"Rabbit WEB-STOMP - WebSockets to Stomp adapter",
          "3.6.10"},
      {rabbitmq_stomp,"RabbitMQ STOMP plugin","3.6.10"},
      {sockjs,"SockJS","0.3.4"},
      {rabbitmq_management,"RabbitMQ Management Console","3.6.10"},
      {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.6.10"},
      {cowboy,"Small, fast, modular HTTP server.","1.0.4"},
      {rabbitmq_management_agent,"RabbitMQ Management Agent","3.6.10"},
      {inets,"INETS  CXC 138 49","6.3.9"},
      {amqp_client,"RabbitMQ AMQP Client","3.6.10"},
      {cowlib,"Support library for manipulating Web protocols.","1.0.2"},
      {os_mon,"CPO  CXC 138 46","2.4.2"},
      {rabbit_common,
          "Modules shared by rabbitmq-server and rabbitmq-erlang-client",
          "3.6.10"},
      {ranch,"Socket acceptor pool for TCP protocols.","1.3.0"},
      {ssl,"Erlang/OTP SSL application","8.1.3"},
      {public_key,"Public key infrastructure","1.4"},
      {asn1,"The Erlang ASN1 compiler version 4.0.4","4.0.4"},
      {compiler,"ERTS  CXC 138 10","7.0.4"},
      {syntax_tools,"Syntax tools","2.1.1"},
      {xmerl,"XML parser","1.3.13"},
      {crypto,"CRYPTO","3.7.4"},
      {sasl,"SASL  CXC 138 11","3.0.3"},
      {stdlib,"ERTS  CXC 138 10","3.3"},
      {kernel,"ERTS  CXC 138 10","5.2"}]},
 {os,{unix,linux}},
 {erlang_version,
     "Erlang/OTP 19 [erts-8.3.3] [source] [64-bit] [async-threads:64] [hipe] [kernel-poll:true]\n"},
 {memory,
     [{total,76313512},
      {connection_readers,0},
      {connection_writers,0},
      {connection_channels,0},
      {connection_other,1172160},
      {queue_procs,999808},
      {queue_slave_procs,837168},
      {plugins,5386312},
      {other_proc,26631080},
      {mnesia,760408},
      {metrics,169808},
      {mgmt_db,620464},
      {msg_index,72728},
      {other_ets,2867016},
      {binary,2681408},
      {code,25345587},
      {atom,1041593},
      {other_system,7895092}]},
 {alarms,[]},
 {listeners,[{clustering,25672,"::"},{amqp,5672,"::"}]},
 {vm_memory_high_watermark,0.75},
 {vm_memory_limit,1445895168},
 {disk_free_limit,50000000},
 {disk_free,6012657664},
 {file_descriptors,
     [{total_limit,65436},
      {total_used,14},
      {sockets_limit,58890},
      {sockets_used,12}]},
 {processes,[{limit,1048576},{used,728}]},
 {run_queue,0},
 {uptime,127829},
 {kernel,{net_ticktime,60}}]

Cluster status of node 'rabbit@ip-10-0-26-214'
[{nodes,[{disc,['rabbit@ip-10-0-26-214','rabbit@ip-10-0-90-245']}]},
 {running_nodes,['rabbit@ip-10-0-90-245','rabbit@ip-10-0-26-214']},
 {cluster_name,<<"rabbit@ip-10-0-90-177.ec2.internal">>},
 {partitions,[]},
 {alarms,[{'rabbit@ip-10-0-90-245',[]},{'rabbit@ip-10-0-26-214',[]}]}]

Application environment of node 'rabbit@ip-10-0-26-214'
[{amqp_client,[{prefer_ipv6,false},{ssl_options,[]}]},
 {asn1,[]},
 {autocluster,
     [{autocluster_failure,ignore},
      {autocluster_log_level,info},
      {aws_autoscaling,true},
      {backend,aws},
      {cleanup_warn_only,false},
      {cluster_cleanup,true}]},
 {compiler,[]},
 {cowboy,[]},
 {cowlib,[]},
 {crypto,[]},
 {inets,[]},
 {kernel,
     [{error_logger,tty},
      {inet_default_connect_options,[{nodelay,true}]},
      {inet_dist_listen_max,25672},
      {inet_dist_listen_min,25672}]},
 {mnesia,
     [{dir,"/var/lib/rabbitmq/mnesia/rabbit@ip-10-0-26-214"},
      {dump_log_write_threshold,256}]},
 {os_mon,
     [{start_cpu_sup,false},
      {start_disksup,false},
      {start_memsup,false},
      {start_os_sup,false}]},
 {public_key,[]},
 {rabbit,
     [{auth_backends,[rabbit_auth_backend_internal]},
      {auth_mechanisms,['PLAIN','AMQPLAIN']},
      {background_gc_enabled,false},
      {background_gc_target_interval,60000},
      {backing_queue_module,rabbit_priority_queue},
      {channel_max,0},
      {channel_operation_timeout,15000},
      {cluster_keepalive_interval,10000},
      {cluster_nodes,{[],disc}},
      {cluster_partition_handling,autoheal},
      {collect_statistics,coarse},
      {collect_statistics_interval,60000},
      {config_entry_decoder,
          [{cipher,aes_cbc256},
           {hash,sha512},
           {iterations,1000},
           {passphrase,undefined}]},
      {credit_flow_default_credit,{400,200}},
      {default_permissions,[<<".*">>,<<".*">>,<<".*">>]},
      {default_user,<<"guest">>},
      {default_user_tags,[administrator]},
      {default_vhost,<<"/">>},
      {delegate_count,16},
      {disk_free_limit,50000000},
      {disk_monitor_failure_retries,10},
      {disk_monitor_failure_retry_interval,120000},
      {enabled_plugins_file,"/etc/rabbitmq/enabled_plugins"},
      {error_logger,{file,"/var/log/rabbitmq/rabbit@ip-10-0-26-214.log"}},
      {fhc_read_buffering,false},
      {fhc_write_buffering,true},
      {frame_max,131072},
      {halt_on_upgrade_failure,true},
      {handshake_timeout,10000},
      {heartbeat,60},
      {hipe_compile,false},
      {hipe_modules,
          [rabbit_reader,rabbit_channel,gen_server2,rabbit_exchange,
           rabbit_command_assembler,rabbit_framing_amqp_0_9_1,rabbit_basic,
           rabbit_event,lists,queue,priority_queue,rabbit_router,rabbit_trace,
           rabbit_misc,rabbit_binary_parser,rabbit_exchange_type_direct,
           rabbit_guid,rabbit_net,rabbit_amqqueue_process,
           rabbit_variable_queue,rabbit_binary_generator,rabbit_writer,
           delegate,gb_sets,lqueue,sets,orddict,rabbit_amqqueue,
           rabbit_limiter,gb_trees,rabbit_queue_index,
           rabbit_exchange_decorator,gen,dict,ordsets,file_handle_cache,
           rabbit_msg_store,array,rabbit_msg_store_ets_index,rabbit_msg_file,
           rabbit_exchange_type_fanout,rabbit_exchange_type_topic,mnesia,
           mnesia_lib,rpc,mnesia_tm,qlc,sofs,proplists,credit_flow,pmon,
           ssl_connection,tls_connection,ssl_record,tls_record,gen_fsm,ssl]},
      {lazy_queue_explicit_gc_run_operation_threshold,1000},
      {log_levels,[{connection,info},{autocluster,info}]},
      {loopback_users,[<<"guest">>]},
      {memory_monitor_interval,2500},
      {mirroring_flow_control,true},
      {mirroring_sync_batch_size,4096},
      {mnesia_table_loading_retry_limit,10},
      {mnesia_table_loading_retry_timeout,30000},
      {msg_store_credit_disc_bound,{4000,800}},
      {msg_store_file_size_limit,16777216},
      {msg_store_index_module,rabbit_msg_store_ets_index},
      {msg_store_io_batch_size,4096},
      {num_ssl_acceptors,1},
      {num_tcp_acceptors,10},
      {password_hashing_module,rabbit_password_hashing_sha256},
      {plugins_dir,
          "/usr/lib/rabbitmq/plugins:/usr/lib/rabbitmq/lib/rabbitmq_server-3.6.10/plugins"},
      {plugins_expand_dir,
          "/var/lib/rabbitmq/mnesia/rabbit@ip-10-0-26-214-plugins-expand"},
      {queue_explicit_gc_run_operation_threshold,1000},
      {queue_index_embed_msgs_below,4096},
      {queue_index_max_journal_entries,32768},
      {reverse_dns_lookups,false},
      {sasl_error_logger,
          {file,"/var/log/rabbitmq/rabbit@ip-10-0-26-214-sasl.log"}},
      {server_properties,[]},
      {ssl_allow_poodle_attack,false},
      {ssl_apps,[asn1,crypto,public_key,ssl]},
      {ssl_cert_login_from,distinguished_name},
      {ssl_handshake_timeout,5000},
      {ssl_listeners,[]},
      {ssl_options,[]},
      {tcp_listen_options,
          [{backlog,128},
           {nodelay,true},
           {linger,{true,0}},
           {exit_on_close,false}]},
      {tcp_listeners,[5672]},
      {trace_vhosts,[]},
      {vm_memory_high_watermark,0.75},
      {vm_memory_high_watermark_paging_ratio,0.9}]},
 {rabbit_common,[]},
 {rabbitmq_aws,[]},
 {rabbitmq_management,
     [{cors_allow_origins,[]},
      {cors_max_age,1800},
      {http_log_dir,none},
      {listener,
          [{port,15672},
           {ssl,false},
           {ssl_opts,
               [{cacertfile,"/etc/rabbitmq/ssl/cacert.pem"},
                {certfile,"/etc/rabbitmq/ssl/server.pem"},
                {keyfile,"/etc/rabbitmq/ssl/server.key"}]}]},
      {load_definitions,none},
      {management_db_cache_multiplier,5},
      {process_stats_gc_timeout,300000},
      {rates_mode,none},
      {sample_retention_policies,
          [{global,[{60,5}]},{basic,[{60,20}]},{detailed,[{5,5}]}]},
      {stats_event_max_backlog,250}]},
 {rabbitmq_management_agent,
     [{rates_mode,basic},
      {sample_retention_policies,
          [{global,[{605,5},{3660,60},{29400,600},{86400,1800}]},
           {basic,[{605,5},{3600,60}]},
           {detailed,[{605,5}]}]}]},
 {rabbitmq_stomp,
     [{default_user,[{login,<<"guest">>},{passcode,<<"guest">>}]},
      {default_vhost,<<"/">>},
      {hide_server_info,false},
      {implicit_connect,false},
      {num_ssl_acceptors,1},
      {num_tcp_acceptors,10},
      {ssl_cert_login,false},
      {ssl_listeners,[]},
      {tcp_listen_options,[{backlog,128},{nodelay,true}]},
      {tcp_listeners,[61613]},
      {trailing_lf,true}]},
 {rabbitmq_web_dispatch,[]},
 {rabbitmq_web_stomp,
     [{cowboy_opts,[]},
      {num_ssl_acceptors,1},
      {num_tcp_acceptors,10},
      {port,15671},
      {sockjs_opts,[]},
      {ssl_config,
          [{port,15674},
           {backlog,1024},
           {recbuf,32768},
           {sndbuf,32768},
           {cacertfile,"/etc/rabbitmq/ssl/cacert.pem"},
           {certfile,"/etc/rabbitmq/ssl/server.pem"},
           {keyfile,"/etc/rabbitmq/ssl/server.key"}]},
      {tcp_config,[]},
      {use_http_auth,true},
      {ws_frame,text}]},
 {ranch,[]},
 {sasl,[{errlog_type,error},{sasl_error_logger,false}]},
 {sockjs,[]},
 {ssl,[{protocol_version,['tlsv1.2','tlsv1.1',tlsv1]}]},
 {stdlib,[]},
 {syntax_tools,[]},
 {xmerl,[]}]
michaelklishin commented 7 years ago

I tried reproducing this by taking the effective config, replacing TLS certificate paths, removing all data directory and log path keys as well as HiPE and backing queue module settings (which are all defaults):

[{amqp_client,[{prefer_ipv6,false},{ssl_options,[]}]},
 {asn1,[]},
 {autocluster,
     [{autocluster_failure,ignore},
      {autocluster_log_level,info},
      {aws_autoscaling,true},
      {backend,aws},
      {cleanup_warn_only,false},
      {cluster_cleanup,true}]},
 {compiler,[]},
 {cowboy,[]},
 {cowlib,[]},
 {crypto,[]},
 {inets,[]},
 {kernel,
     [{error_logger,tty},
      {inet_default_connect_options,[{nodelay,true}]},
      {inet_dist_listen_max,25672},
      {inet_dist_listen_min,25672}]},
 {os_mon,
     [{start_cpu_sup,false},
      {start_disksup,false},
      {start_memsup,false},
      {start_os_sup,false}]},
 {public_key,[]},
 {rabbit,
     [{auth_backends,[rabbit_auth_backend_internal]},
      {auth_mechanisms,['PLAIN','AMQPLAIN']},
      {background_gc_enabled,false},
      {background_gc_target_interval,60000},
      {channel_max,0},
      {channel_operation_timeout,15000},
      {cluster_keepalive_interval,10000},
      {cluster_nodes,{[],disc}},
      {cluster_partition_handling,autoheal},
      {collect_statistics,coarse},
      {collect_statistics_interval,60000},
      {config_entry_decoder,
          [{cipher,aes_cbc256},
           {hash,sha512},
           {iterations,1000},
           {passphrase,undefined}]},
      {credit_flow_default_credit,{400,200}},
      {default_permissions,[<<".*">>,<<".*">>,<<".*">>]},
      {default_user,<<"guest">>},
      {default_user_tags,[administrator]},
      {default_vhost,<<"/">>},
      {delegate_count,16},
      {disk_free_limit,50000000},
      {disk_monitor_failure_retries,10},
      {disk_monitor_failure_retry_interval,120000},
      {fhc_read_buffering,false},
      {fhc_write_buffering,true},
      {frame_max,131072},
      {halt_on_upgrade_failure,true},
      {handshake_timeout,10000},
      {heartbeat,60},
      {hipe_compile,false},
      {lazy_queue_explicit_gc_run_operation_threshold,1000},
      {log_levels,[{connection,info},{autocluster,info}]},
      {loopback_users,[<<"guest">>]},
      {memory_monitor_interval,2500},
      {mirroring_flow_control,true},
      {mirroring_sync_batch_size,4096},
      {mnesia_table_loading_retry_limit,10},
      {mnesia_table_loading_retry_timeout,30000},
      {msg_store_credit_disc_bound,{4000,800}},
      {msg_store_file_size_limit,16777216},
      {msg_store_index_module,rabbit_msg_store_ets_index},
      {msg_store_io_batch_size,4096},
      {num_ssl_acceptors,1},
      {num_tcp_acceptors,10},
      {password_hashing_module,rabbit_password_hashing_sha256},
      {queue_explicit_gc_run_operation_threshold,1000},
      {queue_index_embed_msgs_below,4096},
      {queue_index_max_journal_entries,32768},
      {reverse_dns_lookups,false},
      {server_properties,[]},
      {ssl_allow_poodle_attack,false},
      {ssl_apps,[asn1,crypto,public_key,ssl]},
      {ssl_cert_login_from,distinguished_name},
      {ssl_handshake_timeout,5000},
      {ssl_listeners,[]},
      {ssl_options,[]},
      {tcp_listen_options,
          [{backlog,128},
           {nodelay,true},
           {linger,{true,0}},
           {exit_on_close,false}]},
      {tcp_listeners,[5672]},
      {trace_vhosts,[]},
      {vm_memory_high_watermark,0.75},
      {vm_memory_high_watermark_paging_ratio,0.9}]},
 {rabbit_common,[]},
 {rabbitmq_management,
     [{cors_allow_origins,[]},
      {cors_max_age,1800},
      {http_log_dir,none},
      {listener,
          [{port,15672},
           {ssl,false},
           {ssl_opts,
               [
                {cacertfile, "/Users/antares/Development/Opensource/tls-gen.git/basic/result/ca_certificate.pem"},
                {certfile,   "/Users/antares/Development/Opensource/tls-gen.git/basic/result/server_certificate.pem"},
                {keyfile,    "/Users/antares/Development/Opensource/tls-gen.git/basic/result/server_key.pem"}
                ]}]},
      {load_definitions,none},
      {management_db_cache_multiplier,5},
      {process_stats_gc_timeout,300000},
      {rates_mode,none},
      {sample_retention_policies,
          [{global,[{60,5}]},{basic,[{60,20}]},{detailed,[{5,5}]}]},
      {stats_event_max_backlog,250}]},
 {rabbitmq_management_agent,
     [{rates_mode,basic},
      {sample_retention_policies,
          [{global,[{605,5},{3660,60},{29400,600},{86400,1800}]},
           {basic,[{605,5},{3600,60}]},
           {detailed,[{605,5}]}]}]},
 {rabbitmq_stomp,
     [{default_user,[{login,<<"guest">>},{passcode,<<"guest">>}]},
      {default_vhost,<<"/">>},
      {hide_server_info,false},
      {implicit_connect,false},
      {num_ssl_acceptors,1},
      {num_tcp_acceptors,10},
      {ssl_cert_login,false},
      {ssl_listeners,[]},
      {tcp_listen_options,[{backlog,128},{nodelay,true}]},
      {tcp_listeners,[61613]},
      {trailing_lf,true}]},
 {rabbitmq_web_dispatch,[]},
 {rabbitmq_web_stomp,
     [{cowboy_opts,[]},
      {num_ssl_acceptors,1},
      {num_tcp_acceptors,10},
      {port,15671},
      {sockjs_opts,[]},
      {ssl_config,
          [{port,15674},
           {backlog,1024},
           {recbuf,32768},
           {sndbuf,32768},
           {cacertfile, "/Users/antares/Development/Opensource/tls-gen.git/basic/result/ca_certificate.pem"},
           {certfile,   "/Users/antares/Development/Opensource/tls-gen.git/basic/result/server_certificate.pem"},
           {keyfile,    "/Users/antares/Development/Opensource/tls-gen.git/basic/result/server_key.pem"}]},
      {tcp_config,[]},
      {use_http_auth,true},
      {ws_frame,text}]},
 {ranch,[]},
 {sasl,[{errlog_type,error},{sasl_error_logger,false}]},
 {sockjs,[]},
 {ssl,[{protocol_version,['tlsv1.2','tlsv1.1',tlsv1]}]},
 {stdlib,[]},
 {syntax_tools,[]},
 {xmerl,[]}].

I cannot reproduce the issue.

@laurentluce can you share a Dockerfile or a VM image that reproduces this, or at least your config file? does the problem happens if you don't use rabbitmq-autocluster 0.6.1 (that plugin hijacks RabbitMQ boot process, which can lead to plugin initialization issues such as missing ETS tables).

laurentluce commented 7 years ago

@michaelklishin So far, I did not see the issue without the autocluster plugin, running one node only. In clustering mode, I am using the latest autocluster plugin from https://github.com/rhoegg/rabbitmq-autocluster/releases. Do you recommend using a different version?

michaelklishin commented 7 years ago

@laurentluce see https://github.com/rabbitmq/rabbitmq-autocluster#current-maintainers. 3 members of our team have worked on rabbitmq-autocluster since 3.6.7 and I don't remember this coming up.

@dcorbacho @Gsantomaggio do you? ^^^

michaelklishin commented 7 years ago

New rabbitmq-autocluster releases are announced on rabbitmq-users.

dcorbacho commented 7 years ago

@michaelklishin I haven't seen that with the latest releases

Gsantomaggio commented 7 years ago

me neither

laurentluce commented 7 years ago

@michaelklishin I am going to try with rabbitmq-autocluster 0.7.0 and see how it works.

michaelklishin commented 7 years ago

@laurentluce does this have any observable effect on the management plugin or is it just log noise?

laurentluce commented 7 years ago

@michaelklishin I started seeing issues with connections not reported by the management plugin (/api/connections). I would bring a new web-stomp client and its connection would not always be reported even when the connection was alive. This is when I tried to reset the mgmt DB and you pointed me to Erlang 20 not being supported. After that, I switched to erlang-rpm 19.3.4 and continued seeing the error log messages described above.

laurentluce commented 7 years ago

@michaelklishin I ran some tests with 2 to 4 nodes + rabbitmq-autocluster 0.7.0 and I see the badarg error on about 50% of the nodes. The first error usually happens after the autocluster random startup pause so it seems to match your hypothesis.

=INFO REPORT==== 9-Jun-2017::00:17:55 ===
node           : rabbit@ip-10-0-90-119
home dir       : /var/lib/rabbitmq
config file(s) : /etc/rabbitmq/rabbitmq.config
cookie hash    : svshrPKw+J4rCUxya03Kog==
log            : /var/log/rabbitmq/rabbit@ip-10-0-90-119.log
sasl log       : /var/log/rabbitmq/rabbit@ip-10-0-90-119-sasl.log
database dir   : /var/lib/rabbitmq/mnesia/rabbit@ip-10-0-90-119

=INFO REPORT==== 9-Jun-2017::00:17:55 ===
autocluster: Delaying startup for 1173ms.

=ERROR REPORT==== 9-Jun-2017::00:17:56 ===
** Generic server channel_exchange_metrics_metrics_collector terminating
** Last message in was collect_metrics
** When Server state == {state,channel_exchange_metrics,5000,
                               {[{60,20}],[{5,5}],[{60,5}]},
                               none,
                               #Fun<rabbit_mgmt_metrics_collector.0.6133563>,
                               #Fun<rabbit_mgmt_metrics_collector.1.6133563>,
                               {dict,0,16,16,8,80,48,
                                     {[],[],[],[],[],[],[],[],[],[],[],[],[],
                                      [],[],[]},
                                     {{[],[],[],[],[],[],[],[],[],[],[],[],[],
                                       [],[],[]}}}}
** Reason for termination ==
** {badarg,
       [{ets,safe_fixtable,[channel_exchange_metrics,true],[]},
        {ets,foldl,3,[{file,"ets.erl"},{line,571}]},
        {rabbit_mgmt_metrics_collector,aggregate_metrics,2,
            [{file,"src/rabbit_mgmt_metrics_collector.erl"},{line,158}]},
        {rabbit_mgmt_metrics_collector,handle_info,2,
            [{file,"src/rabbit_mgmt_metrics_collector.erl"},{line,102}]},
        {gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,601}]},
        {gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,667}]},
        {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}
laurentluce commented 7 years ago

@michaelklishin I ran more tests this time with 3.6.6 (instead of 3.6.10) and rabbitmq-autocluster 0.7.0. I see less errors. I see one instead of 5-6 errors. The error is seen when the autocluster plugin starts:

=INFO REPORT==== 11-Jun-2017::19:02:16 ===
autocluster: Delaying startup for 2163ms.

=ERROR REPORT==== 11-Jun-2017::19:02:18 ===
** Generic server rabbit_mgmt_external_stats terminating 
** Last message in was emit_update
** When Server state == {state,65536,
                            [{{io_file_handle_open_attempt,count},10},
                             {{io_file_handle_open_attempt,time},468},
                             {{io_read,bytes},1},
                             {{io_read,count},1},
                             {{io_read,time},344},
                             {{io_reopen,count},0},
                             {{io_seek,count},0},
                             {{io_seek,time},0},
                             {{io_sync,count},0},
                             {{io_sync,time},0},
                             {{io_write,bytes},0},
                             {{io_write,count},0},
                             {{io_write,time},0},
                             {{mnesia_disk_tx,count},12},
                             {{mnesia_ram_tx,count},41},
                             {{msg_store_read,count},0},
                             {{msg_store_write,count},0},
                             {{queue_index_journal_write,count},0},
                             {{queue_index_read,count},0},
                             {{queue_index_write,count},0}],
                            {set,1,16,16,8,80,48,
                                {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                 []},
                                {{[],[],[],[],[],[],[],[],[],[],[],
                                  [{'rabbitmq-cli-45@ip-10-0-26-122',
                                       <0.583.0>}],
                                  [],[],[],[]}}},
                            undefined}
** Reason for termination == 
** {badarg,
       [{ets,match_object,[file_handle_cache_stats,'_'],[]},
        {ets,tab2list,1,[{file,"ets.erl"},{line,710}]},
        {file_handle_cache_stats,get,0,
            [{file,"src/file_handle_cache_stats.erl"},{line,59}]},
        {rabbit_mgmt_external_stats,update_state,1,
            [{file,"src/rabbit_mgmt_external_stats.erl"},{line,381}]},
        {rabbit_mgmt_external_stats,emit_update,1,
            [{file,"src/rabbit_mgmt_external_stats.erl"},{line,359}]},
        {rabbit_mgmt_external_stats,handle_info,2,
            [{file,"src/rabbit_mgmt_external_stats.erl"},{line,347}]},
        {gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,601}]},
        {gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,667}]}]}
michaelklishin commented 7 years ago

3.6.6 uses a different data model for stats, so this is not really an indication of anything.

rabbitmq-autocluster hijacks the boot process in a way that creates a race condition in plugin initialization.

On Sun, Jun 11, 2017 at 10:13 PM, Laurent Luce notifications@github.com wrote:

@michaelklishin https://github.com/michaelklishin I ran more tests this time with 3.6.6 and rabbitmq-autocluster 0.7.0. I see less errors. I see one instead of 5-6. It is again around the time the autocluster plugin starts:

=INFO REPORT==== 11-Jun-2017::19:02:16 === autocluster: Delaying startup for 2163ms.

=ERROR REPORT==== 11-Jun-2017::19:02:18 === Generic server rabbit_mgmt_external_stats terminating Last message in was emit_update When Server state == {state,65536, [{{io_file_handle_open_attempt,count},10}, {{io_file_handle_open_attempt,time},468}, {{io_read,bytes},1}, {{io_read,count},1}, {{io_read,time},344}, {{io_reopen,count},0}, {{io_seek,count},0}, {{io_seek,time},0}, {{io_sync,count},0}, {{io_sync,time},0}, {{io_write,bytes},0}, {{io_write,count},0}, {{io_write,time},0}, {{mnesia_disk_tx,count},12}, {{mnesia_ram_tx,count},41}, {{msg_store_read,count},0}, {{msg_store_write,count},0}, {{queue_index_journal_write,count},0}, {{queue_index_read,count},0}, {{queue_index_write,count},0}], {set,1,16,16,8,80,48, {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[], []}, {{[],[],[],[],[],[],[],[],[],[],[], [{'rabbitmq-cli-45@ip-10-0-26-122', <0.583.0>}], [],[],[],[]}}}, undefined} Reason for termination == ** {badarg, [{ets,match_object,[file_handle_cachestats,''],[]}, {ets,tab2list,1,[{file,"ets.erl"},{line,710}]}, {file_handle_cache_stats,get,0, [{file,"src/file_handle_cache_stats.erl"},{line,59}]}, {rabbit_mgmt_external_stats,update_state,1, [{file,"src/rabbit_mgmt_external_stats.erl"},{line,381}]}, {rabbit_mgmt_external_stats,emit_update,1, [{file,"src/rabbit_mgmt_external_stats.erl"},{line,359}]}, {rabbit_mgmt_external_stats,handle_info,2, [{file,"src/rabbit_mgmt_external_stats.erl"},{line,347}]}, {gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,601}]}, {gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,667}]}]}

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/rabbitmq/rabbitmq-management/issues/411#issuecomment-307650060, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAEQlWKpWtMa40ZtpwG4OM1861rro_xks5sDDxZgaJpZM4NwB26 .

-- MK

Staff Software Engineer, Pivotal/RabbitMQ

laurentluce commented 7 years ago

@michaelklishin Do you recommend delaying even more the rabbitmq-autocluster initialization to avoid this issue?

michaelklishin commented 7 years ago

rabbitmq-autocluster won't start RabbitMQ until after the delay has passed, so it shouldn't matter. You can definitely try.

On Sat, Jun 17, 2017 at 1:15 AM, Laurent Luce notifications@github.com wrote:

@michaelklishin https://github.com/michaelklishin Do you recommend delaying even more the rabbitmq-autocluster initialization to avoid this issue?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/rabbitmq/rabbitmq-management/issues/411#issuecomment-309146785, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAEQj-iXZzagO7QSGMnzSlzVGm8daMQks5sEv6GgaJpZM4NwB26 .

-- MK

Staff Software Engineer, Pivotal/RabbitMQ

michaelklishin commented 6 years ago

This is no longer an issue we see mentioned, possibly because peer discovery is now a core feature and therefore plugin initialisation order/race conditions no longer apply.