search

noxdafox / rabbitmq-cloudwatch-exporter

RabbitMQ Plugin for publishing cluster metrics to AWS CloudWatch
Mozilla Public License 2.0
42 stars 9 forks source link

problem exporting metrics #11

Closed westymalik closed 5 years ago

westymalik commented 5 years ago

Hi,

Getting a crash while exporting the metrics.

My rabbitmq.conf':

                cloudwatch_exporter.aws.region = "eu-west-1"
                cloudwatch_exporter.metrics.1 = overview
                cloudwatch_exporter.metrics.2 = vhost
                cloudwatch_exporter.metrics.3 = node
                log.file.level = debug
** Generic server rabbitmq_cloudwatch_exporter terminating
** Last message in was export_metrics
** When Server state == [[{period,60},{collectors,[overview,vhost,node]},{namespace,<<"RabbitMQ">>}],[{region,<<"eu-west-1">>}]]
** Reason for termination ==
** {#{'__exception__' => true,'__struct__' => 'Elixir.ExAws.Error',message => <<"ExAws Request Error!\n\n{:error, {:option, :server_only, :honor_cipher_order}}\n">>},[{'Elixir.ExAws','request!',2,[{file,"lib/ex_aws.ex"},{line,66}]},{'Elixir.Enum','-map/2-lists^map/1-0-',2,[{file,"lib/enum.ex"},{line,1327}]},{'Elixir.RabbitMQCloudWatchExporter.Exporter',handle_info,2,[{file,"lib/rabbitmq_cloudwatch_exporter/exporter.ex"},{line,60}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,637}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,711}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}
2019-09-30 08:22:11 =CRASH REPORT====
  crasher:
    initial call: Elixir.RabbitMQCloudWatchExporter.Exporter:init/1
    pid: <0.697.0>
    registered_name: []
    exception error: {#{'__exception__' => true,'__struct__' => 'Elixir.ExAws.Error',message => <<"ExAws Request Error!\n\n{:error, {:option, :server_only, :honor_cipher_order}}\n">>},[{'Elixir.ExAws','request!',2,[{file,"lib/ex_aws.ex"},{line,66}]},{'Elixir.Enum','-map/2-lists^map/1-0-',2,[{file,"lib/enum.ex"},{line,1327}]},{'Elixir.RabbitMQCloudWatchExporter.Exporter',handle_info,2,[{file,"lib/rabbitmq_cloudwatch_exporter/exporter.ex"},{line,60}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,637}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,711}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}
    ancestors: ['singleton_0bpF3SXqPZTQc80tAf+lu7E5kqQ','Elixir.Singleton.Supervisor',<0.271.0>]
    message_queue_len: 0
    messages: []
    links: [<0.696.0>]
    dictionary: [{rand_seed,{#{bits => 58,jump => #Fun<rand.13.8986388>,next => #Fun<rand.10.8986388>,type => exsss,uniform => #Fun<rand.11.8986388>,uniform_n => #Fun<rand.12.8986388>},[98387376866142632|287016919119728529]}}]
    trap_exit: false
    status: running
    heap_size: 10958
    stack_size: 27
    reductions: 577379
  neighbours:
    neighbour: [{pid,<0.696.0>},{registered_name,'singleton_0bpF3SXqPZTQc80tAf+lu7E5kqQ'},{initial_call,{'Elixir.Singleton.Manager',init,['Argument__1']}},{current_function,{gen_server,loop,7}},{ancestors,['Elixir.Singleton.Supervisor',<0.271.0>]},{message_queue_len,0},{links,[<0.272.0>,<0.697.0>]},{trap_exit,false},{status,waiting},{heap_size,233},{stack_size,11},{reductions,94},{current_stacktrace,[{gen_server,loop,7,[{file,"gen_server.erl"},{line,394}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}]
2019-09-30 08:22:11 =SUPERVISOR REPORT====
     Supervisor: {local,'Elixir.Singleton.Supervisor'}
     Context:    child_terminated
     Reason:     {#{'__exception__' => true,'__struct__' => 'Elixir.ExAws.Error',message => <<"ExAws Request Error!\n\n{:error, {:option, :server_only, :honor_cipher_order}}\n">>},[{'Elixir.ExAws','request!',2,[{file,"lib/ex_aws.ex"},{line,66}]},{'Elixir.Enum','-map/2-lists^map/1-0-',2,[{file,"lib/enum.ex"},{line,1327}]},{'Elixir.RabbitMQCloudWatchExporter.Exporter',handle_info,2,[{file,"lib/rabbitmq_cloudwatch_exporter/exporter.ex"},{line,60}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,637}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,711}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}
     Offender:   [{pid,<0.696.0>},{id,'Elixir.Singleton.Manager'},{mfargs,{'Elixir.Singleton.Manager',start_link,['Elixir.RabbitMQCloudWatchExporter.Exporter',[],rabbitmq_cloudwatch_exporter,'singleton_0bpF3SXqPZTQc80tAf+lu7E5kqQ']}},{restart_type,transient},{shutdown,5000},{child_type,worker}]
[  ] rabbitmq_auth_backend_cache       3.7.18
[  ] rabbitmq_auth_backend_http        3.7.18
[  ] rabbitmq_auth_backend_ldap        3.7.18
[  ] rabbitmq_auth_mechanism_ssl       3.7.18
[E ] rabbitmq_cloudwatch_exporter      0.2.1
[  ] rabbitmq_consistent_hash_exchange 3.7.18
[  ] rabbitmq_event_exchange           3.7.18
[  ] rabbitmq_federation               3.7.18
[  ] rabbitmq_federation_management    3.7.18
[  ] rabbitmq_jms_topic_exchange       3.7.18
[E*] rabbitmq_management               3.7.18
[e*] rabbitmq_management_agent         3.7.18
[  ] rabbitmq_mqtt                     3.7.18
[  ] rabbitmq_peer_discovery_aws       3.7.18
[  ] rabbitmq_peer_discovery_common    3.7.18
[  ] rabbitmq_peer_discovery_consul    3.7.18
[  ] rabbitmq_peer_discovery_etcd      3.7.18
[  ] rabbitmq_peer_discovery_k8s       3.7.18
[  ] rabbitmq_random_exchange          3.7.18
[  ] rabbitmq_recent_history_exchange  3.7.18
[  ] rabbitmq_sharding                 3.7.18
[  ] rabbitmq_shovel                   3.7.18
[  ] rabbitmq_shovel_management        3.7.18
[  ] rabbitmq_stomp                    3.7.18
[  ] rabbitmq_top                      3.7.18
[  ] rabbitmq_tracing                  3.7.18
[  ] rabbitmq_trust_store              3.7.18
[e*] rabbitmq_web_dispatch             3.7.18
[  ] rabbitmq_web_mqtt                 3.7.18
[  ] rabbitmq_web_mqtt_examples        3.7.18
[  ] rabbitmq_web_stomp                3.7.18
[  ] rabbitmq_web_stomp_examples       3.7.18```

```{running_applications,
     [{rabbitmq_management,"RabbitMQ Management Console","3.7.18"},
      {amqp_client,"RabbitMQ AMQP Client","3.7.18"},
      {rabbitmq_management_agent,"RabbitMQ Management Agent","3.7.18"},
      {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.7.18"},
      {rabbit,"RabbitMQ","3.7.18"},
      {rabbit_common,
          "Modules shared by rabbitmq-server and rabbitmq-erlang-client",
          "3.7.18"},
      {credentials_obfuscation,
          "Helper library that obfuscates sensitive values in process state",
          "1.1.0"},
      {hackney,"simple HTTP client","1.15.1"},
      {cowboy,"Small, fast, modern HTTP server.","2.6.1"},
      {ranch,"Socket acceptor pool for TCP protocols.","1.7.1"},
      {ssl,"Erlang/OTP SSL application","9.4"},
      {public_key,"Public key infrastructure","1.7"},
      {asn1,"The Erlang ASN1 compiler version 5.0.9","5.0.9"},
      {ssl_verify_fun,"SSL verification functions for Erlang\n","1.1.4"},
      {singleton,"Global, supervised singleton processes for Elixir","1.2.0"},
      {ex_aws_cloudwatch,
          "Cloudwatch module for https://github.com/ex-aws/ex_aws","2.0.4"},
      {ex_aws,"Generic AWS client","2.1.0"},
      {logger,"logger","1.8.2"},
      {poison,"An incredibly fast, pure Elixir JSON library","3.1.0"},
      {elixir,"elixir","1.8.2"},
      {inets,"INETS  CXC 138 49","7.1"},
      {mnesia,"MNESIA  CXC 138 12","4.16.1"},
      {certifi,"CA bundle adapted from Mozilla by https://certifi.io","2.5.1"},
      {stdout_formatter,
          "Tools to format paragraphs, lists and tables as plain text",
          "0.2.2"},
      {observer_cli,"Visualize Erlang Nodes On The Command Line","1.5.2"},
      {os_mon,"CPO  CXC 138 46","2.5.1"},
      {recon,"Diagnostic tools for production use","2.5.0"},
      {tools,"DEVTOOLS  CXC 138 16","3.2.1"},
      {sysmon_handler,"Rate-limiting system_monitor event handler","1.1.0"},
      {metrics,"A generic interface to different metrics systems in Erlang.",
          "1.0.1"},
      {idna,"A pure Erlang IDNA implementation","6.0.0"},
      {unicode_util_compat,
          "unicode_util compatibility library for Erlang < 20","0.4.1"},
      {jsx,"a streaming, evented json parsing toolkit","2.9.0"},
      {cowlib,"Support library for manipulating Web protocols.","2.7.0"},
      {mimerl,"Library to handle mimetypes","1.2.0"},
      {xmerl,"XML parser","1.3.22"},
      {crypto,"CRYPTO","4.6"},
      {lager,"Erlang logging framework","3.8.0"},
      {goldrush,"Erlang event stream processor","0.1.9"},
      {compiler,"ERTS  CXC 138 10","7.4.5"},
      {syntax_tools,"Syntax tools","2.2.1"},
      {sasl,"SASL  CXC 138 11","3.4.1"},
      {stdlib,"ERTS  CXC 138 10","3.10"},
      {kernel,"ERTS  CXC 138 10","6.5"}]},
 {os,{unix,linux}},
 {erlang_version,
     "Erlang/OTP 22 [erts-10.5] [source] [64-bit] [smp:1:1] [ds:1:1:10] [async-threads:64] [hipe]\n"},
westymalik commented 5 years ago

the instance role has the required permissions


curl http://169.254.169.254/latest/meta-data/iam/security-credentials/xxxxxx-xxxxx
{
  "Code" : "Success",
  "LastUpdated" : "2019-09-30T09:05:23Z",
  "Type" : "AWS-HMAC",
  "AccessKeyId" : "XXXXXXXXXXXXX",
  "SecretAccessKey" : "xxxxxxxxxxxxxxxxxxxxxxxxx",
  "Token" : "AgoJb3JpZ2luX2VjEOH//////////wEaCWV1LXdlc3QtMSJIMEYCIQDencc/voQ4eauNL2LrY55/8q5MuGFMnL5jcyWeFP60mgIhAJ1BlTkE+LWtFnPXj7kiy6lUS8xFQXbQcCKySov+b3GJKuMDCLr//////////wEQABoMNzU1Nzc3MTg5OTcwIgyZnq316DBqXloRWfcqtwOdoN221g1DfVGVIBWm2Ggu4aBKCnsVG/6kz5Yg5H9MxjzqnuEKYWapNPSAON3pJJutApzsqK8g0ZdnAI+NE9zWa9kkM46cf1TSsWqSLwcKnjpLET9xUWELUDwWqKb8rLThpCGRxVwzQfFNNuTIfTNZLXoChuvXJyp88OQLzksUVfLJERx9G/ka2Rd59d54Qjfi8fqqB5sTmv3V1lTL5kbCbFdRydFOCSkY+7xU//AayFfDlDKpPj270BpM8BjyvvNDP/4xs72syyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy+I/yPCMYtS36iG8/vzSFMZIA+hWKbSDHBx2YEBfud8yZL7kpWHAyr8WwNhky5lSJLQpUN/Q422j2eDqZ5Tbqhyask=",
  "Expiration" : "2019-09-30T15:06:50Z"```
noxdafox commented 5 years ago

Hello, a quick search gave me this result: https://github.com/benoitc/hackney/issues/591

Solutions are either downgrading Erlang to a version prior to 22.1 or re-building the plugin with the latest version of hackney. I just build a new version, could you please verify it works?

plugin.zip

westymalik commented 5 years ago

thanks @noxdafox changing to version 21.3.8.7 made it work perfectly 😃

noxdafox commented 5 years ago

Fix pushed into master.