search

noxdafox / rabbitmq-cloudwatch-exporter

RabbitMQ Plugin for publishing cluster metrics to AWS CloudWatch
Mozilla Public License 2.0
41 stars 9 forks source link

Exporting metrics doesn't work anymore after reinstall #36

Closed falcocoris closed 3 years ago

falcocoris commented 3 years ago

Hi

I've reinstalled my rabbitmq server, hence reinstalled the plugin, but the metrics aren't exporting anymore to cloudwatch.

Erlang version : 24.0.4 plugin files : up-to-date from new files on 1.0.0 release as I post

Getting the following error : 

2021-07-29 09:05:11 =ERROR REPORT====
** Generic server 'Elixir.ExAws.Config.AuthCache' terminating 
** Last message in was {refresh_auth,#{access_key_id => [{system,<<"AWS_ACCESS_KEY_ID">>},instance_role],host => <<"monitoring.eu-west-1.amazonaws.com">>,http_client => 'Elixir.ExAws.Request.Hackney',json_codec => 'Elixir.Jason',normalize_path => true,port => 443,region => <<"eu-west-1">>,retries => [{max_attempts,10},{base_backoff_in_ms,10},{max_backoff_in_ms,10000}],scheme => <<"https://">>,secret_access_key => [{system,<<"AWS_SECRET_ACCESS_KEY">>},instance_role]}}
** When Server state == 'Elixir.ExAws.Config.AuthCache'
** Reason for termination ==
** {'module could not be loaded',[{'Elixir.Jason','decode!',[<<"{\n  \"Code\" : \"Success\",\n  \"LastUpdated\" : \"2021-07-29T08:24:39Z\",\n  \"Type\" : \"AWS-HMAC\",\n  \"AccessKeyId\" : \"(...)\",\n  \"SecretAccessKey\" : \"(...)/02\",\n  \"Token\" : \"(...)==\",\n  \"Expiration\" : \"2021-07-29T14:33:13Z\"\n}">>],[]},{'Elixir.ExAws.InstanceMeta',security_credentials,1,[{file,"lib/ex_aws/instance_meta.ex"},{line,72}]},{'Elixir.ExAws.Config.AuthCache',refresh_auth_now,2,[{file,"lib/ex_aws/config/auth_cache.ex"},{line,116}]},{'Elixir.ExAws.Config.AuthCache',handle_call,3,[{file,"lib/ex_aws/config/auth_cache.ex"},{line,45}]},{gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,721}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,750}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}
** Client <0.20426.3> stacktrace
** [{gen,do_call,4,[{file,"gen.erl"},{line,233}]},{'Elixir.GenServer',call,3,[{file,"lib/gen_server.ex"},{line,1020}]},{'Elixir.ExAws.Config',retrieve_runtime_value,2,[{file,"lib/ex_aws/config.ex"},{line,92}]},{'Elixir.Stream','-map/2-fun-0-',4,[{file,"lib/stream.ex"},{line,572}]},{'Elixir.Enumerable.List',reduce,3,[{file,"lib/enum.ex"},{line,3686}]},{'Elixir.Enumerable.Stream',do_each,4,[{file,"lib/stream.ex"},{line,1609}]},{'Elixir.Enum',find,3,[{file,"lib/enum.ex"},{line,959}]},{'Elixir.ExAws.Config','-retrieve_runtime_config/1-fun-0-',2,[{file,"lib/ex_aws/config.ex"},{line,79}]}]
2021-07-29 09:05:11 =CRASH REPORT====
  crasher:
    initial call: Elixir.ExAws.Config.AuthCache:init/1
    pid: <0.20427.3>
    registered_name: 'Elixir.ExAws.Config.AuthCache'
    exception error: {undef,[{'Elixir.Jason','decode!',[<<"{\n  \"Code\" : \"Success\",\n  \"LastUpdated\" : \"2021-07-29T08:24:39Z\",\n  \"Type\" : \"AWS-HMAC\",\n  \"AccessKeyId\" : \"(...)\",\n  \"SecretAccessKey\" : \"(...)/02\",\n  \"Token\" : \"I(...)Q==\",\n  \"Expiration\" : \"2021-07-29T14:33:13Z\"\n}">>],[]},{'Elixir.ExAws.InstanceMeta',security_credentials,1,[{file,"lib/ex_aws/instance_meta.ex"},{line,72}]},{'Elixir.ExAws.Config.AuthCache',refresh_auth_now,2,[{file,"lib/ex_aws/config/auth_cache.ex"},{line,116}]},{'Elixir.ExAws.Config.AuthCache',handle_call,3,[{file,"lib/ex_aws/config/auth_cache.ex"},{line,45}]},{gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,721}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,750}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}
    ancestors: ['Elixir.ExAws.Supervisor',<0.1025.0>]
    message_queue_len: 0
    messages: []
    links: [<0.1026.0>]
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 10958
    stack_size: 29
    reductions: 221235
  neighbours:
2021-07-29 09:05:11 =SUPERVISOR REPORT====
     Supervisor: {local,'Elixir.ExAws.Supervisor'}
     Context:    child_terminated
     Reason:     {undef,[{'Elixir.Jason','decode!',[<<"{\n  \"Code\" : \"Success\",\n  \"LastUpdated\" : \"2021-07-29T08:24:39Z\",\n  \"Type\" : \"AWS-HMAC\",\n  \"AccessKeyId\" : \"(...)\",\n  \"SecretAccessKey\" : \"(...)/02\",\n  \"Token\" : \"IQo(...)Q==\",\n  \"Expiration\" : \"2021-07-29T14:33:13Z\"\n}">>],[]},{'Elixir.ExAws.InstanceMeta',security_credentials,1,[{file,"lib/ex_aws/instance_meta.ex"},{line,72}]},{'Elixir.ExAws.Config.AuthCache',refresh_auth_now,2,[{file,"lib/ex_aws/config/auth_cache.ex"},{line,116}]},{'Elixir.ExAws.Config.AuthCache',handle_call,3,[{file,"lib/ex_aws/config/auth_cache.ex"},{line,45}]},{gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,721}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,750}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}
     Offender:   [{pid,<0.20427.3>},{id,'Elixir.ExAws.Config.AuthCache'},{mfargs,{'Elixir.ExAws.Config.AuthCache',start_link,[[]]}},{restart_type,permanent},{significant,false},{shutdown,5000},{child_type,worker}]

2021-07-29 09:05:11 =ERROR REPORT====
** Generic server rabbitmq_cloudwatch_exporter terminating 
** Last message in was export_metrics
** When Server state == [{aws,[{region,<<"eu-west-1">>}]},{period,60},{resolution,60},{namespace,<<"RabbitMQ">>},{collectors,[{queue,[{export_regex,#{'__struct__' => 'Elixir.Regex',opts => <<>>,re_pattern => {re_pattern,0,0,0,<<69,82,67,80,87,0,0,0,0,0,0,0,81,0,0,0,255,255,255,255,255,255,255,255,97,0,95,0,0,0,0,0,0,0,64,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,131,0,19,29,97,29,115,29,103,29,95,29,109,29,115,29,95,94,12,120,0,19,0>>},re_version => {<<"8.44 2020-02-12">>,little},source => <<"asg_ms_.*">>}},{export_metrics,[<<"Messages">>]}]}]}]
** Reason for termination ==
** {{{undef,[{'Elixir.Jason','decode!',[<<"{\n  \"Code\" : \"Success\",\n  \"LastUpdated\" : \"2021-07-29T08:24:39Z\",\n  \"Type\" : \"AWS-HMAC\",\n  \"AccessKeyId\" : \"(...)\",\n  \"SecretAccessKey\" : \"(...)/02\",\n  \"Token\" : \"IQoJb3JpZ2luX2VjEIH//////////(...)FLJ/5FQ==\",\n  \"Expiration\" : \"2021-07-29T14:33:13Z\"\n}">>],[]},{'Elixir.ExAws.InstanceMeta',security_credentials,1,[{file,"lib/ex_aws/instance_meta.ex"},{line,72}]},{'Elixir.ExAws.Config.AuthCache',refresh_auth_now,2,[{file,"lib/ex_aws/config/auth_cache.ex"},{line,116}]},{'Elixir.ExAws.Config.AuthCache',handle_call,3,[{file,"lib/ex_aws/config/auth_cache.ex"},{line,45}]},{gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,721}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,750}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]},{'Elixir.GenServer',call,['Elixir.ExAws.Config.AuthCache',{refresh_auth,#{access_key_id => [{system,<<"AWS_ACCESS_KEY_ID">>},instance_role],host => <<"monitoring.eu-west-1.amazonaws.com">>,http_client => 'Elixir.ExAws.Request.Hackney',json_codec => 'Elixir.Jason',normalize_path => true,port => 443,region => <<"eu-west-1">>,retries => [{max_attempts,10},{base_backoff_in_ms,10},{max_backoff_in_ms,10000}],scheme => <<"https://">>,secret_access_key => [{system,<<"AWS_SECRET_ACCESS_KEY">>},instance_role]}},30000]}},[{'Elixir.GenServer',call,3,[{file,"lib/gen_server.ex"},{line,1023}]},{'Elixir.ExAws.Config',retrieve_runtime_value,2,[{file,"lib/ex_aws/config.ex"},{line,92}]},{'Elixir.Stream','-map/2-fun-0-',4,[{file,"lib/stream.ex"},{line,572}]},{'Elixir.Enumerable.List',reduce,3,[{file,"lib/enum.ex"},{line,3686}]},{'Elixir.Enumerable.Stream',do_each,4,[{file,"lib/stream.ex"},{line,1609}]},{'Elixir.Enum',find,3,[{file,"lib/enum.ex"},{line,959}]},{'Elixir.ExAws.Config','-retrieve_runtime_config/1-fun-0-',2,[{file,"lib/ex_aws/config.ex"},{line,79}]},{maps,fold_1,3,[{file,"maps.erl"},{line,410}]}]}
2021-07-29 09:05:11 =CRASH REPORT====
  crasher:
    initial call: Elixir.RabbitMQCloudWatchExporter.Exporter:init/1
    pid: <0.20426.3>
    registered_name: []
    exception exit: {{{undef,[{'Elixir.Jason','decode!',[<<"{\n  \"Code\" : \"Success\",\n  \"LastUpdated\" : \"2021-07-29T08:24:39Z\",\n  \"Type\" : \"AWS-HMAC\",\n  \"AccessKeyId\" : \"(...)\",\n  \"SecretAccessKey\" : \"(...)\",\n  \"Token\" : \(...)\",\n  \"Expiration\" : \"2021-07-29T14:33:13Z\"\n}">>],[]},{'Elixir.ExAws.InstanceMeta',security_credentials,1,[{file,"lib/ex_aws/instance_meta.ex"},{line,72}]},{'Elixir.ExAws.Config.AuthCache',refresh_auth_now,2,[{file,"lib/ex_aws/config/auth_cache.ex"},{line,116}]},{'Elixir.ExAws.Config.AuthCache',handle_call,3,[{file,"lib/ex_aws/config/auth_cache.ex"},{line,45}]},{gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,721}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,750}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]},{'Elixir.GenServer',call,['Elixir.ExAws.Config.AuthCache',{refresh_auth,#{access_key_id => [{system,<<"AWS_ACCESS_KEY_ID">>},instance_role],host => <<"monitoring.eu-west-1.amazonaws.com">>,http_client => 'Elixir.ExAws.Request.Hackney',json_codec => 'Elixir.Jason',normalize_path => true,port => 443,region => <<"eu-west-1">>,retries => [{max_attempts,10},{base_backoff_in_ms,10},{max_backoff_in_ms,10000}],scheme => <<"https://">>,secret_access_key => [{system,<<"AWS_SECRET_ACCESS_KEY">>},instance_role]}},30000]}},[{'Elixir.GenServer',call,3,[{file,"lib/gen_server.ex"},{line,1023}]},{'Elixir.ExAws.Config',retrieve_runtime_value,2,[{file,"lib/ex_aws/config.ex"},{line,92}]},{'Elixir.Stream','-map/2-fun-0-',4,[{file,"lib/stream.ex"},{line,572}]},{'Elixir.Enumerable.List',reduce,3,[{file,"lib/enum.ex"},{line,3686}]},{'Elixir.Enumerable.Stream',do_each,4,[{file,"lib/stream.ex"},{line,1609}]},{'Elixir.Enum',find,3,[{file,"lib/enum.ex"},{line,959}]},{'Elixir.ExAws.Config','-retrieve_runtime_config/1-fun-0-',2,[{file,"lib/ex_aws/config.ex"},{line,79}]},{maps,fold_1,3,[{file,"maps.erl"},{line,410}]}]}
    ancestors: ['singleton_0bpF3SXqPZTQc80tAf+lu7E5kqQ','Elixir.Singleton.Supervisor',<0.1156.0>]
    message_queue_len: 0
    messages: []
    links: [<0.20425.3>]
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 10958
    stack_size: 29
    reductions: 230419
  neighbours:
    neighbour: [{pid,<0.20425.3>},{registered_name,'singleton_0bpF3SXqPZTQc80tAf+lu7E5kqQ'},{initial_call,{'Elixir.Singleton.Manager',init,['Argument__1']}},{current_function,{gen_server,loop,7}},{ancestors,['Elixir.Singleton.Supervisor',<0.1156.0>]},{message_queue_len,0},{links,[<0.1157.0>,<0.20426.3>]},{trap_exit,false},{status,waiting},{heap_size,233},{stack_size,12},{reductions,85},{current_stacktrace,[{gen_server,loop,7,[{file,"gen_server.erl"},{line,443}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}]
2021-07-29 09:05:11 =SUPERVISOR REPORT====
     Supervisor: {local,'Elixir.Singleton.Supervisor'}
     Context:    child_terminated
     Reason:     {{undef,[{'Elixir.Jason','decode!',[<<"{\n  \"Code\" : \"Success\",\n  \"LastUpdated\" : \"2021-07-29T08:24:39Z\",\n  \"Type\" : \"AWS-HMAC\",\n  \"AccessKeyId\" : \"(...)\",\n  \"SecretAccessKey\" : \(...)",\n  \"Token\" : \(...),\n  \"Expiration\" : \"2021-07-29T14:33:13Z\"\n}">>],[]},{'Elixir.ExAws.InstanceMeta',security_credentials,1,[{file,"lib/ex_aws/instance_meta.ex"},{line,72}]},{'Elixir.ExAws.Config.AuthCache',refresh_auth_now,2,[{file,"lib/ex_aws/config/auth_cache.ex"},{line,116}]},{'Elixir.ExAws.Config.AuthCache',handle_call,3,[{file,"lib/ex_aws/config/auth_cache.ex"},{line,45}]},{gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,721}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,750}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]},{'Elixir.GenServer',call,['Elixir.ExAws.Config.AuthCache',{refresh_auth,#{access_key_id => [{system,<<"AWS_ACCESS_KEY_ID">>},instance_role],host => <<"monitoring.eu-west-1.amazonaws.com">>,http_client => 'Elixir.ExAws.Request.Hackney',json_codec => 'Elixir.Jason',normalize_path => true,port => 443,region => <<"eu-west-1">>,retries => [{max_attempts,10},{base_backoff_in_ms,10},{max_backoff_in_ms,10000}],scheme => <<"https://">>,secret_access_key => [{system,<<"AWS_SECRET_ACCESS_KEY">>},instance_role]}},30000]}}
     Offender:   [{pid,<0.20425.3>},{id,'Elixir.Singleton.Manager'},{mfargs,{'Elixir.Singleton.Manager',start_link,['Elixir.RabbitMQCloudWatchExporter.Exporter',[],rabbitmq_cloudwatch_exporter,'singleton_0bpF3SXqPZTQc80tAf+lu7E5kqQ']}},{restart_type,transient},{significant,false},{shutdown,5000},{child_type,worker}]

The plugin seems to be well installed : 

root@ip-10-1-102-210:/home/ubuntu# rabbitmq-plugins is_enabled rabbitmq_cloudwatch_exporter
Asking node rabbit@ip-10-1-102-210 if plugin rabbitmq_cloudwatch_exporter is enabled...
Plugin rabbitmq_cloudwatch_exporter is enabled on node rabbit@ip-10-1-102-210

The conf is simple and used to work with previous versions : 

cloudwatch_exporter.aws.region = eu-west-1
cloudwatch_exporter.metrics.queue.enable = true
cloudwatch_exporter.metrics.queue.export_regex = "asg_ms_.*"
cloudwatch_exporter.metrics.queue.export_metrics.1 = "Messages"

Thx again for your work ! :)

noxdafox commented 3 years ago

Apologies for the disruption, elixir and logger dependencies were updated too.

Please let me know if the issue persists.

noxdafox commented 3 years ago

The package has been built using Erlang 24, the error:

This BEAM file was compiled for a later version of the run-time system than 23.

Suggests you are using erlang <= 23. Are you sure your system is running Erlang 24?

noxdafox commented 3 years ago

I noticed it might take some time between AWS receives the metrics and it renders it. I would recommend you to wait half an hour before checking if new metrics are in.

noxdafox commented 3 years ago

Also please note that Debian Erlang packages are a bit challenging to handle as they split all dependencies into separate packages. Moreover, they tend to be always a bit out-of-date.

RabbitMQ recommends different sources for Erlang packages. I usually use those from Erlang Solutions. They package the whole eco-system into a single .deb which makes it easier to get your environment correctly configured.

falcocoris commented 3 years ago

I've deleted my today's post as I was doing actions on the wrong instance, inviting you to delete yours too to clear out this issue for search purpose, and sorry about that :) Back to the pb, I did update both dependencies logger and elixir but still having this log : 

2021-08-02 13:09:47 =ERROR REPORT====
** Generic server rabbitmq_cloudwatch_exporter terminating 
** Last message in was export_metrics
** When Server state == [{aws,[{region,<<"eu-west-1">>}]},{period,60},{resolution,60},{namespace,<<"RabbitMQ">>},{collectors,[{queue,[{export_regex,#{'__struct__' => 'Elixir.Regex',opts => <<>>,re_pattern => {re_pattern,0,0,0,<<69,82,67,80,50,1,0,0,0,0,0,0,1,0,0,0,255,255,255,255,255,255,255,255,0,0,0,0,0,0,0,0,0,0,64,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,131,0,19,29,97,29,115,29,103,29,95,29,109,29,115,29,95,94,12,119,0,49,29,100,29,108,29,113,29,95,29,109,29,115,29,95,29,109,29,101,29,100,29,105,29,97,29,95,29,103,29,101,29,110,29,101,29,114,29,97,29,116,29,111,29,114,94,12,119,0,47,29,100,29,108,29,113,29,95,29,109,29,115,29,95,29,109,29,101,29,100,29,105,29,97,29,95,29,115,29,101,29,108,29,101,29,99,29,116,29,111,29,114,94,12,119,0,43,29,100,29,108,29,113,29,95,29,115,29,116,29,111,29,114,29,101,29,95,29,112,29,104,29,111,29,116,29,111,29,95,29,105,29,110,29,102,29,111,119,0,21,29,100,29,108,29,113,29,95,29,114,29,97,29,105,29,108,29,115,119,0,59,29,100,29,108,29,113,29,95,29,109,29,115,29,95,29,114,29,97,29,105,29,108,29,115,29,59,29,122,29,105,29,112,29,95,29,112,29,104,29,111,29,116,29,111,29,115,29,117,29,98,29,115,29,101,29,116,120,0,238,0>>},re_version => {<<"8.44 2020-02-12">>,little},source => <<"asg_ms_.*|dlq_ms_media_generator.*|dlq_ms_media_selector.*|dlq_store_photo_info|dlq_rails|dlq_ms_rails;zip_photosubset">>}},{export_metrics,[<<"Messages">>]}]}]}]
** Reason for termination ==
** {{{undef,[{'Elixir.Jason','decode!',[<<"{\n  \"Code\" : \"Success\",\n  \"LastUpdated\" : \"2021-08-02T12:13:18Z\",\n  \"Type\" : \"AWS-HMAC\",\n  \"AccessKeyId\" : \"(...)\",\n  \"SecretAccessKey\" : \"(...)\",\n  \"Token\" : \"(...)\",\n  \"Expiration\" : \"2021-08-02T18:26:00Z\"\n}">>],[]},{'Elixir.ExAws.InstanceMeta',security_credentials,1,[{file,"lib/ex_aws/instance_meta.ex"},{line,72}]},{'Elixir.ExAws.Config.AuthCache',refresh_auth_now,2,[{file,"lib/ex_aws/config/auth_cache.ex"},{line,116}]},{'Elixir.ExAws.Config.AuthCache',handle_call,3,[{file,"lib/ex_aws/config/auth_cache.ex"},{line,45}]},{gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,721}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,750}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]},{'Elixir.GenServer',call,['Elixir.ExAws.Config.AuthCache',{refresh_auth,#{access_key_id => [{system,<<"AWS_ACCESS_KEY_ID">>},instance_role],host => <<"monitoring.eu-west-1.amazonaws.com">>,http_client => 'Elixir.ExAws.Request.Hackney',json_codec => 'Elixir.Jason',normalize_path => true,port => 443,region => <<"eu-west-1">>,retries => [{max_attempts,10},{base_backoff_in_ms,10},{max_backoff_in_ms,10000}],scheme => <<"https://">>,secret_access_key => [{system,<<"AWS_SECRET_ACCESS_KEY">>},instance_role]}},30000]}},[{'Elixir.GenServer',call,3,[{file,"lib/gen_server.ex"},{line,1024}]},{'Elixir.ExAws.Config',retrieve_runtime_value,2,[{file,"lib/ex_aws/config.ex"},{line,92}]},{'Elixir.Stream','-map/2-fun-0-',4,[{file,"lib/stream.ex"},{line,572}]},{'Elixir.Enumerable.List',reduce,3,[{file,"lib/enum.ex"},{line,4251}]},{'Elixir.Enumerable.Stream',do_each,4,[{file,"lib/stream.ex"},{line,1719}]},{'Elixir.Enum',find,3,[{file,"lib/enum.ex"},{line,1105}]},{'Elixir.ExAws.Config','-retrieve_runtime_config/1-fun-0-',2,[{file,"lib/ex_aws/config.ex"},{line,79}]},{maps,fold_1,3,[{file,"maps.erl"},{line,410}]}]}
2021-08-02 13:09:47 =CRASH REPORT====
  crasher:
    initial call: Elixir.RabbitMQCloudWatchExporter.Exporter:init/1
    pid: <0.7554.1>
    registered_name: []
    exception exit: {{{undef,[{'Elixir.Jason','decode!',[<<"{\n  \"Code\" : \"Success\",\n  \"LastUpdated\" : \"2021-08-02T12:13:18Z\",\n  \"Type\" : \"AWS-HMAC\",\n  \"AccessKeyId\" : \"(...)\",\n  \"SecretAccessKey\" : \"(...)\",\n  \"Token\" : \"I(...)g==\",\n  \"Expiration\" : \"2021-08-02T18:26:00Z\"\n}">>],[]},{'Elixir.ExAws.InstanceMeta',security_credentials,1,[{file,"lib/ex_aws/instance_meta.ex"},{line,72}]},{'Elixir.ExAws.Config.AuthCache',refresh_auth_now,2,[{file,"lib/ex_aws/config/auth_cache.ex"},{line,116}]},{'Elixir.ExAws.Config.AuthCache',handle_call,3,[{file,"lib/ex_aws/config/auth_cache.ex"},{line,45}]},{gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,721}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,750}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]},{'Elixir.GenServer',call,['Elixir.ExAws.Config.AuthCache',{refresh_auth,#{access_key_id => [{system,<<"AWS_ACCESS_KEY_ID">>},instance_role],host => <<"monitoring.eu-west-1.amazonaws.com">>,http_client => 'Elixir.ExAws.Request.Hackney',json_codec => 'Elixir.Jason',normalize_path => true,port => 443,region => <<"eu-west-1">>,retries => [{max_attempts,10},{base_backoff_in_ms,10},{max_backoff_in_ms,10000}],scheme => <<"https://">>,secret_access_key => [{system,<<"AWS_SECRET_ACCESS_KEY">>},instance_role]}},30000]}},[{'Elixir.GenServer',call,3,[{file,"lib/gen_server.ex"},{line,1024}]},{'Elixir.ExAws.Config',retrieve_runtime_value,2,[{file,"lib/ex_aws/config.ex"},{line,92}]},{'Elixir.Stream','-map/2-fun-0-',4,[{file,"lib/stream.ex"},{line,572}]},{'Elixir.Enumerable.List',reduce,3,[{file,"lib/enum.ex"},{line,4251}]},{'Elixir.Enumerable.Stream',do_each,4,[{file,"lib/stream.ex"},{line,1719}]},{'Elixir.Enum',find,3,[{file,"lib/enum.ex"},{line,1105}]},{'Elixir.ExAws.Config','-retrieve_runtime_config/1-fun-0-',2,[{file,"lib/ex_aws/config.ex"},{line,79}]},{maps,fold_1,3,[{file,"maps.erl"},{line,410}]}]}
    ancestors: ['singleton_0bpF3SXqPZTQc80tAf+lu7E5kqQ','Elixir.Singleton.Supervisor',<0.1163.0>]
    message_queue_len: 0
    messages: []
    links: [<0.7553.1>]
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 4185
    stack_size: 29
    reductions: 216521
  neighbours:
    neighbour: [{pid,<0.7553.1>},{registered_name,'singleton_0bpF3SXqPZTQc80tAf+lu7E5kqQ'},{initial_call,{'Elixir.Singleton.Manager',init,['Argument__1']}},{current_function,{gen_server,loop,7}},{ancestors,['Elixir.Singleton.Supervisor',<0.1163.0>]},{message_queue_len,0},{links,[<0.1164.0>,<0.7554.1>]},{trap_exit,false},{status,waiting},{heap_size,233},{stack_size,12},{reductions,85},{current_stacktrace,[{gen_server,loop,7,[{file,"gen_server.erl"},{line,443}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}]
2021-08-02 13:09:47 =SUPERVISOR REPORT====
     Supervisor: {local,'Elixir.Singleton.Supervisor'}
     Context:    child_terminated
     Reason:     {{undef,[{'Elixir.Jason','decode!',[<<"{\n  \"Code\" : \"Success\",\n  \"LastUpdated\" : \"2021-08-02T12:13:18Z\",\n  \"Type\" : \"AWS-HMAC\",\n  \"AccessKeyId\" : \"(...)\",\n  \"SecretAccessKey\" : \"(...)\",\n  \"Token\" : \"I(...)==\",\n  \"Expiration\" : \"2021-08-02T18:26:00Z\"\n}">>],[]},{'Elixir.ExAws.InstanceMeta',security_credentials,1,[{file,"lib/ex_aws/instance_meta.ex"},{line,72}]},{'Elixir.ExAws.Config.AuthCache',refresh_auth_now,2,[{file,"lib/ex_aws/config/auth_cache.ex"},{line,116}]},{'Elixir.ExAws.Config.AuthCache',handle_call,3,[{file,"lib/ex_aws/config/auth_cache.ex"},{line,45}]},{gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,721}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,750}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]},{'Elixir.GenServer',call,['Elixir.ExAws.Config.AuthCache',{refresh_auth,#{access_key_id => [{system,<<"AWS_ACCESS_KEY_ID">>},instance_role],host => <<"monitoring.eu-west-1.amazonaws.com">>,http_client => 'Elixir.ExAws.Request.Hackney',json_codec => 'Elixir.Jason',normalize_path => true,port => 443,region => <<"eu-west-1">>,retries => [{max_attempts,10},{base_backoff_in_ms,10},{max_backoff_in_ms,10000}],scheme => <<"https://">>,secret_access_key => [{system,<<"AWS_SECRET_ACCESS_KEY">>},instance_role]}},30000]}}
     Offender:   [{pid,<0.7553.1>},{id,'Elixir.Singleton.Manager'},{mfargs,{'Elixir.Singleton.Manager',start_link,['Elixir.RabbitMQCloudWatchExporter.Exporter',[],rabbitmq_cloudwatch_exporter,'singleton_0bpF3SXqPZTQc80tAf+lu7E5kqQ']}},{restart_type,transient},{significant,false},{shutdown,5000},{child_type,worker}]
noxdafox commented 3 years ago

Please make sure to download all packages anew as some might have been re-compiled and replaced.

Here is how I tested the plugin with a Dockerfile.

FROM rabbitmq:3.8-management

ARG PLUGIN_DIR=plugins/
ARG PLUGIN_URL=https://api.github.com/repos/noxdafox/rabbitmq-cloudwatch-exporter/releases

# Install essential packages
RUN apt update && apt install -y curl jq wget

# Dowload plugin and dependencies
RUN for url in $(curl $PLUGIN_URL | jq .[0].assets[].browser_download_url -r); \
    do wget $url -P $PLUGIN_DIR; done

# Enable plugin
RUN rabbitmq-plugins enable --offline rabbitmq_cloudwatch_exporter

# Configure the broker to use the plugin
RUN echo 'loopback_users.guest = false                                                 \n\
cloudwatch_exporter.aws.region = "YOUR_AWS_REGION"                          \n\
cloudwatch_exporter.aws.access_key_id = "YOUR_KEY_ID"                         \n\
cloudwatch_exporter.aws.secret_access_key = "YOUR_SECRET_KEY"  \n\
cloudwatch_exporter.namespace = "RabbitMQPluginTest"                        \n\
cloudwatch_exporter.export_period = 20                                                 \n\
cloudwatch_exporter.metrics.overview.enable = true'                                    \
>> /etc/rabbitmq/rabbitmq.conf
$ docker build . -t rmq-docker
$ docker run -it -p 15672:15672 rmq-docker:latest

Browse to http://localhost:15672 login with default credential. I created few queues and published few messages. I could see the metrics in AWS CW. No error logs, just a warning:

Problem reading some plugins: [{"/opt/rabbitmq/plugins/lager-3.8.0.ez",
                                duplicate_plugin}]

I cannot remove lager as a package as in some old deployment it was giving me some issues.

Sonic0 commented 3 years ago

Hi guys, tomorrow I'm going to try this plugin with RabbitMQ 3.9.1 and Erlang 24 in an EC2 instance. Today I successfully installed this plugin retrieving packages of the latest version (1.0.0) via GitHub API, in a Docker container (RabbitMQ 3.9.1 and Erlang 24)... without trying metric creation in CloudWatch, though :sweat_smile:.

Maybe useful, GitHub API call:

curl -H "Accept: application/vnd.github.v3+json" https://api.github.com/repos/noxdafox/rabbitmq-cloudwatch-exporter/releases | jq ".[0].assets[].browser_download_url"
Sonic0 commented 3 years ago

I've deleted my today's post as I was doing actions on the wrong instance, inviting you to delete yours too to clear out this issue for search purpose, and sorry about that :) Back to the pb, I did update both dependencies logger and elixir but still having this log :

2021-08-02 13:09:47 =ERROR REPORT====
** Generic server rabbitmq_cloudwatch_exporter terminating 
** Last message in was export_metrics
** When Server state == [{aws,[{region,<<"eu-west-1">>}]},{period,60},{resolution,60},{namespace,<<"RabbitMQ">>},{collectors,[{queue,[{export_regex,#{'__struct__' => 'Elixir.Regex',opts => <<>>,re_pattern => {re_pattern,0,0,0,<<69,82,67,80,50,1,0,0,0,0,0,0,1,0,0,0,255,255,255,255,255,255,255,255,0,0,0,0,0,0,0,0,0,0,64,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,131,0,19,29,97,29,115,29,103,29,95,29,109,29,115,29,95,94,12,119,0,49,29,100,29,108,29,113,29,95,29,109,29,115,29,95,29,109,29,101,29,100,29,105,29,97,29,95,29,103,29,101,29,110,29,101,29,114,29,97,29,116,29,111,29,114,94,12,119,0,47,29,100,29,108,29,113,29,95,29,109,29,115,29,95,29,109,29,101,29,100,29,105,29,97,29,95,29,115,29,101,29,108,29,101,29,99,29,116,29,111,29,114,94,12,119,0,43,29,100,29,108,29,113,29,95,29,115,29,116,29,111,29,114,29,101,29,95,29,112,29,104,29,111,29,116,29,111,29,95,29,105,29,110,29,102,29,111,119,0,21,29,100,29,108,29,113,29,95,29,114,29,97,29,105,29,108,29,115,119,0,59,29,100,29,108,29,113,29,95,29,109,29,115,29,95,29,114,29,97,29,105,29,108,29,115,29,59,29,122,29,105,29,112,29,95,29,112,29,104,29,111,29,116,29,111,29,115,29,117,29,98,29,115,29,101,29,116,120,0,238,0>>},re_version => {<<"8.44 2020-02-12">>,little},source => <<"asg_ms_.*|dlq_ms_media_generator.*|dlq_ms_media_selector.*|dlq_store_photo_info|dlq_rails|dlq_ms_rails;zip_photosubset">>}},{export_metrics,[<<"Messages">>]}]}]}]
** Reason for termination ==
** {{{undef,[{'Elixir.Jason','decode!',[<<"{\n  \"Code\" : \"Success\",\n  \"LastUpdated\" : \"2021-08-02T12:13:18Z\",\n  \"Type\" : \"AWS-HMAC\",\n  \"AccessKeyId\" : \"(...)\",\n  \"SecretAccessKey\" : \"(...)\",\n  \"Token\" : \"(...)\",\n  \"Expiration\" : \"2021-08-02T18:26:00Z\"\n}">>],[]},{'Elixir.ExAws.InstanceMeta',security_credentials,1,[{file,"lib/ex_aws/instance_meta.ex"},{line,72}]},{'Elixir.ExAws.Config.AuthCache',refresh_auth_now,2,[{file,"lib/ex_aws/config/auth_cache.ex"},{line,116}]},{'Elixir.ExAws.Config.AuthCache',handle_call,3,[{file,"lib/ex_aws/config/auth_cache.ex"},{line,45}]},{gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,721}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,750}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]},{'Elixir.GenServer',call,['Elixir.ExAws.Config.AuthCache',{refresh_auth,#{access_key_id => [{system,<<"AWS_ACCESS_KEY_ID">>},instance_role],host => <<"monitoring.eu-west-1.amazonaws.com">>,http_client => 'Elixir.ExAws.Request.Hackney',json_codec => 'Elixir.Jason',normalize_path => true,port => 443,region => <<"eu-west-1">>,retries => [{max_attempts,10},{base_backoff_in_ms,10},{max_backoff_in_ms,10000}],scheme => <<"https://">>,secret_access_key => [{system,<<"AWS_SECRET_ACCESS_KEY">>},instance_role]}},30000]}},[{'Elixir.GenServer',call,3,[{file,"lib/gen_server.ex"},{line,1024}]},{'Elixir.ExAws.Config',retrieve_runtime_value,2,[{file,"lib/ex_aws/config.ex"},{line,92}]},{'Elixir.Stream','-map/2-fun-0-',4,[{file,"lib/stream.ex"},{line,572}]},{'Elixir.Enumerable.List',reduce,3,[{file,"lib/enum.ex"},{line,4251}]},{'Elixir.Enumerable.Stream',do_each,4,[{file,"lib/stream.ex"},{line,1719}]},{'Elixir.Enum',find,3,[{file,"lib/enum.ex"},{line,1105}]},{'Elixir.ExAws.Config','-retrieve_runtime_config/1-fun-0-',2,[{file,"lib/ex_aws/config.ex"},{line,79}]},{maps,fold_1,3,[{file,"maps.erl"},{line,410}]}]}
2021-08-02 13:09:47 =CRASH REPORT====
  crasher:
    initial call: Elixir.RabbitMQCloudWatchExporter.Exporter:init/1
    pid: <0.7554.1>
    registered_name: []
    exception exit: {{{undef,[{'Elixir.Jason','decode!',[<<"{\n  \"Code\" : \"Success\",\n  \"LastUpdated\" : \"2021-08-02T12:13:18Z\",\n  \"Type\" : \"AWS-HMAC\",\n  \"AccessKeyId\" : \"(...)\",\n  \"SecretAccessKey\" : \"(...)\",\n  \"Token\" : \"I(...)g==\",\n  \"Expiration\" : \"2021-08-02T18:26:00Z\"\n}">>],[]},{'Elixir.ExAws.InstanceMeta',security_credentials,1,[{file,"lib/ex_aws/instance_meta.ex"},{line,72}]},{'Elixir.ExAws.Config.AuthCache',refresh_auth_now,2,[{file,"lib/ex_aws/config/auth_cache.ex"},{line,116}]},{'Elixir.ExAws.Config.AuthCache',handle_call,3,[{file,"lib/ex_aws/config/auth_cache.ex"},{line,45}]},{gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,721}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,750}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]},{'Elixir.GenServer',call,['Elixir.ExAws.Config.AuthCache',{refresh_auth,#{access_key_id => [{system,<<"AWS_ACCESS_KEY_ID">>},instance_role],host => <<"monitoring.eu-west-1.amazonaws.com">>,http_client => 'Elixir.ExAws.Request.Hackney',json_codec => 'Elixir.Jason',normalize_path => true,port => 443,region => <<"eu-west-1">>,retries => [{max_attempts,10},{base_backoff_in_ms,10},{max_backoff_in_ms,10000}],scheme => <<"https://">>,secret_access_key => [{system,<<"AWS_SECRET_ACCESS_KEY">>},instance_role]}},30000]}},[{'Elixir.GenServer',call,3,[{file,"lib/gen_server.ex"},{line,1024}]},{'Elixir.ExAws.Config',retrieve_runtime_value,2,[{file,"lib/ex_aws/config.ex"},{line,92}]},{'Elixir.Stream','-map/2-fun-0-',4,[{file,"lib/stream.ex"},{line,572}]},{'Elixir.Enumerable.List',reduce,3,[{file,"lib/enum.ex"},{line,4251}]},{'Elixir.Enumerable.Stream',do_each,4,[{file,"lib/stream.ex"},{line,1719}]},{'Elixir.Enum',find,3,[{file,"lib/enum.ex"},{line,1105}]},{'Elixir.ExAws.Config','-retrieve_runtime_config/1-fun-0-',2,[{file,"lib/ex_aws/config.ex"},{line,79}]},{maps,fold_1,3,[{file,"maps.erl"},{line,410}]}]}
    ancestors: ['singleton_0bpF3SXqPZTQc80tAf+lu7E5kqQ','Elixir.Singleton.Supervisor',<0.1163.0>]
    message_queue_len: 0
    messages: []
    links: [<0.7553.1>]
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 4185
    stack_size: 29
    reductions: 216521
  neighbours:
    neighbour: [{pid,<0.7553.1>},{registered_name,'singleton_0bpF3SXqPZTQc80tAf+lu7E5kqQ'},{initial_call,{'Elixir.Singleton.Manager',init,['Argument__1']}},{current_function,{gen_server,loop,7}},{ancestors,['Elixir.Singleton.Supervisor',<0.1163.0>]},{message_queue_len,0},{links,[<0.1164.0>,<0.7554.1>]},{trap_exit,false},{status,waiting},{heap_size,233},{stack_size,12},{reductions,85},{current_stacktrace,[{gen_server,loop,7,[{file,"gen_server.erl"},{line,443}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}]
2021-08-02 13:09:47 =SUPERVISOR REPORT====
     Supervisor: {local,'Elixir.Singleton.Supervisor'}
     Context:    child_terminated
     Reason:     {{undef,[{'Elixir.Jason','decode!',[<<"{\n  \"Code\" : \"Success\",\n  \"LastUpdated\" : \"2021-08-02T12:13:18Z\",\n  \"Type\" : \"AWS-HMAC\",\n  \"AccessKeyId\" : \"(...)\",\n  \"SecretAccessKey\" : \"(...)\",\n  \"Token\" : \"I(...)==\",\n  \"Expiration\" : \"2021-08-02T18:26:00Z\"\n}">>],[]},{'Elixir.ExAws.InstanceMeta',security_credentials,1,[{file,"lib/ex_aws/instance_meta.ex"},{line,72}]},{'Elixir.ExAws.Config.AuthCache',refresh_auth_now,2,[{file,"lib/ex_aws/config/auth_cache.ex"},{line,116}]},{'Elixir.ExAws.Config.AuthCache',handle_call,3,[{file,"lib/ex_aws/config/auth_cache.ex"},{line,45}]},{gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,721}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,750}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]},{'Elixir.GenServer',call,['Elixir.ExAws.Config.AuthCache',{refresh_auth,#{access_key_id => [{system,<<"AWS_ACCESS_KEY_ID">>},instance_role],host => <<"monitoring.eu-west-1.amazonaws.com">>,http_client => 'Elixir.ExAws.Request.Hackney',json_codec => 'Elixir.Jason',normalize_path => true,port => 443,region => <<"eu-west-1">>,retries => [{max_attempts,10},{base_backoff_in_ms,10},{max_backoff_in_ms,10000}],scheme => <<"https://">>,secret_access_key => [{system,<<"AWS_SECRET_ACCESS_KEY">>},instance_role]}},30000]}}
     Offender:   [{pid,<0.7553.1>},{id,'Elixir.Singleton.Manager'},{mfargs,{'Elixir.Singleton.Manager',start_link,['Elixir.RabbitMQCloudWatchExporter.Exporter',[],rabbitmq_cloudwatch_exporter,'singleton_0bpF3SXqPZTQc80tAf+lu7E5kqQ']}},{restart_type,transient},{significant,false},{shutdown,5000},{child_type,worker}]

@noxdafox @falcocoris same issue for me and no metric on CloudWatch. EC2 instance with Ubuntu18.04, RabbitMQ 3.9.1 and Erlang 24.0.5. Asap I'm going to try installing the RabbitMQ 3.8.X.

Sonic0 commented 3 years ago

This is the error with RabbitMQ 3.8.19

** Client <0.1018.0> stacktrace
** [{gen,do_call,4,[{file,"gen.erl"},{line,233}]},{'Elixir.GenServer',call,3,[{file,"lib/gen_server.ex"},{line,1021}]},{'Elixir.ExAws.Config',retrieve_runtime_value,2,[{file,"lib/ex_aws/conf
ig.ex"},{line,92}]},{'Elixir.Stream','-map/2-fun-0-',4,[{file,"lib/stream.ex"},{line,572}]},{'Elixir.Enumerable.List',reduce,3,[{file,"lib/enum.ex"},{line,4251}]},{'Elixir.Enumerable.Stream'
,do_each,4,[{file,"lib/stream.ex"},{line,1719}]},{'Elixir.Enum',find,3,[{file,"lib/enum.ex"},{line,1105}]},{'Elixir.ExAws.Config','-retrieve_runtime_config/1-fun-0-',2,[{file,"lib/ex_aws/con
fig.ex"},{line,79}]}]
2021-08-03 18:19:07.659 [error] <0.1016.0> CRASH REPORT Process 'Elixir.ExAws.Config.AuthCache' with 0 neighbours crashed with reason: call to undefined function 'Elixir.Jason':'decode!'(<<"
{\n  \"Code\" : \"Success\",\n  \"LastUpdated\" : \"2021-08-03T16:11:24Z\",\n  \"Type\" : \"AWS-HMAC\",\n  \"Acc...">>)
2021-08-03 18:19:07.659 [error] <0.1018.0> ** Generic server rabbitmq_cloudwatch_exporter terminating 
** Last message in was export_metrics
** When Server state == [{aws,[{region,<<"eu-west-1">>}]},{period,60},{resolution,60},{namespace,<<"RabbitMQ">>},{collectors,[{exchange,[{export_metrics,[<<"PublishIn">>,<<"PublishOut">>]}]}
,{node,[]},{overview,[]},{queue,[{export_metrics,[<<"Messages">>,<<"MessagesReady">>]}]},{vhost,[]}]}]
** Reason for termination ==
** {{{undef,[{'Elixir.Jason','decode!',[<<"{\n  \"Code\" : \"Success\",\n  \"LastUpdated\" : \"2021-08-03T16:11:24Z\",\n  \"Type\" : \"AWS-HMAC\",\n  \"AccessKeyId\" : \"...
\",\n  \"SecretAccessKey\" : \"...\",\n  \"Token\" : \"...\",\
n  \"Expiration\" : \"2021-08-03T22:17:42Z\"\n}">>],[]},{'Elixir.ExAws.InstanceMeta',security_credentials,1,[{file,"lib/ex_aws/instance_meta.ex"},{line,72}]},{'Elixir.ExAws.Config.AuthCache'
,refresh_auth_now,2,[{file,"lib/ex_aws/config/auth_cache.ex"},{line,116}]},{'Elixir.ExAws.Config.AuthCache',handle_call,3,[{file,"lib/ex_aws/config/auth_cache.ex"},{line,45}]},{gen_server,tr
y_handle_call,4,[{file,"gen_server.erl"},{line,721}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,750}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]},{'Elixi
r.GenServer',call,['Elixir.ExAws.Config.AuthCache',{refresh_auth,#{access_key_id => [{system,<<"AWS_ACCESS_KEY_ID">>},instance_role],host => <<"monitoring.eu-west-1.amazonaws.com">>,http_cli
ent => 'Elixir.ExAws.Request.Hackney',json_codec => 'Elixir.Jason',normalize_path => true,port => 443,region => <<"eu-west-1">>,retries => [{max_attempts,10},{base_backoff_in_ms,10},{max_bac
koff_in_ms,10000}],scheme => <<"https://">>,secret_access_key => [{system,<<"AWS_SECRET_ACCESS_KEY">>},instance_role]}},30000]}},[{'Elixir.GenServer',call,3,[{file,"lib/gen_server.ex"},{line
,1024}]},{'Elixir.ExAws.Config',retrieve_runtime_value,2,[{file,"lib/ex_aws/config.ex"},{line,92}]},{'Elixir.Stream','-map/2-fun-0-',4,[{file,"lib/stream.ex"},{line,572}]},{'Elixir.Enumerabl
e.List',reduce,3,[{file,"lib/enum.ex"},{line,4251}]},{'Elixir.Enumerable.Stream',do_each,4,[{file,"lib/stream.ex"},{line,1719}]},{'Elixir.Enum',find,3,[{file,"lib/enum.ex"},{line,1105}]},{'E
lixir.ExAws.Config','-retrieve_runtime_config/1-fun-0-',2,[{file,"lib/ex_aws/config.ex"},{line,79}]},{maps,fold_1,3,[{file,"maps.erl"},{line,410}]}]}
2021-08-03 18:19:07.660 [error] <0.1018.0> CRASH REPORT Process <0.1018.0> with 1 neighbours exited with reason: {{undef,[{'Elixir.Jason','decode!',[<<"{\n  \"Code\" : \"Success\",\n  \"Last
Updated\" : \"2021-08-03T16:11:24Z\",\n  \"Type\" : \"AWS-HMAC\",\n  \"AccessKeyId\" : \"...\",\n  \"SecretAccessKey\" : \"....\",\n  \"T
oken\" : \"....">>],...},...]},...} in 'Elixir.GenServer':call/3 line 1024
2021-08-03 18:19:07.660 [error] <0.554.0> Supervisor 'Elixir.ExAws.Supervisor' had child 'Elixir.ExAws.Config.AuthCache' started with 'Elixir.ExAws.Config.AuthCache':start_link([]) at <0.101
6.0> exit with reason call to undefined function 'Elixir.Jason':'decode!'(<<"{\n  \"Code\" : \"Success\",\n  \"LastUpdated\" : \"2021-08-03T16:11:24Z\",\n  \"Type\" : \"AWS-HMAC\",\n  \"Acc.
..">>) in context child_terminated
noxdafox commented 3 years ago

I start getting a grasp of what might be happening in here.

ex_aws claims in their README the default JSON decoder is Poison: https://github.com/ex-aws/ex_aws/blob/b40c33e55ace5820d20888ba2a27c5922daba233/README.md#json-codec-configuration Yet in their defaults they set Jason (changed 2 years ago): https://github.com/ex-aws/ex_aws/blob/554465b8783b903ed831e518f463fb2878d6f50e/lib/ex_aws/config/defaults.ex#L10

In mix, they set Jason as an optional dependency: https://github.com/ex-aws/ex_aws/blob/master/mix.exs#L40

When the plugin is built, Jason is not downloaded (Poison is instead as it's a dependency of the Plugin) and therefore not included in the release. I will replace Poison with Jason and make a fix on ex_aws.

I still need to verify why it works in Docker but I suspect it's because Jason is installed in there as native library.

noxdafox commented 3 years ago

Release updated, Poison replaced with Jason. Tested locally and on a Docker container and it seems to be working.

Could you please confirm the fix?

falcocoris commented 3 years ago

I confirm the fix ! Thx for your work mate, I'll close this up :)

Sonic0 commented 3 years ago

Yes, I confirm too! Both RabbitMQ versions (3.8.x and 3.9.1) work! Thanks @noxdafox