RabbitMQ 3.7.4 - lager_syslog_backend failing to install

[jaredledvina]

System Info:

• RabbitMQ version: 3.7.4-1.el7
• Erlang version: 20.2.3-1.el7.centos
• RabbitMQ plugin information via rabbitmq-plugins list

  # rabbitmq-plugins list
  Configured: E = explicitly enabled; e = implicitly enabled
  | Status: * = running on rabbit@ops-ovb-sensu-1
  |/
  [  ] rabbitmq_amqp1_0                  3.7.4
  [  ] rabbitmq_auth_backend_cache       3.7.4
  [  ] rabbitmq_auth_backend_http        3.7.4
  [  ] rabbitmq_auth_backend_ldap        3.7.4
  [  ] rabbitmq_auth_mechanism_ssl       3.7.4
  [  ] rabbitmq_consistent_hash_exchange 3.7.4
  [  ] rabbitmq_event_exchange           3.7.4
  [  ] rabbitmq_federation               3.7.4
  [  ] rabbitmq_federation_management    3.7.4
  [  ] rabbitmq_jms_topic_exchange       3.7.4
  [E*] rabbitmq_management               3.7.4
  [e*] rabbitmq_management_agent         3.7.4
  [  ] rabbitmq_mqtt                     3.7.4
  [  ] rabbitmq_peer_discovery_aws       3.7.4
  [  ] rabbitmq_peer_discovery_common    3.7.4
  [  ] rabbitmq_peer_discovery_consul    3.7.4
  [  ] rabbitmq_peer_discovery_etcd      3.7.4
  [  ] rabbitmq_peer_discovery_k8s       3.7.4
  [  ] rabbitmq_random_exchange          3.7.4
  [  ] rabbitmq_recent_history_exchange  3.7.4
  [  ] rabbitmq_sharding                 3.7.4
  [  ] rabbitmq_shovel                   3.7.4
  [  ] rabbitmq_shovel_management        3.7.4
  [  ] rabbitmq_stomp                    3.7.4
  [  ] rabbitmq_top                      3.7.4
  [  ] rabbitmq_tracing                  3.7.4
  [  ] rabbitmq_trust_store              3.7.4
  [e*] rabbitmq_web_dispatch             3.7.4
  [  ] rabbitmq_web_mqtt                 3.7.4
  [  ] rabbitmq_web_mqtt_examples        3.7.4
  [  ] rabbitmq_web_stomp                3.7.4
  [  ] rabbitmq_web_stomp_examples       3.7.4

• Operating system, version, and patch level: CentOS 7.4.1708

• /etc/rabbitmq/rabbitmq.config:

  % This file managed by Puppet
  % Template Path: rabbitmq/templates/rabbitmq.config
  [
  {rabbit, [
  {cluster_nodes, {['rabbit@server1', 'rabbit@server2', 'rabbit@server3'], disc}},
  {cluster_partition_handling, pause_minority},
  {tcp_listen_options,
       [binary,
       {packet,        raw},
       {reuseaddr,     true},
       {backlog,       128},
       {nodelay,       true},
       {exit_on_close, false}]
  },
  {tcp_listeners, []},
  {ssl_listeners, [5671]},
  {ssl_options, [
                 {cacertfile,"/etc/ipa/ca.crt"},
                 {certfile,"/etc/pki/tls/certs/rabbitmq.pem"},
                 {keyfile,"/etc/pki/tls/private/rabbitmq.key"},
                 {verify,verify_peer},
                 {fail_if_no_peer_cert,true}
                ]},
  {log, [{syslog, [{enabled, true}]}]},
  {default_user, <<"superdupersecret">>},
  {default_pass, <<"superdupersecret">>}
  ]},
  {kernel, [
  
  ]}
  ,
  {rabbitmq_management, [
  {listener, [
    {port, 15671},
    {ssl, true},
    {ssl_opts, [
                {cacertfile, "/etc/ipa/ca.crt"},
  
                {certfile, "/etc/pki/tls/certs/rabbitmq.pem"},
                {keyfile, "/etc/pki/tls/private/rabbitmq.key"}
               ]}
  ]}
  ]}
  ].
  % EOF

  Problem:

Hi! I'm trying to configure RabbitMQ to log natively to syslog, reading through https://www.rabbitmq.com/logging.html it seems like this should be simple and only require me adding the {log, [{syslog, [{enabled, true}]}]}, section to the config. However, after doing so, and restarting rabbitmq-server I get the following in the RabbitMQ logs:

2018-03-15 15:26:05.771 [error] <0.99.0> Lager failed to install handler lager_syslog_backend into lager_event, retrying later : {'EXIT',
                                                                                         {undef,
                                                                                          [{lager_syslog_backend,
                                                                                            init,
                                                                                            [[{facility,
                                                                                               daemon},
                                                                                              {formatter_config,
                                                                                               [date,
                                                                                                " ",
                                                                                                time,
                                                                                                " ",
                                                                                                color,
                                                                                                "[",
                                                                                                severity,
                                                                                                "] ",
                                                                                                {pid,
                                                                                                 []},
                                                                                                " ",
                                                                                                message,
                                                                                                "\n"]},
                                                                                              {identity,
                                                                                               "rabbitmq"},
                                                                                              {level,
                                                                                               info}]],
                                                                                            []},
                                                                                           {gen_event,
                                                                                            server_add_handler,
                                                                                            4,
                                                                                            [{file,
                                                                                              "gen_event.erl"},
                                                                                             {line,
                                                                                              469}]},
                                                                                           {gen_event,
                                                                                            handle_msg,
                                                                                            6,
                                                                                            [{file,
                                                                                              "gen_event.erl"},
                                                                                             {line,
                                                                                              314}]},
                                                                                           {proc_lib,
                                                                                            init_p_do_apply,
                                                                                            3,
                                                                                            [{file,
                                                                                              "proc_lib.erl"},
                                                                                             {line,
                                                                                              247}]}]}}
2018-03-15 15:26:05.778 [error] <0.127.0> Lager failed to install handler lager_syslog_backend into rabbit_log_upgrade_lager_event, retrying later : {'EXIT',
                                                                                                            {undef,
                                                                                                             [{lager_syslog_backend,
                                                                                                               init,
                                                                                                               [[{facility,
                                                                                                                  daemon},
                                                                                                                 {formatter_config,
                                                                                                                  [date,
                                                                                                                   " ",
                                                                                                                   time,
                                                                                                                   " ",
                                                                                                                   color,
                                                                                                                   "[",
                                                                                                                   severity,
                                                                                                                   "] ",
                                                                                                                   {pid,
                                                                                                                    []},
                                                                                                                   " ",
                                                                                                                   message,
                                                                                                                   "\n"]},
                                                                                                                 {identity,
                                                                                                                  "rabbitmq"},
                                                                                                                 {level,
                                                                                                                  info}]],
                                                                                                               []},
                                                                                                              {gen_event,
                                                                                                               server_add_handler,
                                                                                                               4,
                                                                                                               [{file,
                                                                                                                 "gen_event.erl"},
                                                                                                                {line,
                                                                                                                 469}]},
                                                                                                              {gen_event,
                                                                                                               handle_msg,
                                                                                                               6,
                                                                                                               [{file,
                                                                                                                 "gen_event.erl"},
                                                                                                                {line,
                                                                                                                 314}]},
                                                                                                              {proc_lib,
                                                                                                               init_p_do_apply,
                                                                                                               3,
                                                                                                               [{file,
                                                                                                                 "proc_lib.erl"},
                                                                                                                {line,
                                                                                                                 247}]}]}}

I also tested out enabling the console logging with the config: {log, [{console, [{enabled, true}]}]}, and those are viewable without issue via journalctl -o cat -fu rabbitmq-server.service. My current understanding is that this is specifically an issue only with the lager_syslog_backend.

Happy to provide any other information, this is a very basic RabbitMQ server setup, with a single user, single vhost, in a 3 node cluster.

[michaelklishin]

Lager does not support syslog out of the box and we don't have the consensus about shipping Lager plugins with RabbitMQ.

[michaelklishin]

FTR, the exception says "module lager_syslog_backend is undefined" (undef).

[jaredledvina]

Hey @michaelklishin,

Thanks so much for the quick reply! Would it be worth me PR'ing an update to https://github.com/rabbitmq/rabbitmq-website/blob/live/site/logging.md#logging-to-syslog to explicitly mark that configuration as not supported for 3.7?

[michaelklishin]

Yes, we’d definitely appreciate a doc update. Thank you!

Perhaps putting together an example of Lager + Syslog for 3.7.x would be useful, it definitely can be a small community plugin even if we decide to not include the [Lager] plugin into RabbitMQ core or distribution.

I’m not a Lager expert, so you’ll have to bribe @lukebakken or @hairyhum :)

[majormoses]

Should we not re-open this until we fixup the changelog and documentation about this? I don't think its a good stance to break working systems, not call our the changes, and close the issue before steps are taken to warn users.

[michaelklishin]

@majormoses syslog was never supported out of the box. We didn't break anything.

[michaelklishin]

I edited out all Syslog mentions in the logging guide (which was introduced together with 3.7)

If someone wants to discuss a RabbitMQ plugin that would allow logging to syslog, you are welcome to do it on the mailing list.

[majormoses]

Thanks for the clarification and quick turn around to updating the documentation.