Transport should log connection failure error

[cwjohnston]

Now that we can expect errors with more context via ruby-amqp (see https://github.com/sensu/sensu/issues/842 and #29), it seems like we should probably surface those errors to the log. For example, I installed Sensu 0.24 on a client, validated configuration, then changed the RabbitMQ password and restarted sensu-client. Even with debug logging, there's no indication that the connection is being retried or that there was an error during connection attempts:

{"timestamp":"2016-06-10T01:10:42.278285+0000","level":"warn","message":"loading config file","file":"/etc/sensu/config.json"}
{"timestamp":"2016-06-10T01:10:42.278686+0000","level":"warn","message":"loading config files from directory","directory":"/etc/sensu/conf.d"}
{"timestamp":"2016-06-10T01:10:42.278929+0000","level":"warn","message":"loading config file","file":"/etc/sensu/conf.d/client.json"}
{"timestamp":"2016-06-10T01:10:42.279222+0000","level":"warn","message":"config file applied changes","file":"/etc/sensu/conf.d/client.json","changes":{"client":[null,{"name":"test","address":"192.168.10.18","subscriptions":["test"]}]}}
{"timestamp":"2016-06-10T01:10:42.279464+0000","level":"warn","message":"loading config file","file":"/etc/sensu/conf.d/rabbitmq.json"}
{"timestamp":"2016-06-10T01:10:42.279707+0000","level":"warn","message":"config file applied changes","file":"/etc/sensu/conf.d/rabbitmq.json","changes":{}}
{"timestamp":"2016-06-10T01:10:42.293507+0000","level":"warn","message":"loading extension files from directory","directory":"/etc/sensu/extensions"}
{"timestamp":"2016-06-10T01:10:42.294156+0000","level":"info","message":"configuring sensu spawn","settings":{"limit":12}}
{"timestamp":"2016-06-10T01:10:42.294411+0000","level":"debug","message":"setting eventmachine threadpool size","size":22}
{"timestamp":"2016-06-10T01:10:42.346103+0000","level":"debug","message":"connecting to transport","name":"rabbitmq","settings":{"host":"192.168.11.15","port":5672,"user":"sensu","password":"secret","vhost":"/sensu"}}

[ This is where retries are happening silently, no log messages indicating password is wrong ]
[ Once I change the password back to "secret", the transport is connected and startup continues ]

{"timestamp":"2016-06-10T01:12:32.566374+0000","level":"debug","message":"binding client tcp and udp sockets","options":{"bind":"127.0.0.1","port":3030}}
{"timestamp":"2016-06-10T01:12:32.567658+0000","level":"debug","message":"scheduling keepalives"}
{"timestamp":"2016-06-10T01:12:32.567919+0000","level":"debug","message":"publishing keepalive","payload":{"name":"test","address":"192.168.10.18","subscriptions":["test"],"version":"0.24.1","timestamp":1465521152}}
{"timestamp":"2016-06-10T01:12:32.568329+0000","level":"debug","message":"subscribing to client subscriptions"}
{"timestamp":"2016-06-10T01:12:32.568473+0000","level":"debug","message":"subscribing to a subscription","subscription":"test"}
{"timestamp":"2016-06-10T01:12:32.568865+0000","level":"debug","message":"scheduling standalone checks"}
{"timestamp":"2016-06-10T01:12:32.569003+0000","level":"info","message":"reconnected to transport"}

To some extent this might apply only to RabbitMQ, I haven't tested other transports for same behavior.

Relates to https://github.com/sensu/sensu/issues/1058, https://github.com/sensu/sensu/issues/842

[moises-silva]

@cwjohnston I've found this issue annoying in the past as well. I've created PR #34 to address this.

[portertech]

At a minimum, we should be logging any issues/errors in regards to AMQP connectivity, specially while periodically attempting to reconnect.

[portertech]

I merged https://github.com/sensu/sensu-transport/pull/33

We need to update Sensu Core 0.27 to use these changes.

[portertech]

We can ship a sensu-transport release with the changes in https://github.com/sensu/sensu-transport/pull/33 and the changes to enable AMQP reconnecting logging.

[moises-silva]

@portertech PR #34 is also related iirc, but the build is failing. It seem @logger is not defined somehow? it's been a while since I made the changes but they were working for sure. I wonder if I missed introducing the @logger in that scope somehow or if this is a test environment problem.

[cwjohnston]

Closed by #34.