sensu-client fails to reconnect to transport and dies

trombik commented 7 years ago

ISSUE TYPE

Bug Report

ROLE VERSION

1.2.0

CONFIGURATION

see tests/integration/example.

OS / ENVIRONMENT

FreeBSD, but it is likely to happen on other platforms.

SUMMARY

when sensu-client on sensu1 initially boots in the integration test, it fails to start.

STEPS TO REPRODUCE

remove files/FreeBSD/patch_undefined_method_for_nilclass and patch task in tasks/install-FreeBSD.yml
run the integration test

{"timestamp":"2017-07-12T01:50:45.061714+0000","level":"warn","message":"loading config file","file":"/usr/local/etc/sensu/config.json"}
{"timestamp":"2017-07-12T01:50:45.061789+0000","level":"warn","message":"loading config files from directory","directory":"/usr/local/etc/sensu/conf.d"}
{"timestamp":"2017-07-12T01:50:45.061822+0000","level":"warn","message":"applied sensu client overrides","client":{"name":"sensu1.virtualbox.reallyenglish.com","address":"10.0.2.15","subscriptions":["client:sensu1.virtualbox.reallyenglish.com"]}}
{"timestamp":"2017-07-12T01:50:45.063673+0000","level":"warn","message":"loading extension files from directory","directory":"/usr/local/etc/sensu/extensions"}
{"timestamp":"2017-07-12T01:50:45.063738+0000","level":"info","message":"configuring sensu spawn","settings":{"limit":12}}
{"timestamp":"2017-07-12T01:50:45.146030+0000","level":"warn","message":"transport connection error","reason":"tcp connection lost"}
/opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-transport-7.0.2/lib/sensu/transport/rabbitmq.rb:210:in `block in setup_connection': undefined method `[]' for nil:NilClass (NoMethodError)
        from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/amqp-1.6.0/lib/amqp/session.rb:1054:in `handle_close'
        from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/amqp-1.6.0/lib/amqp/session.rb:1140:in `block in <class:Session>'
        from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/amqp-1.6.0/lib/amqp/session.rb:959:in `receive_frameset'
        from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/amqp-1.6.0/lib/amqp/session.rb:933:in `receive_frame'
        from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/amqp-1.6.0/lib/amqp/session.rb:671:in `receive_data'
        from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/eventmachine-1.2.2/lib/eventmachine.rb:194:in `run_machine'
        from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/eventmachine-1.2.2/lib/eventmachine.rb:194:in `run'
        from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-1.0.0/lib/sensu/client/process.rb:19:in `run'
        from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-1.0.0/exe/sensu-client:10:in `<top (required)>'
        from /opt/sensu/bin/sensu-client:22:in `load'
        from /opt/sensu/bin/sensu-client:22:in `<main>'

EXPECTED RESULTS

sensu-client starts and reconnects to the transport.

ACTUAL RESULTS

it fails to reconnect and dies.

trombik commented 7 years ago

see https://github.com/reallyenglish/ansible-role-sensu-client/pull/7/commits/de8c54b80d6551e146b15ea9b50988245fb62173

trombik commented 7 years ago

this happens when sensu-client json files are empty. specifically, when rabbitmq user is not defined. although the issue is caused by invalid configuration, sensu-transport should have been more careful.

trombik commented 7 years ago

the patch is just a workaround. the issue should be fixed in validation logic and ensure options[:user] is never be nil or an empty hash. pending.

trombik commented 7 years ago

workarounded in #7

reallyenglish / ansible-role-sensu-client