search

sensu / sensu-dashboard

A dashboard for Sensu, for displaying & managing events & clients.
http://sensuapp.org
MIT License
96 stars 44 forks source link

sensu-dashboard dies with "undefined method..." #150

Open amosshapira opened 11 years ago

amosshapira commented 11 years ago

I'm setting up a test Sensu server on a Ubuntu 12.04 vagrant box (and Puppet 2.7) and keep getting the following in the logs once I hit the first page (which render with an "Oh no!" message and a red Bootstrap error pup-up at the bottom right corner of the page):

"timestamp":"2013-09-09T01:49:59.858940+0000","level":"info","message":"GET /all","remote_address":"33.33.33.1","user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.65 Safari/537.36","request_method":"GET","request_uri":"/all?_=1378691404350","request_body":""}
/opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-dashboard-0.10.0/lib/sensu-dashboard/server.rb:210:in `block (3 levels) in <class:Server>': undefined method `response' for nil:NilClass (NoMethodError)
        from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-dashboard-0.10.0/lib/sensu-dashboard/server.rb:209:in `each'
        from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-dashboard-0.10.0/lib/sensu-dashboard/server.rb:209:in `detect'
        from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-dashboard-0.10.0/lib/sensu-dashboard/server.rb:209:in `block (2 levels) in <class:Server>'
        from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/eventmachine-1.0.3/lib/em/deferrable.rb:151:in `call'
        from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/eventmachine-1.0.3/lib/em/deferrable.rb:151:in `set_deferred_status'
        from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/eventmachine-1.0.3/lib/em/deferrable.rb:191:in `succeed'
        from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/em-http-request-1.0.3/lib/em-http/multi.rb:53:in `check_progress'
        from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/em-http-request-1.0.3/lib/em-http/multi.rb:42:in `block in add'
        from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/eventmachine-1.0.3/lib/em/deferrable.rb:158:in `call'
        from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/eventmachine-1.0.3/lib/em/deferrable.rb:158:in `set_deferred_status'
        from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/eventmachine-1.0.3/lib/em/deferrable.rb:198:in `fail'
        from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/em-http-request-1.0.3/lib/em-http/client.rb:123:in `on_error'
        from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/em-http-request-1.0.3/lib/em-http/client.rb:117:in `unbind'
        from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/em-http-request-1.0.3/lib/em-http/http_connection.rb:168:in `block in unbind'
        from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/em-http-request-1.0.3/lib/em-http/http_connection.rb:168:in `map'
        from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/em-http-request-1.0.3/lib/em-http/http_connection.rb:168:in `unbind'
        from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/em-http-request-1.0.3/lib/em-http/http_connection.rb:31:in `unbind'
        from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/eventmachine-1.0.3/lib/eventmachine.rb:1438:in `event_callback'
        from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/eventmachine-1.0.3/lib/eventmachine.rb:187:in `run_machine'
        from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/eventmachine-1.0.3/lib/eventmachine.rb:187:in `run'
        from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-dashboard-0.10.0/lib/sensu-dashboard/server.rb:36:in `run'
        from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-dashboard-0.10.0/bin/sensu-dashboard:10:in `<top (required)>'
        from /opt/sensu/bin/sensu-dashboard:23:in `load'
        from /opt/sensu/bin/sensu-dashboard:23:in `<main>'

And of course the dashboard process dies, leaving the stale pid file behind.

Is this expected? How stable is it in general and can we rely on it for production purposes?

gposton commented 10 years ago

+1 sensu 10.2 on centOS

skymob commented 10 years ago

This is also happening for me on a Vagrant running the the Sensu stack at 0.10.2-1:

[root@sensu-server ~]# cat /etc/redhat-release
CentOS release 6.4 (Final)
[root@sensu-server ~]# rpm -qa | grep sensu
sensu-0.10.2-1.x86_64

I've not seen this problem on my full testing or production deployments, which are both running Sensu 0.10.0-1.

skymob commented 10 years ago

Here's a gist of the full stacktrace and line before it from /var/log/sensu/sensu-dashboard.log: https://gist.github.com/skymob/6617807

amosshapira commented 10 years ago

BTW - I'm not sure which of the steps I made is the one which "did it" but in general it seems that once I got the authentication to RabbitMQ sorted out the dashboard came to life. On another instance we also got this problem even though the authentication seemed to have been done right and the only way we got around it was to nuke the Vagrant instance and start from a fresh Vagrant (+Puppet) install. Voodoo.

skymob commented 10 years ago

@amosshapira ah, thanks. I didn't even notice that RabbitMQ had died because I'd changed the Vagrant hostname manually. I destroyed and re-upped the Vagrant and now Dashboard is fine. Dashboard should probably be able to handle that situation though.

amdprophet commented 10 years ago

A fix for this is being worked on in Pull Request #152.

amdprophet commented 10 years ago

@amosshapira @skymob are either of you able to test Sensu Dashboard 0.10.4 to see if this issue is fixed?

amosshapira commented 10 years ago

@amdprophet sorry I don't have a ready environment to test it right now (changed jobs, wiped my laptop, went on holiday).

amdprophet commented 10 years ago

@amosshapira No problem, thanks for reporting the bug!