concerto / bandshell

A Local Management Utility for the Concerto Player
Other
7 stars 8 forks source link

Screen control stops working if bandshell-timer.rb crashes #50

Open asquared opened 8 years ago

asquared commented 8 years ago

I noticed on one of the RPI screens that was getting stuck "off" that bandshell-timer.rb wasn't running. It had crashed with what appears to be a HTTP timeout:

bandshell-timer.rb: process with pid 2370 started.

bandshell-timer.rb (2015-12-07 04:03:22 +0000): Bandshell timer starting up.
bandshell-timer.rb (2015-12-07 04:03:22 +0000): connecting to bandshell at http://localhost:4567
/usr/lib/ruby/1.9.1/net/protocol.rb:146:in `rescue in rbuf_fill': Timeout::Error (Timeout::Error)
    from /usr/lib/ruby/1.9.1/net/protocol.rb:140:in `rbuf_fill'
    from /usr/lib/ruby/1.9.1/net/protocol.rb:122:in `readuntil'
    from /usr/lib/ruby/1.9.1/net/protocol.rb:132:in `readline'
    from /usr/lib/ruby/1.9.1/net/http.rb:2562:in `read_status_line'
    from /usr/lib/ruby/1.9.1/net/http.rb:2551:in `read_new'
    from /usr/lib/ruby/1.9.1/net/http.rb:1319:in `block in transport_request'
    from /usr/lib/ruby/1.9.1/net/http.rb:1316:in `catch'
    from /usr/lib/ruby/1.9.1/net/http.rb:1316:in `transport_request'
    from /usr/lib/ruby/1.9.1/net/http.rb:1293:in `request'
    from /usr/lib/ruby/1.9.1/net/http.rb:1195:in `request_get'
    from /usr/lib/ruby/1.9.1/net/http.rb:455:in `block in get_response'
    from /usr/lib/ruby/1.9.1/net/http.rb:745:in `start'
    from /usr/lib/ruby/1.9.1/net/http.rb:454:in `get_response'
    from /var/lib/gems/1.9.1/gems/bandshell-1.6/bin/bandshell-timer.rb:36:in `block in <top (required)>'
    from /var/lib/gems/1.9.1/gems/bandshell-1.6/bin/bandshell-timer.rb:27:in `loop'
    from /var/lib/gems/1.9.1/gems/bandshell-1.6/bin/bandshell-timer.rb:27:in `<top (required)>'
    from /var/lib/gems/1.9.1/gems/daemons-1.2.3/lib/daemons/application.rb:218:in `load'
    from /var/lib/gems/1.9.1/gems/daemons-1.2.3/lib/daemons/application.rb:218:in `start_load'
    from /var/lib/gems/1.9.1/gems/daemons-1.2.3/lib/daemons/application.rb:297:in `start'
    from /var/lib/gems/1.9.1/gems/daemons-1.2.3/lib/daemons/controller.rb:56:in `run'
    from /var/lib/gems/1.9.1/gems/daemons-1.2.3/lib/daemons.rb:144:in `block in run'
    from /var/lib/gems/1.9.1/gems/daemons-1.2.3/lib/daemons/cmdline.rb:88:in `call'
    from /var/lib/gems/1.9.1/gems/daemons-1.2.3/lib/daemons/cmdline.rb:88:in `catch_exceptions'
    from /var/lib/gems/1.9.1/gems/daemons-1.2.3/lib/daemons.rb:143:in `run'
    from /var/lib/gems/1.9.1/gems/bandshell-1.6/bin/bandshelld:22:in `<top (required)>'
    from /usr/local/bin/bandshelld:23:in `load'
    from /usr/local/bin/bandshelld:23:in `<main>'

It's probably not good that this is crashing the whole program, but more importantly, perhaps we should look into running this every 10 seconds or so from crontab and avoiding the long-running process. I'm less worried about the actual Bandshell app crashing and burning because Sinatra should have some error handling that limits that to one request.

Also of note is that this seems to have happened just seconds after boot. Maybe it's racing against the main bandshell process's startup?

mikldt commented 8 years ago

I think that's likely: Sinatra likely takes longer to come up, and the way bandshelld is set up it won't wait for Sinatra to boot to start the timer. I agree with the catch-all in f344578 - this is the second time we've come across a one-time exception that would be resolved by keeping the process alive a little longer.

My only issue with cron would be that it makes it a little harder to run bandshell standalone for development, but if we can come up with something that can work either way, I'd be ok with that.

asquared commented 8 years ago

I'm not really opposed to the daemon as long as there's a mechanism to keep it running. Perhaps once we move the player to Debian jessie we can have systemd supervise the two daemons and restart them if they crash.

On Fri, Dec 18, 2015 at 11:10 AM, Mike notifications@github.com wrote:

I think that's likely: Sinatra likely takes longer to come up, and the way bandshelld is set up it won't wait for Sinatra to boot to start the timer. I agree with the catch-all in f344578 https://github.com/concerto/bandshell/commit/f344578291f71c078c8948e2da3694115fb30fd6

  • this is the second time we've come across a one-time exception that would be resolved by keeping the process alive a little longer.

My only issue with cron would be that it makes it a little harder to run bandshell standalone for development, but if we can come up with something that can work either way, I'd be ok with that.

— Reply to this email directly or view it on GitHub https://github.com/concerto/bandshell/issues/50#issuecomment-165819177.