search

nesquena / backburner

Simple and reliable beanstalkd job queue for ruby
http://nesquena.github.com/backburner
MIT License
428 stars 68 forks source link

If beanstalkd dies, ::Forking workers never reconnect #93

Open mexisme opened 9 years ago

mexisme commented 9 years ago

When using the ::Forking worker, I'm seeing errors like the below when beanstalkd is stopped or dies (e.g. server restart). The worker fails over and over, and never makes a fresh connection.

Exception Beaneater::NotConnected -> Could not connect!
   .bundle/ruby/2.0.0/bundler/gems/beaneater-6eaa016693bd/lib/beaneater/pool.rb:154:in `rescue in safe_transmit'
   .bundle/ruby/2.0.0/bundler/gems/beaneater-6eaa016693bd/lib/beaneater/pool.rb:141:in `safe_transmit'
   .bundle/ruby/2.0.0/bundler/gems/beaneater-6eaa016693bd/lib/beaneater/pool.rb:109:in `transmit_to_rand'
   .bundle/ruby/2.0.0/bundler/gems/beaneater-6eaa016693bd/lib/beaneater/pool_command.rb:40:in `method_missing'
   .bundle/ruby/2.0.0/bundler/gems/beaneater-6eaa016693bd/lib/beaneater/tube/collection.rb:44:in `reserve'
   .bundle/ruby/2.0.0/gems/backburner-0.4.6/lib/backburner/worker.rb:128:in `work_one_job'
   .bundle/ruby/2.0.0/gems/backburner-0.4.6/lib/backburner/workers/forking.rb:33:in `block in fork_one_job'
   .bundle/ruby/2.0.0/gems/backburner-0.4.6/lib/backburner/workers/forking.rb:31:in `fork'
   .bundle/ruby/2.0.0/gems/backburner-0.4.6/lib/backburner/workers/forking.rb:31:in `fork_one_job'
   .bundle/ruby/2.0.0/gems/backburner-0.4.6/lib/backburner/workers/forking.rb:25:in `block in start'
   .bundle/ruby/2.0.0/gems/backburner-0.4.6/lib/backburner/workers/forking.rb:25:in `loop'
   .bundle/ruby/2.0.0/gems/backburner-0.4.6/lib/backburner/workers/forking.rb:25:in `start'
   .bundle/ruby/2.0.0/gems/backburner-0.4.6/lib/backburner/worker.rb:48:in `start'
   .bundle/ruby/2.0.0/gems/backburner-0.4.6/lib/backburner.rb:39:in `work'

I have a suspicion it has something to do with conflation between https://github.com/nesquena/backburner/blob/master/lib/backburner/worker.rb#L58 where you're using a class var (@connection) but in https://github.com/nesquena/backburner/blob/master/lib/backburner/workers/forking.rb#L32 you appear to be trying to override an instance var (also @connection) and yet https://github.com/nesquena/backburner/blob/master/lib/backburner/worker.rb#L126 uses the above self.connection class method (and class var).

Does this mean that all the forked children are trying to reuse the parent's @connection and it's no-longer valid, because the associated beanstalkd has gone/restarted?

It's possible this may be related to #61?

Thanks.

contentfree commented 8 years ago

@mexisme I see what you're saying with the conflation bit there. What it should probably do is pass the newly-minted Connection into the work_one_job method (which accepts an optional Connection instance). The Worker doesn't seem to have/use a @connection instance var.

contentfree commented 8 years ago

@nesquena Memoizing the connection as a static/class variable seems to be causing several problems. What do you think about moving it strictly to an instance variable? Then no more problems like this.

nesquena commented 8 years ago

@contentfree :+1: makes sense, thanks for tracking this down

mexisme commented 8 years ago

Thanks, @contentfree, that looks like a smarter way forward.