jconley commented 15 years ago

Received this in our delayed_job.log... and then the DJ process dies. My best guess so far is there was a momentary network disruption between firewalls of our app and db networks.

<ActiveRecord::StatementInvalid: Mysql::Error: MySQL server has gone away: UPDATE `delayed_jobs` SET locked_by = null, locked_at = null WHERE (locked_by = 'delayed_job host:myhost pid:12345') >

Would be nice if DJ had more robust connection handling... i.e. catch this issue, and retry the connection until it is successful... otherwise we have to write a script to restart DJ periodically, or detect when jobs in the queue are stale.

smasry commented 14 years ago

Experiencing the same issue even when the db is on the same box.

cyberfox commented 14 years ago

Try setting 'reconnect: true' in your database.yml for the appropriate environment; under Rails 2.3.x that should alleviate the problem. (I'm setting that up now myself, so I'll know for sure in a few days, but it makes sense.)

kelyar commented 14 years ago

"reconnect: true" works, but leads to problems with transactions, so I don't think that is a proper solution

cyberfox commented 14 years ago

True, my use case was specific: the only database actions being done in the application were against the DJ table via DJ, so the issue with transactions was not relevant.

You could still do it if your DJ database was on a separate connection from your normal DB, but that certainly becomes more complex to set up.

TeuF commented 14 years ago

Same issue with rails 3 beta4 .... Can't make it work.

$ script/delayed_job start delayed_job: process with pid 20648 started.

=> SQL (111.9ms) UPDATE delayed_jobs SET locked_by = null, locked_at = null WHERE (locked_by = 'delayed_job host:desk pid:20648') Mysql::Error: Lost connection to MySQL server during query: SELECT delayed_jobs.* FROM delayed_jobs WHERE ((run_at <= '2010-08-06 14:26:41' AND (locked_at IS NULL OR locked_at < '2010-08-06 10:26:41') OR locked_by = 'delayed_job host:desk pid:20648') AND failed_at IS NULL) ORDER BY priority ASC, run_at ASC LIMIT 5

And when I start DJ with rake, every things working:

$ rake jobs:work [Worker(host:desk pid:20663)] Starting job worker

=>SQL (24.5ms) SHOW TABLES DEPRECATION WARNING: RAILS_DEFAULT_LOGGER is deprecated! Use Rails.logger instead. (called from say at /home/www/.bundle/ruby/1.8/bundler/gems/delayed_job-0ca4e5f2bd8a5e17dca7c1abab692a049c1669e1-rails3-mailer/lib/delayed/worker.rb:161) 2010-08-06T16:28:04+0200: [Worker(host:desk pid:20663)] Starting job worker

threetee commented 14 years ago

I'm seeing the same behavior with Rails 2.3.5 and the collectiveidea fork of delayed_job (version 2.0.3). When starting from script/delayed_job, it immediately dies with a "MySQL server has gone away" message, but when starting using the work:jobs rake task, it starts up properly.

Rails application log logs this in conjunction with the failed startup:

Delayed::Backend::ActiveRecord::Job Update (0.0ms) Mysql::Error: MySQL server has gone away: UPDATE delayed_jobs SET locked_by = null, locked_at = null WHERE (locked_by = 'delayed_job host:myhost pid:2293') Mysql::Error: MySQL server has gone away: UPDATE delayed_jobs SET locked_by = null, locked_at = null WHERE (locked_by = 'delayed_job host:myhost pid:2293')

kelyar commented 14 years ago

Solution that I found somewhere on internet was to downgrade daemons gem (which is internal dependancy of DJ, I believe) from 1.1.0 to 1.0.10. That completely solved problems with dying delayed job.

thank you, Anonymous!

smasry commented 13 years ago

I've also experienced many "Mysql Server gone away errors" with activerecord. The fix was to increase the 'max_allowed_packet'. If the stacktrace for an error was too large mysql would terminate the connection.