search

collectiveidea / delayed_job

Database based asynchronous priority queue system -- Extracted from Shopify
http://groups.google.com/group/delayed_job
MIT License
4.81k stars 957 forks source link

Delayed Job 3.0 Memory Leak #336

Closed jimishjoban closed 9 years ago

jimishjoban commented 12 years ago

I recently upgraded to use delayed_job 3.0 gem and I am facing this issue of memory leak. Even if I dont start delayed job, my rails app takes on entire memory of 2GB and ultimately crashes.

The moment I downgrade delayed_job gem to 2.1.4, the rails app behaves normally. Is this something related to delayed_job or could it also be my rails app? (remember, i dont even need to start delayed_job to eat up all the memory using the 3.0 gem)

My environment:

Rails 3.1 ruby 1.9.2p290 (2011-07-09 revision 32553) [i686-linux] Delayed_job 3.0.0 Daemons 1.0.10 delayed_job_active_record (0.3.1)

How should I go debugging the problem? Any idea?

bryckbost commented 12 years ago

Check the jobs table and daemons log. Could be failed jobs that are causing memory issues.

fcheung commented 12 years ago

This probably isn't really delayed job's fault. Prior to 3.0 delayed job forced the use of the syck YAML engine. 3.0 doesn't any more so you get the default of psych instead (unless you are specifying this elsewhere). Unfortunately the version of psych in currently released versions of 1.9.2 has quite a big memory leak in YAML.load. One fix is to use the psych gem (which has a newer version of psych), you can also switch back to the old syck YAML engine and I believe ruby 1.9.3 also has a fixed version of psych

jjb commented 12 years ago

Perhaps DJ could require the psych gem if the version of rails detected has the memory leak?

fcheung commented 12 years ago

It's gets a bit fiddly - bundler loads psych during startup so by the time delayed job turns up it's too late. (for that reason it's also not enough to just add it to your Gemfile). Printing a warning with information on where to get more details would be easy though

ismell commented 10 years ago

I'm assuming this can be closed as of 1.9.3?