Work not being done in production

[moomerman]

I've just tested sending mail using girl_friday in development and everything works great, I deployed to production and it isn't working (Ruby 1.9.2, Unicorn 3.5.0, Rails 3.0.7). Strangely, it works fine in production from the console but in the HTTP request it isn't doing anything. I've debugged it all the way to the point that it gets enqueued and that seems to be correct, but debug in the GirlFriday::WorkQueue do |msg| block isn't printed. Very strange... any ideas?

[moomerman]

EMAIL_QUEUE.status

{:user_email=>{:pid=>7527, :pool_size=>5, :ready=>5, :busy=>0, :backlog=>0, :total_queued=>0, :total_processed=>0, :total_errors=>0, :uptime=>34, :created_at=>1303393887}}

EMAIL_QUEUE << {:some => 'work}

EMAIL_QUEUE.status

{:user_email=>{:pid=>7527, :pool_size=>5, :ready=>5, :busy=>0, :backlog=>0, :total_queued=>0, :total_processed=>0, :total_errors=>0, :uptime=>34, :created_at=>1303393887}}

So it's like it is just getting sent into the ether.

[mperham]

I think this may be an issue with how unicorn works. I suspect you need to create the queues inside an after_fork block. I will confirm and follow up soon.

[mperham]

This is what I mean:

http://unicorn.bogomips.org/Unicorn/Configurator.html#method-i-after_fork

Try putting your queue declarations in there. I'll reproduce and document a fix for the issue this weekend.

[moomerman]

Thanks for the pointer. I've just quickly tried:

after_fork do |server, worker|
  ...
   Moocode::Application::EMAIL_QUEUE = ....
  ...
end

But it didn't work. Will do some more investigation and report back.

Thanks!

[moomerman]

Bingo. It does work if I use the full namespace Moocode::Application::EMAIL_QUEUE to access the queue from within my code as well.

Thanks!

[mperham]

I just pushed a fix which should more gracefully handle this situation. You shouldn't need to do anything in an after_fork block now as I'm lazy initializing the worker actors now. I'll release a new gem version this weekend.

[moomerman]

I've just quickly tried a deployment against the latest git version and reverted my previous changes and it doesn't seem to be working - just letting you know before a release. Should have a bit more time tomorrow to do some digging.

[mperham]

Doh. I presume it still works in the after_fork block. It should not create the worker threads until you first pass the queue some work which I'm assuming should be well after the unicorn worker process has been forked. I'll add some tests to verify this.

[mperham]

I tried to reproduce your Unicorn behavior on OSX in production mode but it worked with both eager and lazy created workers. I'm not sure what the problem might be. It should be safer to lazy create the workers so I'll leave it the way it is now. Let me know if you find out anything more.

[betelgeuse]

I can't get work processed with unicorn in production either. Here's what status tells me:

{"queue":{"pid":13903,"pool_size":1,"ready":1,"busy":0,"backlog":0,"total_queued":0,"total_processed":0,"total_errors":0,"uptime":947,"created_at":1310374032}}

The queue is created in an initializer as according to this issue it should work. I guess I need to try after_fork next.

[betelgeuse]

I tried on Gentoo with unicorn in production mode and after_fork and could not get anything this time either. I tried configuring it with Redis but no difference.

$ bin/unicorn -E production  -c config/unicorn.rb -l localhost:5000

$ curl -i http://localhost:5000/basic/status
HTTP/1.1 200 OK
Date: Sat, 23 Jul 2011 19:28:28 GMT
Status: 200 OK
Connection: close
Content-Type: application/json; charset=utf-8
X-UA-Compatible: IE=Edge,chrome=1
ETag: "ea647823a6cc3426a9b49456269555f7"
Cache-Control: must-revalidate, private, max-age=0
X-Runtime: 0.002087
X-Rack-Cache: miss

{"flight_data":{"pid":29711,"pool_size":1,"ready":1,"busy":0,"backlog":0,"total_queued":0,"total_processed":0,"total_errors":0,"uptime":21,"created_at":1311449287}}