Fetcher and passenger issue?

[schneems]

I'm using Fetcher and monit on a rails app (thanks MMS2R). Monit keeps the process up for days, and Fetcher always grabs emails from the inbox i point it at. However after an unspecified period of time (roughly 6+ hours) after restarting the process, it fails to deliver the mail to my receive method. It still pulls them off the postfix server (verified through email log) but when i check my production database, there are no new entries.

For the first 6-ish hours i do see entries in my production DB, and at any time I can manually go into RAILS_ENV=production script/console and successfully test the receive method and verify entries in my production DB. Has anyone run into something similar?

If this isn't the proper place to address this issue please point me in the right direction. rails 2.2.2, ubuntu hardy, passenger, and ruby 1.8.6 .

[look]

Hey there...sorry you're having trouble.

I'm guessing you're using the daemon to run the fetcher, is that correct?

I had problems with long-running processes, so I switched to running the Fetcher via cron.

This worked great for a while, but I started having problems there too. My script would freeze just like yours. Finally, with the help of gdb I was able to figure out that Ruby was hanging on the network connection. I believe this was due to green threads hanging on a system call.

I recommend getting GDB and attaching to the hanging process by PID, then running this command:

call rb_raise((int)rb_eException, "Forced exception from GDB to get a valid ruby stack trace")

You'll get the stack trace. Post it here.

If you haven't already, you can also get the most recent version of Fetcher from github. It uses the SystemTimer gem (http://ph7spot.com/articles/system_timer) to reliably time out the network connection that was hanging my process.

[schneems]

Thanks for getting back to me. I'm using the most recent version of the plugin, do i need to do anything to make use of SystemTimer?

I'm unfamilliar with GDB, i used an apt-get to install, and using it was easy enough, though when i tried that command i get this error:

(gdb) call rb_raise((int)rb_eException, "Forced exception from GDB to get a valid ruby stack trace")

Program received signal SIGSEGV, Segmentation fault. 0x00007f2f56f8add2 in ?? () from /usr/lib/libruby1.8.so.1.8 The program being debugged was signaled while in a function called from GDB. GDB has restored the context to what it was before the call. To change this behavior use "set unwindonsignal off" Evaluation of the expression containing the function (rb_raise) will be abandoned.

I tried with "set unwindonsignal off" and "set unwindonsignal on". As I mentioned i'm new to GDB, is there something else i need to configure?

[look]

Did you install fetcher from GitHub? script/plugin install git://github.com/look/fetcher.git

That's the version that requires SystemTimer. Just make sure you have SystemTimer installed, too. sudo gem install SystemTimer I believe.

Regarding gdb, the process shouldn't segfault, that's not what this is supposed to do. When it works, it will print the Ruby stack trace of whatever your process is stuck on.

I took that function from the e-book "Troubleshooting Ruby Processes" (which is worth checking out). The full instructions they give are:

gdb /usr/bin/ruby pid

OR

sudo gdb /usr/bin/ruby pid

[Note: are you using REE? Make sure you use the path to that instead.]

Then use the cal rb_raise thing.

Hope that helps.

[johnhampton]

Hi,

I don't see any references to SystemTimer in the current code on github. I'm having the same problem with the fetcher process hanging. Does installing the SystemTimer gem automatically enable better timeout support?

[look]

Well that's incredibly embarrassing. I made that change in another repository and never committed it to this one.

I just pushed the SystemTimer IMAP fix. Please try it and see if it solves your problem.