zombie killer not killing passenger processes

[taraspos]

I am running zombie_passenger_killer with options

sudo zombie_passenger_killer --history 10 --timing --cpu 50 --interval 30 --pattern Rails

but it isn't working, i got 3 processes hanging with 60% CPU all night

[root@app02 log]# ps aux | grep zombie
root     20953  0.0  0.1  18232  6100 pts/0    S<   00:34   0:00 zombie_passenger_killer-0.3.0                                                                                           
root     21063  0.0  0.0  61192   744 pts/0    R+   00:37   0:00 grep zombie
[root@app02 log]# passenger-status
----------- General information -----------
max      = 6
count    = 6
active   = 3
inactive = 3
Waiting on global queue: 0

----------- Application groups -----------
/usr/local/apache2/webapps/current:
  App root: /usr/local/apache2/webapps/current
  * PID: 8096    Sessions: 0    Processed: 18201   Uptime: 11h 1m 10s
  * PID: 8461    Sessions: 0    Processed: 16278   Uptime: 10h 42m 33s
  * PID: 6774    Sessions: 1    Processed: 133     Uptime: 12h 10m 1s
  * PID: 8269    Sessions: 1    Processed: 41      Uptime: 10h 52m 16s
  * PID: 7290    Sessions: 1    Processed: 1892    Uptime: 11h 42m 35s

/usr/local/apache2/webapp2:
  App root: /usr/local/apache2/webapp2
  * PID: 8185    Sessions: 0    Processed: 183     Uptime: 10h 56m 57s

[root@app02 log]# ps aux | grep Rails
apache    6774 62.1  2.6 205008 99012 ?        R    Dec01 471:03 Rails: /usr/local/apache2/webapps/current                                                                                                                     
apache    7290 54.4  2.6 205028 99044 ?        R    Dec01 396:56 Rails: /usr/local/apache2/webapps/current                                                                                                                     
apache    8096  0.7  2.6 205480 99624 ?        S    Dec01   5:09 Rails: /usr/local/apache2/webapps/current                                                                                                                     
apache    8185  0.1  1.5 146840 57132 ?        S    Dec01   0:47 Rails: /usr/local/apache2/webapp2                                                                                                                               
apache    8269 58.3  2.5 204596 98600 ?        R    Dec01 395:02 Rails: /usr/local/apache2/webapps/current                                                                                                                     
apache    8461  0.6  2.6 207608 101748 ?       S    Dec01   4:33 Rails: /usr/local/apache2/webapps/current                                                                                                                     
root     21083  0.0  0.0  61192   748 pts/0    R+   00:37   0:00 grep Rails

atop shows me

 6774          -       apache        apache            1        0.06s         6.81s          0K           0K          0K           0K      --         -      R           0       71%       ruby
 7290          -       apache        apache            1        0.06s         5.52s          0K           0K          0K           0K      --         -      R           1       58%       ruby
 8269          -       apache        apache            1        0.10s         5.25s          0K           0K          0K           0K      --         -      R           1       56%       ruby

what is the possible issue?? p.s. i am using Phusion Passenger version 3.0.2, ruby 1.8.6 (2008-08-11 patchlevel 287) [x86_64-linux] and Rails 2.3.5

[grosser]

Your options look right, not sure why it did not work, maybe they were not over 60% consistently, adding a few puts for debugging (ideally a --verbose mode) could help tracking it down.

[taraspos]

they were like this for whole night, and eat like 95% of my 2 vCPU. I also wrote daemon output to file, for some logging, and there is nothing... only line that tells me that daemon started and that it. zombie_passenger_killer --verbose and i am getting 'Invalid option' error

[grosser]

you have to actually open the gem and add the option or add the puts, then run it via ./bin/zombie_passenger_killer

On Thu, Dec 4, 2014 at 12:36 AM, Taras notifications@github.com wrote:

they were like this for whole night, and eat like 95% of my 2 vCPU. I also wrote daemon output to file, for some logging, and there is nothing... only line that tells me that daemon started and that it. zombie_passenger_killer --verbose and i am getting 'Invalid option' error

— Reply to this email directly or view it on GitHub https://github.com/grosser/zombie_passenger_killer/issues/8#issuecomment-65552380 .

[taraspos]

Line below return nil value: pids = %x(#{'rvmsudo ' if @rvmsudo}passenger-status|grep PID).split("\n").map { |x| x.strip.match(/PID: \d*/).to_s.split[1].to_i } Do you have some thoughts what can be wrong with that parsing line? :( I thought, may be script can't reach passenger-status command, but system 'passenger-status' works well.

[grosser]

... passenger-status does not include PID -> empty result looks like we need a new way to get pids, maybe take second item in each line split(/\s+/)[1] or so

[taraspos]

i am not sure, if i understood it right(i dont really know ruby), but i tried

 some = %x(passenger-status|grep PID)
 print @some

and got nill too

[grosser]

added a --verbose mode, worked fine for me locally

just do passenger-status|grep PID and see what you get locally, otherwise run with --verbose --once to see what it is seeing / trying to do

[taraspos]

Thank you for --verbose mode i'm pretty sure it will help us to locate problem. In the new version of zombie_passenger_killer i got error

/usr/local/lib/ruby/gems/1.8/gems/zombie_passenger_killer-0.3.1/lib/zombie_passenger_killer/reaper.rb:76:in `passenger_pids': wrong argument type Symbol (expected Proc) (TypeError)

(its probably because you used some features from new Ruby version, i'm running ruby 1.8.6 but this not the point) so i tried to replace line

pids = %x(#{'rvmsudo ' if @rvmsudo}passenger-status).split("\n").map { |l| l[/PID: (\d+)/, 1] }.compact.map(&:to_i)

with the old one

pids = %x(#{'rvmsudo ' if @rvmsudo}passenger-status|grep PID).split("\n").map { |x| x.strip.match(/PID: \d*/).to_s.split[1].to_i }

and run --verbose mode. The output was

[root@app02 ~]# zombie_passenger_killer --verbose --pattern Rails
PID: 10580 Version: 0.3.1 Started at Fri Dec 05 00:58:03 -0800 2014
Active pids in status: [5771, 6812, 10337]
Active pids in processlist: [{:cpu=>0.4, :pid=>5771}, {:cpu=>0.2, :pid=>6812}, {:cpu=>1.9, :pid=>10337}]
Zombies: []

so everything looks fine, lets wait until new CPU overload, then i will write update here.

[grosser]

Active pids in status: nil does not make any sense, is that with or without your modifications ?

I'm getting: Active pids in status: [] Active pids in processlist: [] Zombies: []

On Fri, Dec 5, 2014 at 2:33 AM, Taras notifications@github.com wrote:

But on my local environment i got:

[root@localhost bin]# passenger-status ----------- General information ----------- max = 6 count = 1 active = 0 inactive = 1 Waiting on global queue: 0

----------- Application groups ----------- /usr/local/apache2/timebridge/webapps/current: App root: /usr/local/apache2/timebridge/webapps/current

• PID: 9890 Sessions: 0 Processed: 2 Uptime: 13m 11s

[root@localhost bin]# ./zombie_passenger_killer --verbose --pattern Rails PID: 11724 Version: 0.3.1 Started at Fri Dec 05 10:32:19 +0000 2014 Active pids in status: nil Active pids in processlist: [{:cpu=>0.0, :pid=>9890}] Zombies: [] Active pids in status: nil Active pids in processlist: [{:cpu=>0.0, :pid=>9890}] Zombies: []

— Reply to this email directly or view it on GitHub https://github.com/grosser/zombie_passenger_killer/issues/8#issuecomment-65772387 .

[taraspos]

Dont mind on that comment, i dont know what was wrong but its looks ok now.

[taraspos]

Can you test(and may be improve) my modification please? I'm not good with ruby, but i think i did it ok. I need that option, becouse like once or twice per week daemon fails because of some passenger-status bug(I'm forced to use old passenger, there is alot of bugs), I'm pretty sure, that it just need couple chances to call passenger-status before exit.

[grosser]

you should have something monitoring it -> gets restarted when it fails

[grosser]

I'm unable to install ruby 1.8.6, so if you have a patch, please make a PR

[urielhdz]

It doesn't seem to work for me either Top output:

424 nobody    20   0  594m 146m 5784 S 97.7  7.3  84:09.08 ruby                
13934 root      20   0  126m 4320 3120 S  2.0  0.2   0:00.06 sshd                
28859 root      20   0 1009m 7048 3124 S  0.7  0.3  15:45.30 PassengerHelper     
17408 mysql     20   0 1248m 178m 2712 S  0.3  8.9 691:48.50 mysqld     

Passenger-status output:

* PID: 424     Sessions: 1       Processed: 822     Uptime: 2h 24m 54s
    CPU: 58%     Memory  : 141M    Last used: 1h 25m 1
  * PID: 7581    Sessions: 0       Processed: 721     Uptime: 1h 9m 51s
    CPU: 1%      Memory  : 199M    Last used: 11s ago
  * PID: 12726   Sessions: 0       Processed: 273     Uptime: 10m 43s
    CPU: 3%      Memory  : 140M    Last used: 1s ago

I've tried running zombie_passenger_killer like this:

zombie_passenger_killer -c 50 &

Also like this:

zombie_passenger_killer -c 50 -1

Neither of them killed the process that was consuming 50%+ of my cpu and have been there for a couple of hours.

My CPU has been going to 100% from time to time so I was expecting this gem to help me.

Am I doing anything wrong?

Thanks!

[grosser]

your top does not show any passenger processes, try ps -ef | grep Ruby or rack or rails or so to find them and then use this as pattern, running with --verbose shows you the output zombie_passenger_killer finds

[urielhdz]

Aren't process executed by the nobody user from Passenger? I'll see what's the output with verbose once I see a "zombie" . Could you elaborate a little on what should I search for when running ps -ef | grep Ruby? I get a different process for each Ruby / Rails / Rack.

Thanks for your help

[grosser]

just any pattern that makes grep get the passenger processes, you can run it now and should still see some output like the processes that are currently running

On Mon, Mar 9, 2015 at 9:53 AM, Uriel Hernández notifications@github.com wrote:

Aren't process executed by the nobody user from Passenger? I'll see what's the output with verbose once I see a "zombie" . Could you elaborate a little on what should I search for when running ps -ef | grep Ruby? I get a different process for each Ruby / Rails / Rack.

Thanks for your help

— Reply to this email directly or view it on GitHub https://github.com/grosser/zombie_passenger_killer/issues/8#issuecomment-77894068 .