search

phusion / passenger

A fast and robust web server and application server for Ruby, Python and Node.js
https://www.phusionpassenger.com/
MIT License
4.99k stars 548 forks source link

Irregular ApplicationPool server crashes, Ruby EE, 1.8.6 20090201 #299

Closed FooBarWidget closed 10 years ago

FooBarWidget commented 10 years ago

From antifuchs on February 05, 2009 21:23:02

What steps will reproduce the problem? 0. Run debian stable, apache2 from there (aptitude install apache2 apache2-dev)

  1. Install Ruby enterprise edition version 1.8.6-20090201 (haven't tried older versions)
  2. Install REE's phusion passenger module via /opt/ruby/bin/passenger-install-apache2-module
  3. Run a web application with it (see config attached)
  4. Wait a few hours, observe log file entries. What is the expected output? What do you see instead? Apache and Phusion should keep running and serving requests. Instead, the ApplicationPool server quits with a segmentation fault.

If the ApplicationPool server must exit, the module should re-start it seamlessly. Instead, an /etc/init.d/apache2 restart is necessary to fix this condition.

Apache2's error.log says this:

[Wed Feb 04 22:07:40 2009] [error] [client 192.168.96.2] * Unexpected error in Passenger: The ApplicationPool server unexpectedly closed the connection., referer: http://www.soup.io/everyone [Wed Feb 04 22:07:40 2009] [notice] child pid 29557 exit signal Segmentation fault (11) [Wed Feb 04 22:07:40 2009] [warn] long lost child came home! (pid 29557) [Wed Feb 04 22:07:41 2009] [error] [client 192.168.96.2] * Unexpected error in Passenger: The ApplicationPool server exited unexpectedly. [Wed Feb 04 22:07:41 2009] [error] [client 192.168.96.2] * Unexpected error in Passenger: The ApplicationPool server exited unexpectedly., referer: http://(elided).soup.io/ [Wed Feb 04 22:07:41 2009] [error] [client 192.168.96.2] * Unexpected error in Passenger: The ApplicationPool server exited unexpectedly. [Wed Feb 04 22:07:42 2009] [error] [client 192.168.96.2] *\ Unexpected error in Passenger: The ApplicationPool server exited unexpectedly. (excerpt)

This happens irregularly (first time time was two hours after we had switched to Phusion Passenger )-:), sometimes after a few hours, sometimes more than a few days go by without a crash. What version of the product are you using? On what operating system? This is Ruby Enterprise Edition 1.8.6-20090201, its packaged phusion passenger (passenger-config --version reports 2.0.6). OS is Debian 4.0. Apache was installed by "aptitude install apache2 apache2-dev", which gets these packages:

ii apache2 2.2.3-4+etch6 Next generation, scalable, extendable web server ii apache2-mpm-worker 2.2.3-4+etch6 High speed threaded model for Apache HTTPD 2.1 ii apache2-threaded-dev 2.2.3-4+etch6 development headers for apache2 ii apache2-utils 2.2.3-4+etch6 utility programs for webservers ii apache2.2-common 2.2.3-4+etch6 Next generation, scalable, extendable web server

Configuration is the debian-provided stuff verbatim, with the exception of the two files pasted below. Please provide any additional information below. I load the phusion passenger module with this configuration (resides at /etc/apache2/mods-enabled/phusion.load):

LoadModule passenger_module /opt/ruby/lib/ruby/gems/1.8/gems/passenger-2.0.6/ext/apache2/mod_passenger.so PassengerRoot /opt/ruby/lib/ruby/gems/1.8/gems/passenger-2.0.6 PassengerRuby /opt/ruby/bin/ruby

PassengerDefaultUser app PassengerUseGlobalQueue on PassengerMaxPoolSize 12

The virtual host is set up like this (/etc/apache2/sites-enabled/000-default):

<VirtualHost *:80> ServerAdmin (elided email address) DocumentRoot /soup/current/public

Original issue: http://code.google.com/p/phusion-passenger/issues/detail?id=199

FooBarWidget commented 10 years ago

From antifuchs on February 05, 2009 12:27:37

For people suffering from similar crashes on debian, I have created a god config to monitor the web connectivity, and restart apache if it crashes. That takes away most of the pain (if you can survive having the web site offline for up to 10 seconds).

You may need to edit the .god file and replace "www.soup.io" with your web site's host name. On non-debian platforms, you may have to change the start/restart script names and pid file locations, as well.

Attachment: passenger.god

FooBarWidget commented 10 years ago

From honglilai on March 15, 2009 03:26:49

Does anybody still experience this problem with Phusion Passenger 2.1.2?

FooBarWidget commented 10 years ago

From antifuchs on March 16, 2009 04:06:34

I'll upgrade and test to passenger 2.1 next week. Will post my findings here.

FooBarWidget commented 10 years ago

From jnewland on March 16, 2009 18:56:54

I just experienced this issue today on 2.1.2 on CentOS i686. Here's the backtrace:

[Mon Mar 16 18:31:28 2009] errorCannot allocate memory: fork: Unable to fork new process [ pid=1605 file=Hooks.cpp:527 time=2009-03-16 18:31:28.675 ]: Unexpected error in mod_passenger: Could not send data to the ApplicationPool server: write() failed: Broken pipe (32) Backtrace: in 'virtual boost::shared_ptrPassenger::Application::Session Passenger::ApplicationPoolServer::Client::get(const Passenger::PoolOptions&)' (ApplicationPoolServer.h:386) in 'int Hooks::handleRequest(request_rec*)' (Hooks.cpp:414)

[ pid=1604 file=Hooks.cpp:527 time=2009-03-16 18:31:28.675 ]: Unexpected error in mod_passenger: Could not connect to the ApplicationPool server: Connection reset by peer (104) Backtrace: in 'Passenger::ApplicationPoolPtr Passenger::ApplicationPoolServer::connect()' (ApplicationPoolServer.h:730) in 'int Hooks::handleRequest(request_rec*)' (Hooks.cpp:414)

The 'Cannot allocate memory' line seems to indicate this is an OOM situation, however this is an app server with 1.25gb of memory running nothing other than Apache/2.2.3 and Passenger.

I'd be happy to do any debugging that may help further isolate this.

FooBarWidget commented 10 years ago

From honglilai on March 17, 2009 01:32:05

jnewland, could you check your VM size ulimit? And if that's not the problem, are you running on a VPS? Some VPSes are known to limit process VM sizes at the kernel level, ignoring any ulimit settings.

FooBarWidget commented 10 years ago

From jnewland on March 21, 2009 05:40:14

Sorry this took so long to get back to you:

ulimit -a

core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited file size (blocks, -f) unlimited pending signals (-i) 1024 max locked memory (kbytes, -l) 32 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 stack size (kbytes, -s) 10240 cpu time (seconds, -t) unlimited max user processes (-u) 153599 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited

This is a VPS running under Parallels Virtuozzo.

I experienced this crash once again this week. Thoughts?

FooBarWidget commented 10 years ago

From honglilai on March 24, 2009 03:19:05

Could you try obtaining a gdb backtrace of when ApplicationPoolServerExecutable crashes? You can do it like this:

  1. Run Apache.
  2. Type: gdb /path-to-passenger/ext/apache2/ApplicationPoolServerExecutable pid-of-ApplicationPoolServerExecutable-here
  3. Do whatever you can to make ApplicationPoolServerExecutable crash as usual.
  4. When it crashes, go back to gdb, type 'thread apply all bt full' and paste the output.
FooBarWidget commented 10 years ago

From jnewland on March 24, 2009 04:10:58

Just experienced the crash again. After restarting apache, I opened gdb in a screen session hooked to the ApplicationPoolServerExecutable. This hasn't been easily reproducible, so I'll just wait until it crashes again and then get the backtrace. Thanks for your continued help. Hopefully this backtrace will help you get to the bottom of this!

FooBarWidget commented 10 years ago

From jnewland on March 24, 2009 04:21:18

Well, actually, that didn't work. I just got a notification that the site was down again, and when I went back to the gdb prompt, I saw this:

... (gdb) c Continuing. Could not open /proc/12123/status

I then Quit gdb and restarted apache. Do you have another suggestion as to how we might be able to get a backtrace? Am I doing something wrong with gdb?

FooBarWidget commented 10 years ago

From honglilai on March 24, 2009 04:35:09

I forgot to mention that you need to type 'c' right after attaching gdb to ApplicationPoolServerExecutable. Also, you may need to run gdb as root.

If that doesn't work either please paste the relevant gdb output and I'll take a look.

FooBarWidget commented 10 years ago

From jnewland on March 24, 2009 04:40:14

That's what I did:

[root@app ~]# gdb /opt/ruby-enterprise-1.8.6-20081215/lib/ruby/gems/1.8/gems/passenger- 2.1.2/ext/apache2/ApplicationPoolServerExecutable 15570 GNU gdb Red Hat Linux (6.5-37.el5_2.2rh) Copyright (C) 2006 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-redhat-linux-gnu"...Using host libthread_db library "/lib/libthread_db.so.1".

Attaching to program: /opt/ruby-enterprise-1.8.6-20081215/lib/ruby/gems/1.8/gems/passenger- 2.1.2/ext/apache2/ApplicationPoolServerExecutable, process 15570 Reading symbols from /lib/libpthread.so.0...done. [Thread debugging using libthread_db enabled] [New Thread -1208998192 (LWP 15570)] [New Thread -1215468656 (LWP 4051)] [New Thread -1218712688 (LWP 4050)] [New Thread -1219118192 (LWP 3841)] [New Thread -1218983024 (LWP 3840)] [New Thread -1218577520 (LWP 3837)] [New Thread -1218307184 (LWP 3835)] [New Thread -1217631344 (LWP 3830)] [New Thread -1217090672 (LWP 3826)] [New Thread -1216685168 (LWP 3823)] [New Thread -1215063152 (LWP 3811)] [New Thread -1214927984 (LWP 3808)] [New Thread -1214657648 (LWP 3806)] [New Thread -1213056112 (LWP 3800)] [New Thread -1212920944 (LWP 3799)] [New Thread -1209406576 (LWP 1548)] [New Thread -1211974768 (LWP 21680)] [New Thread -1211839600 (LWP 21574)] [New Thread -1211028592 (LWP 17903)] [New Thread -1210893424 (LWP 17874)] [New Thread -1210623088 (LWP 15959)] [New Thread -1209136240 (LWP 15573)] [New Thread -1209001072 (LWP 15572)] Loaded symbols for /lib/libpthread.so.0 Reading symbols from /usr/lib/libstdc++.so.6...done. Loaded symbols for /usr/lib/libstdc++.so.6 Reading symbols from /lib/libm.so.6...done. Loaded symbols for /lib/libm.so.6 Reading symbols from /lib/libgcc_s.so.1...done. Loaded symbols for /lib/libgcc_s.so.1 Reading symbols from /lib/libc.so.6...done. Loaded symbols for /lib/libc.so.6 Reading symbols from /lib/ld-linux.so.2...done. Loaded symbols for /lib/ld-linux.so.2 0x007027f2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 (gdb) c Continuing.

I received another notification that the site was, down and the output had this amended to it:

Could not open /proc/12123/status (gdb)

Is this normal? Is that the point at which I should have run the 'thread apply all bt full' command? The failure behavior of apache was slightly different at this point - it was hanging, not returning a 500.

FooBarWidget commented 10 years ago

From honglilai on March 24, 2009 05:11:35

The "could not open /proc/12123/status" message is not normal, but I don't know what's causing it. You can try typing 'thread apply all bt full' anyway and hope that it works.

Which version of RedHat are you running, with which kernel version? Is your procfs properly mounted?

FooBarWidget commented 10 years ago

From jnewland on March 24, 2009 11:52:50

[jnewland@app ~]$ cat /etc/redhat-release CentOS release 5.2 (Final) [jnewland@app ~]$ uname -a Linux app.corkd.com 2.6.9-023stab048.6-enterprise #1 SMP Mon Nov 17 19:09:18 MSK 2008 i686 i686 i386 GNU/Linux [jnewland@app ~]$ cat /proc/stat cpu 9078955 0 1561896 698698646 32847 0 0 cpu0 1917309 0 625908 174787657 9125 0 0 cpu1 2399897 0 386538 174542918 9899 0 0 cpu2 2335701 0 278069 174727071 7105 0 0 cpu3 2426047 0 271379 174640999 6717 0 0 intr 0 swap 0 0 ctxt 2442717107 btime 1236145630 processes 11124832 procs_running 1 procs_blocked 0

I'll attach gdb again in a couple hours and see if the same behavior happens.

FooBarWidget commented 10 years ago

From matt.manning on April 20, 2009 09:36:13

Did you guys ever resolve the issue? I'm having the exact same problem on CentOS running passenger 2.2.1.

FooBarWidget commented 10 years ago

From honglilai on April 20, 2009 10:07:11

Unfortunately not. The cause of this problem is still unknown, and we can't reproduce it on any of our machines.

FooBarWidget commented 10 years ago

From matt.manning on April 20, 2009 10:39:29

That's too bad. We're reproducing it like crazy :). I have a partial gdb backtrace that I might be able to post. Would that be of interest to anyone here?

FooBarWidget commented 10 years ago

From matt.manning on April 20, 2009 10:44:43

Here's the backtrace from the last crash. If anyone as any input at all it would be greatly appreciated.

Attachment: u2_crash_bt.txt

FooBarWidget commented 10 years ago

From jnewland on April 20, 2009 10:55:14

I'm continuing to experience this bug, but haven't been around to trigger a backtrace the past couple of times it's happened. Thanks for posting yours, Matt!

FooBarWidget commented 10 years ago

From honglilai on April 20, 2009 12:00:50

matt.manning: Unfortunately I didn't get any wiser after inspecting your backtrace. How often do you experience this problem?

If possible I'd like to have access to a staging server on which this problem can be easily reproduced. Do you happen to have one?

FooBarWidget commented 10 years ago

From jnewland on April 20, 2009 12:41:39

honglilai - I'm happy to give you access to the staging environment for the site which is experiencing this issue. Please send your SSH public key to jnewland (at) gmail dot com and I'll get you setup.

FooBarWidget commented 10 years ago

From will.pink on May 13, 2009 17:11:21

Did this issue ever get looked at? It appears we are experiencing the same with Ubuntu server 8.10

[ pid=26431 file=ext/apache2/Hooks.cpp:566 time=2009-05-14 01:00:06.55 ]: Unexpected error in mod_passenger: The ApplicationPool server unexpectedly closed the connection while we're reading a response for the 'get' command. Backtrace: (empty)

FooBarWidget commented 10 years ago

From honglilai on May 14, 2009 00:11:54

Yes but there isn't much I can do about it unless I can reproduce it. Unfortunately I was not able to reproduce the problem on jnewland's server either.

FooBarWidget commented 10 years ago

From therealdave.myron on May 17, 2009 17:28:01

Having the same issue on Ubuntu 8.10 on a c1.medium EC2 instance with passenger 2.2.2. RailsMaxPoolSize is set to 16. Ran into the issue during load testing.

Sorry I don't have a gdb backtrace.

Consider this a vote for more eyes on this issue.

FooBarWidget commented 10 years ago

From therealdave.myron on May 18, 2009 14:18:31

Here's the message we have in our apache2 error logs. Restarting apache fixes things, of course...

[ pid=27055 file=ext/apache2/Hooks.cpp:566 time=2009-05-18 21:14:47.991 ]: Unexpected error in mod_passenger: Could not connect to the ApplicationPool server: Broken pipe (32) Backtrace: (empty)

FooBarWidget commented 10 years ago

From therealdave.myron on May 18, 2009 14:19:34

And we're getting this on a c1.xlarge EC2 instance (7gb of memory) with PassengerMaxPoolSize of 20. Seems pretty stable at 8.

FooBarWidget commented 10 years ago

From therealdave.myron on May 18, 2009 14:38:06

Ruby version: ruby 1.8.7 (2008-08-11 patchlevel 72) [x86_64-linux]

FooBarWidget commented 10 years ago

From therealdave.myron on May 18, 2009 14:55:28

Don't know if this helps at all, but this appears to start the brokenness (in our logs):

* Exception Errno::EPIPE in Passenger RequestHandler (Broken pipe) (process 12771): from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/rack/request_handler.rb:83:in write' from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/rack/request_handler.rb:83:in process_request' from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/abstract_request_handler.rb:203:in main_loop' from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/railz/application_spawner.rb:340:in start_request_handler' from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/railz/application_spawner.rb:298:in handle_spawn_application' from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/utils.rb:181:in safe_fork' from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/railz/application_spawner.rb:296:in handle_spawn_application' from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/abstract_server.rb:337:in send**' from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/abstract_server.rb:337:in main_loop' from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/abstract_server.rb:187:in start_synchronously' from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/abstract_server.rb:154:in start' from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/railz/application_spawner.rb:192:in start' from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/spawn_manager.rb:257:in spawn_rails_application' from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/abstract_server_collection.rb:126:in lookup_or_add' from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/spawn_manager.rb:251:in spawn_rails_application' from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/abstract_server_collection.rb:80:in synchronize' from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/abstract_server_collection.rb:79:in synchronize' from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/spawn_manager.rb:250:in spawn_rails_application' from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/spawn_manager.rb:153:in spawn_application' from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/spawn_manager.rb:282:in handle_spawn_application' from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/abstract_server.rb:337:in __send__' from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/abstract_server.rb:337:in main_loop' from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/abstract_server.rb:187:in `start_synchronously' from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/bin/passenger-spawn-server:61 [ pid=18415 file=ext/apache2/Hooks.cpp:566 time=2009-05-18 20:42:47.24 ]: Unexpected error in mod_passenger: write() failed: Broken pipe (32) Backtrace: (empty) [ pid=17057 file=ext/apache2/Hooks.cpp:566 time=2009-05-18 20:42:47.31 ]: Unexpected error in mod_passenger: Could not connect to the ApplicationPool server: Broken pipe (32) Backtrace: (empty) [ pid=16655 file=ext/apache2/Hooks.cpp:566 time=2009-05-18 20:42:47.36 ]: Unexpected error in mod_passenger: write() failed: Broken pipe (32) Backtrace: (empty) [ pid=18276 file=ext/apache2/Hooks.cpp:566 time=2009-05-18 20:42:47.36 ]: Unexpected error in mod_passenger: Could not connect to the ApplicationPool server: Broken pipe (32) Backtrace: (empty) [ pid=18415 file=ext/apache2/Hooks.cpp:566 time=2009-05-18 20:42:47.37 ]: Unexpected error in mod_passenger: write() failed: Broken pipe (32) Backtrace: (empty) [ pid=17998 file=ext/apache2/Hooks.cpp:566 time=2009-05-18 20:42:47.38 ]: Unexpected error in mod_passenger: write() failed: Broken pipe (32) Backtrace: (empty) [ pid=16786 file=ext/apache2/Hooks.cpp:566 time=2009-05-18 20:42:47.38 ]: Unexpected error in mod_passenger: write() failed: Broken pipe (32) Backtrace: (empty) [ pid=18415 file=ext/apache2/Hooks.cpp:566 time=2009-05-18 20:42:47.40 ]: Unexpected error in mod_passenger: write() failed: Broken pipe (32)

FooBarWidget commented 10 years ago

From honglilai on May 19, 2009 16:03:47

Issue 248 has been merged into this issue.

FooBarWidget commented 10 years ago

From honglilai on May 19, 2009 16:04:19

I've made some changes in the 'experimental' branch on Github which might help. Could you try that?

FooBarWidget commented 10 years ago

From therealdave.myron on May 19, 2009 16:50:19

I'll need to spin up a new instance and then run ab against it. Will let you know the results. What were the changes in regards to?

FooBarWidget commented 10 years ago

From honglilai on May 20, 2009 02:06:05

I modified the application pool server to use a filesystem-backed Unix socket instead of an anonymous Unix socket pair. It's unlikely but might have something to do with the crashes that you see.

FooBarWidget commented 10 years ago

From honglilai on May 22, 2009 02:34:20

Did the changes help?

FooBarWidget commented 10 years ago

From therealdave.myron on May 22, 2009 13:06:23

Hey Hongli, I haven't had a chance to test it out - been busy firefighting from our huge press coverage (for Glympse - glympse.com). It'll be next week.

FooBarWidget commented 10 years ago

From roman@snitko.ru on June 04, 2009 13:19:26

Passenger 2.2.2 on Ubuntu 9.04 + apache2 + RubyEnterprise, same bug.

I was thinking, could it have something to do with RubyEnterprise? Because running it on almost identical machine with regular Ruby 1.8.7 for a long time did not reveal any problems.

FooBarWidget commented 10 years ago

From therealdave.myron on June 04, 2009 13:20:52

I'm running on regular Ruby 1.8.7, not REE, so probably not.

FooBarWidget commented 10 years ago

From roman@snitko.ru on June 04, 2009 13:30:37

Anything similar happens on nginx? I'm thinking about moving to it.

FooBarWidget commented 10 years ago

From honglilai on June 04, 2009 13:48:58

Nobody has reported about NginxHelperServer crashing yet.

FooBarWidget commented 10 years ago

From hayafirst on June 08, 2009 10:09:44

I am running nginx along with Passenger 2.2.2. I have the same problem

FooBarWidget commented 10 years ago

From justindossey on June 09, 2009 10:21:55

I was having this every 15 minutes on a busy Apache server. Disabling mod_xsendfile helped-- now it only happens every few hours. This is a pretty major problem, though.

FooBarWidget commented 10 years ago

From jason.garber on June 10, 2009 02:20:43

Just this week it's been doing this to me daily. I use mod_xsendfile too. Passenger 2.2.2, ruby 1.8.6 (2007- 09-24 patchlevel 111) [x86_64-linux] on Ubuntu.

My appache error log always looks like this when it crashes:

[Wed Jun 10 03:47:52 2009] [error] cgid daemon process died, restarting [Wed Jun 10 03:47:52 2009] [warn] long lost child came home! (pid 14915) [ pid=14920 file=ext/apache2/Hooks.cpp:566 time=2009-06-10 03:47:53.401 ]: Unexpected error in mod_passenger: Could not send data to the ApplicationPool server: write() failed: Broken pipe (32) Backtrace: (empty) [ pid=23783 file=ext/apache2/Hooks.cpp:566 time=2009-06-10 03:48:29.313 ]: Unexpected error in mod_passenger: Could not connect to the ApplicationPool server: Broken pipe (32) Backtrace: (empty) [ pid=14920 file=ext/apache2/Hooks.cpp:566 time=2009-06-10 03:48:41.214 ]: Unexpected error in mod_passenger: Could not send data to the ApplicationPool server: write() failed: Broken pipe (32) Backtrace: (empty) [ pid=23783 file=ext/apache2/Hooks.cpp:566 time=2009-06-10 03:48:59.799 ]: Unexpected error in mod_passenger: Could not connect to the ApplicationPool server: Broken pipe (32) Backtrace: (empty) [ pid=14920 file=ext/apache2/Hooks.cpp:566 time=2009-06-10 03:50:12.640 ]: Unexpected error in mod_passenger: Could not send data to the ApplicationPool server: write() failed: Broken pipe (32) Backtrace: (empty) ...

FooBarWidget commented 10 years ago

From atimetunnel on June 10, 2009 10:33:41

I had also seen this issue using mod_passenger and apache2. I had noticed that there were stuck rails/passenger processes running on the box. I wasn't aware of these at first since I figured apache was supposed to be taking care of killing these off. Turns out I needed to kill them off after I stopped apache and all was fine. I had modified my init script to deal with making sure these were dead upon stopping/restarting apache. Can someone on here having these problems check into that to see if what I found jives?

FooBarWidget commented 10 years ago

From jason.garber on June 11, 2009 05:50:10

Maybe I don't understand how to check for stuck processes, but I'm not noticing any. My monitoring service alerts me there's a problem with all my websites (they all go down together), I log in, restart apache, and everything's fine again. I check the error log and I always see what's pasted above.

FooBarWidget commented 10 years ago

From hayafirst on June 11, 2009 06:37:32

I finally get some backtrace:

[ pid=382 file=ext/nginx/HelperServer.cpp:460 time=2009-06-09 16:12:16.588 ]:
  Uncaught exception in PassengerServer client thread:
   exception: write() failed: Broken pipe (32)
   backtrace:
     in 'void
Client::forwardResponse(boost::shared_ptr&,
FileDescriptor&)' (HelperServer.cpp:344)
     in 'bool Client::handleRequest(FileDescriptor&)' (HelperServer.cpp:438)
     in 'void Client::threadMain()' (HelperServer.cpp:484)
cacerts: /usr/local/lib/ruby/gems/1.8/gems/httpclient-2.1.5/lib/httpclient/cacert.p7s
loading failed
cacerts: /usr/local/lib/ruby/gems/1.8/gems/httpclient-2.1.5/lib/httpclient/cacert.p7s
loading failed
[ pid=382 file=ext/nginx/HelperServer.cpp:460 time=2009-06-09 16:12:17.13 ]:
  Uncaught exception in PassengerServer client thread:
   exception: write() failed: Broken pipe (32)
   backtrace:
     in 'void
Client::forwardResponse(boost::shared_ptr&,
FileDescriptor&)' (HelperServer.cpp:344)
     in 'bool Client::handleRequest(FileDescriptor&)' (HelperServer.cpp:438)
     in 'void Client::threadMain()' (HelperServer.cpp:484)
[ pid=382 file=ext/nginx/HelperServer.cpp:460 time=2009-06-09 16:12:17.244 ]:
  Uncaught exception in PassengerServer client thread:
   exception: write() failed: Broken pipe (32)
   backtrace:
     in 'void
Client::forwardResponse(boost::shared_ptr&,
FileDescriptor&)' (HelperServer.cpp:344)
     in 'bool Client::handleRequest(FileDescriptor&)' (HelperServer.cpp:438)
     in 'void Client::threadMain()' (HelperServer.cpp:484)
FooBarWidget commented 10 years ago

From hayafirst on June 11, 2009 07:27:31

here's more. sorry for the spam. But it doesn;t seem I can edit my own comment so I have to post a new one

[ pid=477 file=ext/nginx/HelperServer.cpp:460 time=2009-06-11 08:38:16.782 ]: Uncaught exception in PassengerServer client thread: exception: write() failed: Broken pipe (32) backtrace: in 'void Client::forwardResponse(boost::shared_ptrPassenger::Application::Session&, FileDescriptor&)' (HelperServer.cpp:344) in 'bool Client::handleRequest(FileDescriptor&)' (HelperServer.cpp:438) in 'void Client::threadMain()' (HelperServer.cpp:484)

[ pid=477 file=ext/nginx/HelperServer.cpp:460 time=2009-06-11 08:38:17.182 ]: Uncaught exception in PassengerServer client thread: exception: write() failed: Broken pipe (32) backtrace: in 'void Client::forwardResponse(boost::shared_ptrPassenger::Application::Session&, FileDescriptor&)' (HelperServer.cpp:344) in 'bool Client::handleRequest(FileDescriptor&)' (HelperServer.cpp:438) in 'void Client::threadMain()' (HelperServer.cpp:484)

* Exception Errno::EPIPE in Passenger RequestHandler (Broken pipe) (process 1075): from /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/rack/request_handler.rb:98:in write' from /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/rack/request_handler.rb:98:in process_request' from /Users/ywen/projects/next_gen/wos_app/vendor/rails/actionpack/lib/action_controller/response.rb:155:in each_line' from /Users/ywen/projects/next_gen/wos_app/vendor/rails/actionpack/lib/action_controller/response.rb:155:in each' from /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/rack/request_handler.rb:97:in process_request' from /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/abstract_request_handler.rb:203:in main_loop' from /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/railz/application_spawner.rb:340:in start_request_handler' from /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/railz/application_spawner.rb:298:in handle_spawn_application' from /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/utils.rb:181:in safe_fork' from /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/railz/application_spawner.rb:296:in handle_spawn_application' from /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/abstract_server.rb:337:in __send__' from /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/abstract_server.rb:337:in main_loop' from /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/abstract_server.rb:187:in start_synchronously' from /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/abstract_server.rb:154:in start' from /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/railz/application_spawner.rb:192:in start' from /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/spawn_manager.rb:257:in spawn_rails_application' from /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/abstract_server_collection.rb:126:in lookup_or_add' from /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/spawn_manager.rb:251:in spawn_rails_application' from /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/abstract_server_collection.rb:80:in synchronize' from /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/abstract_server_collection.rb:79:in synchronize' from /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/spawn_manager.rb:250:in spawn_rails_application' from /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/spawn_manager.rb:153:in spawn_application' from /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/spawn_manager.rb:282:in handle_spawn_application' from /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/abstract_server.rb:337:in send**' from /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/abstract_server.rb:337:in main_loop' from /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/abstract_server.rb:187:in start_synchronously' from /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.2/bin/passenger-spawn-server:61 [ pid=477 file=ext/nginx/HelperServer.cpp:460 time=2009-06-11 08:38:17.306 ]: Uncaught exception in PassengerServer client thread: exception: write() failed: Broken pipe (32) backtrace: in 'void Client::forwardResponse(boost::shared_ptrPassenger::Application::Session&, FileDescriptor&)' (HelperServer.cpp:344) in 'bool Client::handleRequest(FileDescriptor&)' (HelperServer.cpp:438) in 'void Client::threadMain()' (HelperServer.cpp:484)

FooBarWidget commented 10 years ago

From jeffrafter on June 18, 2009 18:36:39

Hi all, I am having this issue very repeatably. No requests at all are being served. passenger version 2.2.2 2.0.3 Am willing to give honglilai access to the box. I checked for stuck processes and there were some, but killing them off and restarting had no effect.

FooBarWidget commented 10 years ago

From jeffrafter on June 18, 2009 19:55:45

Okay, I just loaded 2.2.3 (sudo gem install passenger) then reinstalled (sudo passenger-install-apache2-module) then grabbed the latest config info and put it into passenger.conf. Restarted and now it is working perfectly again.

FooBarWidget commented 10 years ago

From justindossey on July 06, 2009 15:08:48

I'm still seeing this in 2.2.4.

FooBarWidget commented 10 years ago

From millisami on July 07, 2009 11:01:27

Me too seeing the same error with 2.2.4!!

FooBarWidget commented 10 years ago

From rqbanerjee on July 08, 2009 21:52:53

One more failure on 2.2.4 for me.

FooBarWidget commented 10 years ago

From zhangyu34 on July 12, 2009 18:24:26

I have such a problem too. What I get is: \ Exception Errno::ENOMEM in Passenger::Railz::ApplicationSpawner (Cannot allocate memory - fork(2)) (process 9575): from /usr/local/ruby/lib/ruby/gems/1.8/gems/passenger-2.0.6/lib/passenger/utils.rb:161:in fork' from /usr/local/ruby/lib/ruby/gems/1.8/gems/passenger-2.0.6/lib/passenger/utils.rb:161:in safe_fork' from /usr/local/ruby/lib/ruby/gems/1.8/gems/passenger-2.0.6/lib/passenger/railz/application_spawner.rb:280:in `handle_spawn_application'

It seems there is no memory. However, the machine has 4GB ram and the load level is not high. After I restart apache, it's ok. Another machine with almost the same configures doesn't have this problem. The only differnces are:

  1. I define 'PassengerPoolIdleTime 1200' in the correct one. The error one remains default.
  2. All the requests on static resources are responed by nginx instead of apache(the correct one).

After I restart the apache on Jul11, I find the 'Passenger spawn server' never exit. The passenger-memory status now is 26548 1 289.0 MB ? Passenger spawn server 6285 1 289.6 MB ? Passenger FrameworkSpawner: 2.1.2 18303 1 297.9 MB ? Rails: /opt/deploy/msg/releases/20090519074121 29084 1 291.7 MB ? Passenger ApplicationSpawner: /opt/deploy/msg160/releases/20090519074121 29086 1 296.1 MB ? Rails: /opt/deploy/msg/releases/20090519074121

The process status is: root 26548 26547 0 Jul11 ? 00:00:15 Passenger spawn server

root 6285 26548 0 06:24 ? 00:00:02 Passenger FrameworkSpawner: 2.1.2

deploy 29084 6285 0 08:37 ? 00:00:02 Passenger ApplicationSpawner: /opt/deploy/msg/releases/20090519074121

As I mentioned, the spawn server doesn't exit until now and it's memory is pretty high.