search

phusion / passenger

A fast and robust web server and application server for Ruby, Python and Node.js
https://www.phusionpassenger.com/
MIT License
5k stars 547 forks source link

Phusion Passenger + Ruby 1.9.3: forking results in closing internal Ruby file descriptors #813

Closed FooBarWidget closed 10 years ago

FooBarWidget commented 10 years ago

From garett.shulman on November 01, 2011 05:08:14

What steps will reproduce the problem? 1. install ruby 1.9.3

  1. gem install passenger
  2. build apache plugin or run standalone What is the expected output? What do you see instead? Expect to see app running. In standalone see nothing. In apache see PhusionPassenger::Rack::ApplicationSpawner::Error In logs see [ASYNC BUG] thread_timer: select What version of Phusion Passenger are you using? Which version of Rails? On what operating system? Passenger 3.0.9 Ruby 1.9.3-RC1 Rails 3.1.1 Ubuntu 10.04.2 LTS Please provide any additional information below. Thank you!

Original issue: http://code.google.com/p/phusion-passenger/issues/detail?id=714

FooBarWidget commented 10 years ago

From alexey.zatsepin@verdacom.com on November 06, 2011 05:49:39

same here but with ruby 1.9.3p0 Ubuntu 10.04 Passenger 3.0.9 apache 2

FooBarWidget commented 10 years ago

From jistr@jistr.com on November 07, 2011 07:43:17

Same here.

Ruby 1.9.3-p0 Passenger 3.0.9 Nginx 1.0.6 Rails 3.1.1 Ubuntu 10.04 LTS

FooBarWidget commented 10 years ago

From honglilai on November 07, 2011 08:36:23

I tested on both OS X Snow Leopard and Ubuntu 10.04 but I could reproduce the problem on neither systems. I need some feedback from the people who are experiencing this problem:

Summary: Phusion Passenger + Ruby 1.9.3: forking results in closing internal Ruby file descriptors

FooBarWidget commented 10 years ago

From gr2020 on November 07, 2011 08:37:18

Same problem, Ubuntu 10.04, ruby 1.9.3p0, Rails 3.1.1, Apache. Happy to help debug or gather more information if that would be helpful...

FooBarWidget commented 10 years ago

From gr2020 on November 07, 2011 08:44:10

  • Were you on a previous version of Ruby before, or did you start out with Ruby 1.9.3?

I've tried both. Upgrading from 1.9.2p180 to 1.9.3p0, and also installing fresh with 1.9.3p0 both show the same behavior.

  • If you were on a previous version of Ruby, how did you upgrade to Ruby 1.9.3? Did you just use 'rvm install ruby-1.9.3' and tried to make Phusion Passenger use it (i.e. changing the PassengerRuby directive)?

I'm not using rvm on my staging/production machines, so I just rebuilt and re-installed ruby 1.9.3 from source. Then changed the stuff in the apache config file as directed.

  • What happens if you reinstall the passenger gem/tarball?

Well, I can repro when starting from scratch, so I assume that gets to the point...I've tried a) installing 3.0.9 from scratch, and also b) upgrading from 3.0.7, same behavior in both.

Happy to try that; although not sure what to do after I pull down the source...?

FooBarWidget commented 10 years ago

From gr2020 on November 07, 2011 08:50:01

I could probably build a VMware VM pretty quickly that will demonstrate the problem...would that be helpful?

FooBarWidget commented 10 years ago

From honglilai on November 07, 2011 08:52:42

Installation instructions are in the README. Basically you just run the same commands, except this time they're in bin/. Thanks for the offer but let's try the normal route first before playing around with multi-megabyte VM images.

FooBarWidget commented 10 years ago

From gregr@tradervue.com on November 07, 2011 09:24:14

I just tried installing via git master on Ubuntu 10.04 - same error message.

FooBarWidget commented 10 years ago

From m.veldthuis on November 07, 2011 10:10:49

Having the same issues, using Ubuntu 11.10, rbenv, ruby 1.9.3-p0 and Passenger 3.0.9.

I "upgraded" from 1.9.2-p290, but since it's a completely seperate GEM_HOME I guess it's more like a fresh install (since it's not really installing it on top of the existing Ruby).

I'm using rbenv with the latest ruby-build. Did a rbenv install 1.9.3-p0, then installed the passenger gem, and ran passenger-install-apache2-module.

Gives the same results. I'd be happy to provide more specific version numbers of Ubuntu packages if you can give a list of the interesting ones.

FooBarWidget commented 10 years ago

From gregr@tradervue.com on November 07, 2011 10:49:41

I've got a VMware VM now that reproduces this with a "rails new ..." app; unfortunately the VM is 2.2GB. I think it's mostly empty disk, but I don't know of an easy way to shrink it further. Anyway, happy to upload it somewhere you can get to it if you want.

I could also set up a Linode with a similar simple configuration that fails, and let you ssh into it, if that would be useful instead.

FooBarWidget commented 10 years ago

From jdcrutchley@spiffworks.com on November 07, 2011 10:55:52

Here's my info FWIW:

Started with Ruby 1.9.2p290 (installed via RVM) and Passenger 3.0.9. Installed Ruby 1.9.3p0 rvm get head rvm reload rvm install 1.9.3 rvm --default use 1.9.3 Installed passenger 3.0.9 gem install passenger -v '3.0.9' Stopped nginx (1.0.6) and did a custom re-build ./configure --prefix='/usr/local/nginx' --with-http_ssl_module --with-http_gzip_static_module --with-pcre='/usr/local/src/pcre-8.12' --add-module='/usr/local/src/nginx-upload-module-2.2.0' --add-module='/usr/local/src/nginx-upload-progress-module-0.8.2' --add-module='/usr/local/rvm/gems/ruby-1.9.3-p0/gems/passenger-3.0.9/ext/nginx' make checkinstall --default --pkgname=nginx --pkgversion=1.0.6 --nodoc Updated nginx.conf to point at new 1.9.3 installation - DIDN'T WORK. Put nginx.conf back the way it was - WORKS AGAIN.

FooBarWidget commented 10 years ago

From honglilai on November 08, 2011 01:28:16

gregr: would be great if you can provide access to a Linode for further investigation.

FooBarWidget commented 10 years ago

From gregr@tradervue.com on November 08, 2011 08:45:42

hongli: ok, I built a Linode that's demonstrating the problem. I emailed details for you to get to it; I sent it to both your gmail address here, and the phusion.nl address you use on the mailing list.

FooBarWidget commented 10 years ago

From mfwhite5 on November 11, 2011 20:15:43

Got this error message on Debian Lenny (5.0) and Apache 2.2.16. Had Ruby 1.9.2p290 (installed from source) and Passenger 3.0.9 working fine. After upgrading to Ruby 1.9.3p0 (installed from source) the error started occurring. Reinstalling Passenger gem and rebuilding Apache module didn't help.

FooBarWidget commented 10 years ago

From jeff.duckett on November 13, 2011 14:48:56

I created a video of what I think is the same error. http://www.youtube.com/watch?v=T9-zLOThxds Attached is the error log from the video.

Attachment: error.log

FooBarWidget commented 10 years ago

From honglilai on November 16, 2011 04:37:05

Hi Greg. Thanks to your test machine I was able to find the cause of the problem.

Phusion Passenger checks for the existence of the function rb_reserved_fd_p(), which was introduced in Ruby 1.9.3. We use this API function to properly close file descriptors. On your machine the test for rb_reserved_fd_p() fails because apparently the linking flags as reported by Ruby are wrong. Here's a snippet from mkmf.log:

have_func: checking for rb_reserved_fd_p()... -------------------- no

"gcc -o conftest -I/usr/local/include/ruby-1.9.1/i686-linux -I/usr/local/include/ruby-1.9.1/ruby/backward -I/usr/lo cal/include/ruby-1.9.1 -I.. -D_FILE_OFFSET_BITS=64 -O3 -ggdb -Wextra -Wno-unused-parameter -Wno-parentheses -Wno -long-long -Wno-missing-field-initializers -Wpointer-arith -Wwrite-strings -Wdeclaration-after-statement -Wimplicit -function-declaration -g conftest.c -L. -L/usr/local/lib -Wl,-R/usr/local/lib -L. -rdynamic -Wl,-export-dynamic
-Wl,-R -Wl,/usr/local/lib -L/usr/local/lib -lruby-static " /usr/local/lib/libruby-static.a(thread.o): In function ruby_thread_from_native': /home/phusion/railsready/src/ruby-1.9.3-p0/thread_pthread.c:420: undefined reference topthread_getspecific' ... /home/phusion/railsready/src/ruby-1.9.3-p0/thread_pthread.c:357: undefined reference to clock_gettime' ... /home/phusion/railsready/src/ruby-1.9.3-p0/bignum.c:1029: undefined reference toceil' ...etcetera

How did you install Ruby? It doesn't look like you've installed it through RVM. And it looks like you've configured Ruby to use static libraries. Is that right?

FooBarWidget commented 10 years ago

From gregr@tradervue.com on November 16, 2011 05:50:34

Hi Hongli,

Ruby was built from source; we don't use rvm on our production machines.

It was built with the following:

tar -xzf ruby-1.9.3-p0.tar.gz cd ruby-1.9.3-p0 ./configure --prefix=/usr/local make sudo make install

Just FYI, these were the same steps used for 1.9.2p180, where Passenger works great.

Not sure if it matters, but the Ubuntu libraries installed prior to this were from

sudo apt-get install \ build-essential clang \ bison openssl zlib1g \ libxslt1.1 libssl-dev libxslt1-dev \ libxml2 libffi-dev libyaml-dev \ libxslt-dev autoconf libc6-dev \ libreadline6-dev zlib1g-dev libcurl4-openssl-dev

I'm also not sure if I've configured ruby to use static libraries, but hopefully the answer is in the above. :-)

Thanks!

FooBarWidget commented 10 years ago

From jdcrutchley@spiffworks.com on November 16, 2011 06:08:42

Hongli,

I DID install with RVM, on two different Linodes, and get the same results. I do have RVM installed system-wide if that makes any difference. Please see my comments above for my exact installation sequence. The only thing I did manually was the nginx build because I needed some extra modules. Would that have caused the same thing you're seeing?

Thanks for looking into this.

FooBarWidget commented 10 years ago

From honglilai on November 16, 2011 06:12:50

jdcrutchley: on Greg's machine the problem is caused because of Ruby linking flag problems. I want to check whether that's also the case on your machine. Could you follow these instructions?

  1. Type 'passenger-config --root'. You will see a directory name.
  2. Type 'cd '
  3. Type 'cd ext/ruby'
  4. Somewhere in a subfolder you will find a file called mkmf.log. Please attach it here.
FooBarWidget commented 10 years ago

From jdcrutchley@spiffworks.com on November 16, 2011 06:21:47

Hongli,

I can't get a full log file right now, but I can cut and paste parts. This looks like it's relevant:

have_func: checking for rb_reserved_fd_p()... -------------------- yes

I'll get you the full log file when I get back to my own network. Thanks!

FooBarWidget commented 10 years ago

From honglilai on November 16, 2011 06:32:24

jdcrutchley: it looks like you're having a different problem. The problem should only appear if the check for rb_reserved_fd_p() fails. Could you provide me access to a machine that displays the problem?

FooBarWidget commented 10 years ago

From honglilai on November 16, 2011 06:37:07

Greg: could you try Phusion Passenger from git master? I've committed a workaround that may solve your problem.

FooBarWidget commented 10 years ago

From gregr@tradervue.com on November 16, 2011 06:39:31

hongli: Sure - I'm headed out for a bit at the moment, but I'll try it later today. You're also welcome to do it on that test machine if you like - it has git installed, and I can re-image it back to the original if need be...

FooBarWidget commented 10 years ago

From gregr@tradervue.com on November 16, 2011 09:59:21

hongli: pulling the code from git master indeed appears to fix the problem! Nice work.

So now two questions:

  1. Is that workaround suitable for production use, or is it still a work in progress?
  2. Looking at https://github.com/FooBarWidget/passenger/compare/release-3.0.9...master I see there have been quite a few commits between release-3.0.9 and master; do you think it's safe to deploy as of this latest commit? Or, do you have plans for a 3.0.10-ish release anytime soon?

Thanks again!

FooBarWidget commented 10 years ago

From cary@rightpact.com on November 16, 2011 11:16:49

I've also verified installing passenger from master on Ubuntu 11.04/Apache/ruby 1.9.3p0 worked for me. Thanks as well!

FooBarWidget commented 10 years ago

From john.nishinaga.placeholder@patdeegan.com on November 16, 2011 11:22:36

Hi Hongli,

Using Centos 6.0, Ruby 1.9.3p0 and Apache 2.2.15-5, the latest changes to passenger on the master branch also worked for me.

Thanks, John

FooBarWidget commented 10 years ago

From honglilai on November 16, 2011 13:52:36

Yes, git master is suitable for production use. Will release 3.0.10 soon.

Labels: Milestone-3.0.10

FooBarWidget commented 10 years ago

From honglilai on November 16, 2011 13:53:16

On a related note, someone should report a bug at the Ruby bug tracker. This is definitely a bug in their build system.

FooBarWidget commented 10 years ago

From gregr@tradervue.com on November 16, 2011 17:33:37

Hongli, do you still need that Linode for testing, or can I shut it down now?

FooBarWidget commented 10 years ago

From honglilai on November 17, 2011 01:25:06

If everything works for you then you can shut it down now. Thanks again for providing it.

FooBarWidget commented 10 years ago

From gregr@tradervue.com on November 17, 2011 16:04:28

Pretty sure it's working, so I took it down. I'll do some more testing tomorrow, and will let you know if anything strange happens related to this. Thanks so much for looking into this and fixing it! :-)

FooBarWidget commented 10 years ago

From frankske on November 23, 2011 01:30:49

Debian Squeeze, Ruby 1.9.3 from source has same issue with "stock" passenger 3.0.9 gem.

I can confirm it works with the current master branch on github!

FooBarWidget commented 10 years ago

From m.veldthuis on November 23, 2011 03:08:47

Yep, this fix works fine for me as well.

FooBarWidget commented 10 years ago

From pete@peterlyons.com on December 21, 2011 14:25:37

This was fixed and released in 3.0.10, right? Should be able to mark as fixed now I believe.

FooBarWidget commented 10 years ago

From honglilai on December 21, 2011 21:34:02

Status: Fixed
Labels: -Priority-Medium Priority-Critical