search

phusion / passenger

A fast and robust web server and application server for Ruby, Python and Node.js
https://www.phusionpassenger.com/
MIT License
5.01k stars 548 forks source link

passenger-config fails with "too long unix socket path" on Ubuntu 18.04 LTS #2397

Open bpfh opened 3 years ago

bpfh commented 3 years ago

Issue report

When running Passenger on hosts upgraded from Ubuntu 16.04 LTS to 18.04 LTS, passenger-config fails with 

Traceback (most recent call last):
    9: from /usr/bin/passenger-config:37:in `<main>'
    8: from /usr/lib/ruby/vendor_ruby/phusion_passenger/config/main.rb:79:in `run!'
    7: from /usr/lib/ruby/vendor_ruby/phusion_passenger/config/restart_app_command.rb:46:in `run'
    6: from /usr/lib/ruby/vendor_ruby/phusion_passenger/config/restart_app_command.rb:144:in `select_app_group_name'
    5: from /usr/lib/ruby/vendor_ruby/phusion_passenger/config/restart_app_command.rb:174:in `select_app_group_name_interactively'
    4: from /usr/lib/ruby/vendor_ruby/phusion_passenger/config/restart_app_command.rb:164:in `query_group_names'
    3: from /usr/lib/ruby/vendor_ruby/phusion_passenger/config/restart_app_command.rb:255:in `query_pool_xml'
    2: from /usr/lib/ruby/vendor_ruby/phusion_passenger/admin_tools/instance.rb:95:in `http_request'
    1: from /usr/lib/ruby/vendor_ruby/phusion_passenger/admin_tools/instance.rb:95:in `new'
/usr/lib/ruby/vendor_ruby/phusion_passenger/admin_tools/instance.rb:95:in `initialize': too long unix socket path (116bytes given but 108bytes max) (ArgumentError)

Question 1: What is the problem?

Running passenger-config is possible.

passenger-config fails with the above stack trace. Debugging shows that the UNIX socket path Passenger tries to use is something like

/tmp/systemd-private-673f3b327552487c9d5cc4d934decaf-apache2.service-GfIBRc/tmp/passenger.UqAQsAS/agents.s/core_api

which is over the system limit.

I have not tried on a vanilla 18.04, but there seems to be additional evidence here, with no solution:

https://stackoverflow.com/questions/70012839/phusion-passenger-status-crashing-with-too-long-socket-name

Your answer:

Question 2: Passenger version and integration mode:

Your answer:

Ubuntu libapache2-mod-passenger 1:6.0.12-1~bionic1

Question 3: OS or Linux distro, platform (including version):

Your answer:

Ubuntu 18.04.6 LTS

Question 4: Passenger installation method:

Your answer:

4.15.0-162-generic #170-Ubuntu SMP Mon Oct 18 11:38:05 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

[] RubyGems + Gemfile [ ] RubyGems, no Gemfile [x] Phusion APT repo [ ] Phusion YUM repo [ ] OS X Homebrew [ ] source tarball [ ] Other, please specify:

Question 5: Your app's programming language (including any version managers) and framework (including versions):

Your answer:

System Ruby is ruby 2.5.1p57 (2018-03-29 revision 63029) [x86_64-linux-gnu]

Question 6: Are you using a PaaS and/or containerization? If so which one?

Your answer:

No, plain VM

Question 7: Anything else about your setup that we should know?

Your answer:

As mentioned before, the system was upgraded from Ubuntu 16.04 LTS.

CamJN commented 3 years ago

Yes, this is because the path generated by the systemd private tmp feature is 80 chars long, which only leaves 27 chars for an application to use, and passenger uses 35 chars in its path to its socket.

Passenger's part of the path is: passenger.UqAQsAS/agents.s/core_api

The first part of the path identifies that passenger made the directory, and has a random segment to prevent symlink attacks, then the rest is just the path to the socket.

There are several ways to deal with this, such as:

bpfh commented 3 years ago

Thanks for the pointers @CamJN !

For posterity, I resolved this by turning off the systemd private tmp feature by commenting out the line

PrivateTmp=true

in /etc/systemd/system/multi-user.target.wants/apache2.service and then executing

$ sudo systemctl daemon-reload
$ sudo systemctl restart apache2

Now passenger-control has been restored to its former glory.

coopeu commented 2 years ago

How to fix this same error /usr/lib/ruby/vendor_ruby/phusion_passenger/admin_tools/instance.rb: `initialize': too long unix socket path (116bytes given but 108bytes max) (ArgumentError) of an Apache/2.4.51 (Debian 11 BullsEye) Phusion_Passenger/6.0.12) Ruby 3.1.0 and Rails 7.0.0 /etc/systemd/user/sockets.target.wants not including apache2.service file

bpfh commented 2 years ago

@coopeu, I suspect following the instructions in https://github.com/phusion/passenger/issues/2397#issuecomment-982327504 would work for Debian also.

coopeu commented 2 years ago

Maybe the passenger error origin is different: The detailed diagnostic is: Initialize language runtime — 0.2s The backtrace is: in 'bool Passenger::SpawningKit::HandshakePerform::checkCurrentState()' (Perform.h:238) ... The environment variables are: Environment variables LANGUAGE=en_US APACHE_RUN_DIR=/var/run/apache2 APACHE_PID_FILE=/var/run/apache2/apache2.pid JOURNAL_STREAM=8:264760 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin INVOCATION_ID=3b7132069baf4f2f8fd3a2773ca024ea APACHE_LOCK_DIR=/var/lock/apache2 LANG=C APACHE_RUN_USER=www-data APACHE_RUN_GROUP=www-data APACHE_LOG_DIR=/var/log/apache2 PWD=/ PASSENGER_USE_FEEDBACK_FD=true SERVER_SOFTWARE=Apache/2.4.51 (Debian) Phusion_Passenger/6.0.12

I hope this help to identify the error and to find the solution. Thanks in advance!

SeanSith commented 2 years ago

I'm confused why this got closed, honestly? This, to me, is a bug that needs to be fixed, not band-aided by adjusting apache2's systemd file. This package, including the one compatible with Ubuntu 20.04 is effectively broken straight out of the box.

@CamJN is there any chance that Phusion would take this back to the drawing board or raise an issue upstream with systemd?

dlundgren commented 2 years ago

For Ubuntu 18+ I'd recommend doing this instead of altering the systemd file

This should work ok as the passenger tooling looks there as one of it's default paths for the instance registry

CamJN commented 2 years ago

That sounds like a great approach, i'll see if i can add it to the .deb packages.

dlundgren commented 2 years ago

The correct settings for /usr/lib/tmpfiles.d/passenger.conf is d /var/run/passenger-instreg 0755 root root -

CamJN commented 1 year ago

6.0.16 packages are out now and have the mitigation for this.

saiqulhaq-hh commented 1 year ago

6.0.16 causing 4 hours down in my backend because I don't know that there was issue on passenger passenger can't create /var/run/passenger-instreg directory

dlundgren commented 1 year ago

Due to the way that the tmpfiles works, the system needs to be rebooted in order for the folder to be created automatically.

The package installation should probably try to create that folder if it doesn't already exist. This should help prevent the need to reboot.

saiqulhaq-hh commented 1 year ago

I use Kubernetes there is no Dockerfile changes have rebuilt the image few times already before I found out that the issue is on /var/run/

CamJN commented 1 year ago

You can run systemd-tmpfile --create to apply the change without rebooting, in fact, in my testing that is run by dpkg automatically when the new passenger is installed.

CamJN commented 1 year ago

In a docker/kubernetes type setup systemd may be missing, so it probably won't create the dir for you.

CamJN commented 1 year ago

I had to deal with that over in https://github.com/phusion/passenger-docker/issues/352 by creating the dir manually in the passenger-docker images.

saiqulhaq-hh commented 1 year ago

I think 6.0.16 should be a new major version i did the same on my Dockerfile had to add RUN mkdir /var/run/passenger-instreg to make it run

bashcoder commented 1 year ago

Just wanted to document that this exact same issue is present with nginx running on Amazon Linux 2023 and Passenger v 6.0.18. The same remedy works in this environment. Add the directory and add this directive to /usr/conf/nginx.conf:

passenger_instance_registry_dir /var/run/passenger-instreg;