passenger_native_support errors

[stdweird]

on a el8 ondemadn 2.0.18 setup, i get non-fatal errors in the user error logs e.g.

App 57712 output:  [passenger_native_support.so] trying to compile for the current user (myusername) and Ruby interpreter...
App 57712 output:      (set PASSENGER_COMPILE_NATIVE_SUPPORT_BINARY=0 to disable)
App 57712 output:      Warning: compilation didn't succeed. To learn why, read this file:
App 57712 output:      /tmp/passenger_native_support-1v1mw2e.log
App 57712 output:  [passenger_native_support.so] finding downloads for the current Ruby interpreter...
App 57712 output:      (set PASSENGER_DOWNLOAD_NATIVE_SUPPORT_BINARY=0 to disable)
App 57712 output:      Could not download https://github.com/phusion/passenger/releases/download/release-6.0.7/rubyext-ruby-2.7.4-x86_64-linux.tar.gz: The requested URL returned error: 404 
App 57712 output:      Trying next mirror...
App 57712 output:      Could not download https://oss-binaries.phusionpassenger.com/binaries/passenger/by_release/6.0.7/rubyext-ruby-2.7.4-x86_64-linux.tar.gz: The requested URL returned error: 404 
App 57712 output:      Trying next mirror...
App 57712 output:      Could not download https://s3.amazonaws.com/phusion-passenger/binaries/passenger/by_release/6.0.7/rubyext-ruby-2.7.4-x86_64-linux.tar.gz: The requested URL returned error: 403 Forbidden
App 57712 output:  [passenger_native_support.so] will not be used (can't compile or download) 
App 57712 output:   --> Passenger will still operate normally.

the erorr log in /tmp looks like

# current user is: nyusername
# mkdir -p /myhome/.passenger/native_support/6.0.7/ruby-2.7.4-x86_64-linux
# cd /myhome/.passenger/native_support/6.0.7/ruby-2.7.4-x86_64-linux
# /usr/bin/ruby /opt/rh/ondemand/root/usr/share/passenger/ruby_extension_source/extconf.rb
checking for alloca.h... *** /opt/rh/ondemand/root/usr/share/passenger/ruby_extension_source/extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers.  Check the mkmf.log file for more details.  You may
need configuration options.

Provided configuration options:
    --with-opt-dir
    --without-opt-dir
    --with-opt-include
    --without-opt-include=${opt-dir}/include
    --with-opt-lib
    --without-opt-lib=${opt-dir}/lib64
    --with-make-prog
    --without-make-prog
    --srcdir=/opt/rh/ondemand/root/usr/share/passenger/ruby_extension_source
    --curdir
    --ruby=/usr/bin/$(RUBY_BASE_NAME)
    --with-alloca-dir
    --without-alloca-dir
    --with-alloca-include
    --without-alloca-include=${alloca-dir}/include
    --with-alloca-lib
    --without-alloca-lib=${alloca-dir}/lib64
/usr/share/ruby/mkmf.rb:471:in `try_do': The compiler failed to generate an executable file. (RuntimeError)
You have to install development tools first.
    from /usr/share/ruby/mkmf.rb:613:in `try_cpp'
    from /usr/share/ruby/mkmf.rb:1124:in `block in have_header'
    from /usr/share/ruby/mkmf.rb:971:in `block in checking_for'
    from /usr/share/ruby/mkmf.rb:361:in `block (2 levels) in postpone'
    from /usr/share/ruby/mkmf.rb:331:in `open'
    from /usr/share/ruby/mkmf.rb:361:in `block in postpone'
    from /usr/share/ruby/mkmf.rb:331:in `open'
    from /usr/share/ruby/mkmf.rb:357:in `postpone'
    from /usr/share/ruby/mkmf.rb:970:in `checking_for'
    from /usr/share/ruby/mkmf.rb:1123:in `have_header'
    from /opt/rh/ondemand/root/usr/share/passenger/ruby_extension_source/extconf.rb:45:in `<main>'

something missing? can i install something so this isn't required for all users?

[treydock]

How did you install OnDemand? Because if you installed via the RPMs, which is the best method for EL8, the file /opt/rh/ondemand/root/usr/lib64/ruby/vendor_ruby/passenger_native_support.so should be present and loaded when OnDemand goes to launch. The package ondemand-passenger-devel has that extconf.rb from output but that package should not be necessary to run OnDemand. The file /opt/rh/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/locations.ini with RPM installs will have the location for passenger_native_support.so directory via ruby_libdir parameter.

[stdweird]

it's an rpm install. the ini has ruby_libdir=/opt/rh/ondemand/root/usr/share/ruby/vendor_ruby which points to usr/share, not usr/lib64. the .so file is there in the location you mentioned

[treydock]

Have you done anything custom to environment launching OnDemand? We still run OnDemand on RHEL7 but we have E2E testing and Docker images that run OnDemand on Rocky 8 and unable to reproduce this in those environments. Are you running SELinux by chance? If you have SELinux enabled, might be something in /var/log/audit/audit.log to indicate a denial of some kind.

The ondemand SCL is what sets RUBYLIB environment variable to include /opt/ood/ondemand/root/usr/lib64/ruby/vendor_ruby and that SCL is loaded by several things.

A few things to check I can think of, the value $USERNAME would be replaced with whichever user you're logging into OnDemand with:

# grep ruby /var/lib/ondemand-nginx/config/puns/$USERNAME.conf 
  passenger_root /opt/ood/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/locations.ini;
  passenger_ruby /opt/ood/nginx_stage/bin/ruby;

# scl enable ondemand -- env | grep RUBYLIB
RUBYLIB=/opt/ood/ondemand/root/usr/share/ruby/vendor_ruby:/opt/ood/ondemand/root/usr/lib64/ruby/vendor_ruby

[stdweird]

@treydock no selinux, but yes on custom env. when i run the scl enable command in a shell in that env, i get same output. i'll see if strace points to something

[stdweird]

@treydock i ran it all though strace, neither path in the scl RUBYLIB is checked. lots of other paths are checked, incl the ondemand gems.

[treydock]

@treydock no selinux, but yes on custom env.

Could you describe the custom env?

The thing to launch NGINX which is what uses Passenger is nginx_stage, specifically /opt/ood/nginx_stage/sbin/nginx_stage. That script will source two files, one is /opt/ood/nginx_stage/etc/profile which loads ondemand SCL and the other is /etc/ood/profile, if it exists. Do you have anything in /etc/ood/profile?

[stdweird]

@treydock our custom wrappers missed to pass an env variable. this was a mistake on our side apologies to waste your time.