Install openssl 1.0 on EL9 Jenkins nodes through RVM

[ehelms]

I realize this won't work for our jobs that perform unit tests and still need Ruby 2.7.

please read /usr/local/rvm/log/1705408021_ruby-2.5.1/make.log

There has been an error while running make. Halting the installation.
Error: /Stage[main]/Slave::Rvm/Slave::Rvm_config[ruby-2.5]/Rvm_system_ruby[ruby-2.5.1]/ensure: change from 'absent' to 'present' failed: Execution of '/usr/local/rvm/bin/rvm install ruby-2.5.1 ' returned 2: ruby-2.5.1 - #removing src/ruby-2.5.1..
Searching for binary rubies, this might take some time.
No binary rubies available for: centos/9/x86_64/ruby-2.5.1.
Continuing with compilation. Please read 'rvm help mount' to get more information on binary rubies.
Checking requirements for centos.
Requirements installation successful.
Installing Ruby from source to: /usr/local/rvm/rubies/ruby-2.5.1, this may take a while depending on your cpu(s)...
ruby-2.5.1 - #downloading ruby-2.5.1, this may take a while depending on your connection...
ruby-2.5.1 - #extracting ruby-2.5.1 to /usr/local/rvm/src/ruby-2.5.1.....
ruby-2.5.1 - #applying patch /usr/local/rvm/patches/ruby/2.5.1/libressl_2_7.patch.
ruby-2.5.1 - #configuring...................................................................
ruby-2.5.1 - #post-configuration..
ruby-2.5.1 - #compiling.......................................................................................................
Error running '__rvm_make -j8',
please read /usr/local/rvm/log/1705408021_ruby-2.5.1/make.log

There has been an error while running make. Halting the installation.
Notice: /Stage[main]/Slave::Rvm/Slave::Rvm_config[ruby-2.5]/Rvm_alias[ruby-2.5]: Dependency Rvm_system_ruby[ruby-2.5.1] has failures: true
Warning: /Stage[main]/Slave::Rvm/Slave::Rvm_config[ruby-2.5]/Rvm_alias[ruby-2.5]: Skipping because of failed dependencies
Warning: /Stage[main]/Slave::Rvm/Slave::Rvm_config[ruby-2.5]/Exec[ruby-2.5.1/update_rubygems]: Skipping because of failed dependencies
Error: Execution of '/usr/local/rvm/bin/rvm install ruby-2.7.4 ' returned 2: ruby-2.7.4 - #removing src/ruby-2.7.4..
Searching for binary rubies, this might take some time.
No binary rubies available for: centos/9/x86_64/ruby-2.7.4.
Continuing with compilation. Please read 'rvm help mount' to get more information on binary rubies.
Checking requirements for centos.
Requirements installation successful.
Installing Ruby from source to: /usr/local/rvm/rubies/ruby-2.7.4, this may take a while depending on your cpu(s)...
ruby-2.7.4 - #downloading ruby-2.7.4, this may take a while depending on your connection...
ruby-2.7.4 - #extracting ruby-2.7.4 to /usr/local/rvm/src/ruby-2.7.4.....
ruby-2.7.4 - #configuring........................................................................
ruby-2.7.4 - #post-configuration..
ruby-2.7.4 - #compiling......................................................................................................-
Error running '__rvm_make -j8',
please read /usr/local/rvm/log/1705408149_ruby-2.7.4/make.log

There has been an error while running make. Halting the installation.
Error: /Stage[main]/Slave::Rvm/Slave::Rvm_config[ruby-2.7]/Rvm_system_ruby[ruby-2.7.4]/ensure: change from 'absent' to 'present' failed: Execution of '/usr/local/rvm/bin/rvm install ruby-2.7.4 ' returned 2: ruby-2.7.4 - #removing src/ruby-2.7.4..
Searching for binary rubies, this might take some time.
No binary rubies available for: centos/9/x86_64/ruby-2.7.4.
Continuing with compilation. Please read 'rvm help mount' to get more information on binary rubies.
Checking requirements for centos.
Requirements installation successful.
Installing Ruby from source to: /usr/local/rvm/rubies/ruby-2.7.4, this may take a while depending on your cpu(s)...
ruby-2.7.4 - #downloading ruby-2.7.4, this may take a while depending on your connection...
ruby-2.7.4 - #extracting ruby-2.7.4 to /usr/local/rvm/src/ruby-2.7.4.....
ruby-2.7.4 - #configuring........................................................................
ruby-2.7.4 - #post-configuration..
ruby-2.7.4 - #compiling......................................................................................................-
Error running '__rvm_make -j8',
please read /usr/local/rvm/log/1705408149_ruby-2.7.4/make.log

There has been an error while running make. Halting the installation.

[ehelms]

Any idea why Ruby 2.7 fails to build? Whats in /usr/local/rvm/log/1705408149_ruby-2.7.4/make.log.

I am still digging, it appears to be Openssl related (EL9 comes with openssl 3.0). There are a number of similar issues https://github.com/rvm/rvm/issues

[ehelms]

I am wondering if we will need to use rvm to install openssl 1.1 to then use it, e.g.

/usr/local/rvm/bin/rvm pkg install openssl

Beware, 'rvm pkg ...' is deprecated, read about the new autolibs feature: 'rvm help autolibs'.

Checking requirements for centos.
Requirements installation successful.
Fetching openssl-1.0.1i.tar.gz to /usr/local/rvm/archives
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 4318k  100 4318k    0     0  2848k      0  0:00:01  0:00:01 --:--:-- 2846k
Extracting openssl to /usr/local/rvm/src/openssl-1.0.1i.....
Configuring openssl in /usr/local/rvm/src/openssl-1.0.1i...................................
Compiling openssl in /usr/local/rvm/src/openssl-1.0.1i........................................................................
Installing openssl to /usr/local/rvm/usr.....................................................................................|

Please note that it's required to reinstall all rubies:

    rvm reinstall all --force

Updating openssl certificates....

[ehelms]

And then it gets used like this for installs:

/usr/local/rvm/bin/rvm install ruby-3 --with-openssl-dir=/usr/local/rvm/usr

Where we are not replacing system openssl, just providing 1.1 for compilation by RVM. The alternatives from what I can tell would be:

• Stay on EL8 only til this gets resolved for our nodes
• Swap to running inside a container for our tests in Jenkins

[ekohl]

https://bugs.ruby-lang.org/issues/18658 suggests you could install the openssl v3 gem on Ruby 2.7. Interestingly enough, it also mentions Ruby 3.0 not getting the backport. That makes me wonder if RVM with Ruby 3.0 would have problems on EL9 hosts.

[evgeni]

Having compat-openssl11 is not sufficient, right? As it doesn't have -devel?

[ehelms]

https://bugs.ruby-lang.org/issues/18658 suggests you could install the openssl v3 gem on Ruby 2.7. Interestingly enough, it also mentions Ruby 3.0 not getting the backport. That makes me wonder if RVM with Ruby 3.0 would have problems on EL9 hosts.

When I tested this on my own, I did hit the same problem with Ruby 3.0. On a Jenkins node, Ruby 3 and 3.1 did not throw any errors.

[ehelms]

Having compat-openssl11 is not sufficient, right? As it doesn't have -devel?

I was not able to use it to success and as you say, couldn't find a devel.

[ehelms]

I have updated the code and intent of this PR to install openssl through rvm pkg. While this is a bit ugly, I cannot find another viable solution. And while rvm pkg is deprecated, the replacement autolibs does not solve the problem nor does it provide a replacement method to solve the problem.

[evgeni]

I think long-term we should get rid of rvm (either by using rbenv, which knows that it needs to build OpenSSL for those old Rubies, or by running those tests in dedicated throw-away containers). But for now I think this is a good enough band aid.

[ehelms]

I also now looking closer think it's using Openssl 1.0 and not 1.1:

# ls /usr/local/rvm/usr/lib/
engines  libcrypto.a  libcrypto.so  libcrypto.so.1.0.0  libssl.a  libssl.so  libssl.so.1.0.0  pkgconfig

[evgeni]

@ehelms did you try to run our testsuite on a node built with this? Reading https://github.com/rvm/rvm/issues/5209 it seems that a Ruby built with OpenSSL 1.0 will segfault when trying to use a rubygem-pq that is linked against system OpenSSL (so 3.0) via libpq. I'd prefer not to have also compile PostgreSQL.

Can we spin up a few (RH)EL8 nodes and use those in the meantime?

[ehelms]

Based on the discussions, here's how I see proceeding:

1. Update existing EL 7 nodes to EL 8 first
2. Switch to rbenv from rvm
3. Update to EL 9

[ekohl]

Perhaps we need to start using labels for available Ruby versions

[ehelms]

Perhaps we need to start using labels for available Ruby versions

I get the thought there, but upon further testing even Ruby 3 via RVM does not want to work reliably on EL 9. That is why I am saying we at least undertake updates to EL 8 and then consider a swap to rbenv instead of RVM.

[ekohl]

I have no love for RVM so I don't object to anything else. Even using system Ruby would be fine by me if we can dind a good way to isolate tests (GEM_HOME might be an option for that) but the downside is that we currently don't have parallel installable Rubies on RPM and EL9 wouldn't have 2.7 either. Rbenv works well for me but the concept of gemsets that RVM has looks like a separate gem that was questionable in my first impression. Which is why I didn't dare the migration myself yet

On the Puppet side the RVM module (and especially its GPG dependency) is on life support so there I also considered rbenv already

[evgeni]

Do we really need gemsets when we use Bundler?

gemsets create an empty environment whenever we run tests, which means we gotta download and compile all the things every time. but what is later used in the process is managed by bundler, which already does enough isolation for the running process? or is the worry that things get overwritten/corrupted when multiple tests in parallel try to write to ~/.cache/gems etc?

[ekohl]

I think that's a worry, but I have no idea how real that worry is. Another is that rubygems < 3.5 doesn't use user installs by default, so if the Ruby is system wide it wants to do nasty things like use sudo etc. Rubygems 3.5+ detects the directory is not writable and uses user mode installs. I wonder if you can teach older bundler versions this.

With the move to GitHub Actions I think we should ask ourselves: how are we going to continue testing in source build pipelines? Do we even need to do that anymore? Can we avoid Ruby testing in Jenkins altogether somehow?

[evgeni]

I think setting BUNDLE_PATH (or gem home/path) should fix that.

We only really need the gem testing in Jenkins as part of the nightly source generation/store. We could use the GitHub gem repository for that, but not sure I like that.

[ekohl]

Yes, the bundle paths is afaik also what gha does so we can certainly try that

[ehelms]

Closed in favor of https://github.com/theforeman/foreman-infra/pull/2022