SCF 2.19.1 on IBM Kubenetes Cluster cf push ruby sample app failed with openssl library

[EmilyEmily]

Describe the bug I deployed SCF 2.19.1 on IBM Kebenetes Cluster with full eirini feature enabled. After deployment, I can push nodejs, staticfile applications successfully. But when I push ruby simple application, it failed with error of version OPENSSL_1_1_0 not defined in file libcrypto.so.1.1 with link time reference - /tmp/contents273973045/deps/0/ruby/lib/ruby/2.4.0/x86_64-linux/openssl.so (LoadError) .

I asked this in #quarks-dev slack channel, Jan Dubois mentioned that it may be caused by cflinuxfs3. Then I manually verified that the existing eirinifs (version 40.0.0) tar package has issue with openssl command, while a newer version of eirinifs 47.0.0 seems ok when running it’s openssl command. So I submitted a PR to publish a new bits-service release with eirinifs 47.0.0, the PR is approved but no one is merging it.

Do you have any suggestions? Could you let me know SCF's plan to upgrade cflinuxfs3?

To Reproduce Deployed SCF 2.19.1 on IBM Kubenetes Cluster with ubuntu based image, and then push ruby simple application from CF CATs repository.

Expected behavior cf push ruby simple application succeeded.

Logs

:/gopath/src/github.com/cloudfoundry/cf-smoke-tests/smoke/logging# cf push testruby -b ruby_buildpack -p ../../assets/ruby_simple -d teststream.eu-de.containers.appdomain.cloud
......
   -----> Installing ruby 2.4.9
   Download [https://buildpacks.cloudfoundry.org/dependencies/ruby/ruby-2.4.9-linux-x64-cflinuxfs3-b8eb2847.tgz]
   -----> Update rubygems from 2.6.14.4 to 3.0.6
   -----> Installing rubygems 3.0.6
   Download [https://buildpacks.cloudfoundry.org/dependencies/rubygems/rubygems-3.0.6-any-stack-1a97a90e.tgz]
   -----> Update rubygems from 2.6.14.4 to 3.0.6
   -----> Installing rubygems 3.0.6
   Download [https://buildpacks.cloudfoundry.org/dependencies/rubygems/rubygems-3.0.6-any-stack-1a97a90e.tgz]
   /tmp/rubygems640991822/rubygems-3.0.6/lib/rubygems/core_ext/kernel_require.rb:54:in `require': /tmp/contents273973045/deps/0/ruby/lib/ruby/2.4.0/x86_64-linux/openssl.so: symbol EC_GROUP_new_curve_GF2m, version OPENSSL_1_1_0 not defined in file libcrypto.so.1.1 with link time reference - /tmp/contents273973045/deps/0/ruby/lib/ruby/2.4.0/x86_64-linux/openssl.so (LoadError)
   from /tmp/rubygems640991822/rubygems-3.0.6/lib/rubygems/core_ext/kernel_require.rb:54:in `require'
   from /tmp/contents273973045/deps/0/ruby/lib/ruby/2.4.0/openssl.rb:13:in `<top (required)>'
   from /tmp/rubygems640991822/rubygems-3.0.6/lib/rubygems/core_ext/kernel_require.rb:54:in `require'
   from /tmp/rubygems640991822/rubygems-3.0.6/lib/rubygems/core_ext/kernel_require.rb:54:in `require'
   from /tmp/rubygems640991822/rubygems-3.0.6/lib/rubygems/security.rb:12:in `<top (required)>'
   from /tmp/rubygems640991822/rubygems-3.0.6/lib/rubygems/core_ext/kernel_require.rb:54:in `require'
   from /tmp/rubygems640991822/rubygems-3.0.6/lib/rubygems/core_ext/kernel_require.rb:54:in `require'
   from /tmp/rubygems640991822/rubygems-3.0.6/lib/rubygems/package.rb:44:in `<top (required)>'
   from /tmp/rubygems640991822/rubygems-3.0.6/lib/rubygems/core_ext/kernel_require.rb:54:in `require'
   from /tmp/rubygems640991822/rubygems-3.0.6/lib/rubygems/core_ext/kernel_require.rb:54:in `require'
   from /tmp/rubygems640991822/rubygems-3.0.6/lib/rubygems/commands/pristine_command.rb:3:in `<top (required)>'
   from /tmp/rubygems640991822/rubygems-3.0.6/lib/rubygems/core_ext/kernel_require.rb:54:in `require'
   from /tmp/rubygems640991822/rubygems-3.0.6/lib/rubygems/core_ext/kernel_require.rb:54:in `require'
   from /tmp/rubygems640991822/rubygems-3.0.6/lib/rubygems/commands/setup_command.rb:617:in `regenerate_binstubs'
   from /tmp/rubygems640991822/rubygems-3.0.6/lib/rubygems/commands/setup_command.rb:176:in `execute'
   from /tmp/rubygems640991822/rubygems-3.0.6/lib/rubygems/command.rb:321:in `invoke_with_build_args'
   from /tmp/rubygems640991822/rubygems-3.0.6/lib/rubygems/command_manager.rb:184:in `process_args'
   from /tmp/rubygems640991822/rubygems-3.0.6/lib/rubygems/command_manager.rb:148:in `run'
   from /tmp/rubygems640991822/rubygems-3.0.6/lib/rubygems/gem_runner.rb:59:in `run'
   from setup.rb:41:in `<main>'
   **ERROR** Bundler 1.17.3 installed
   RubyGems 3.0.6 installed
   **ERROR** Unable to update rubygems: Could not install rubygems: exit status 1
   /tmp/rubygems640991822/rubygems-3.0.6/lib/rubygems/core_ext/kernel_require.rb:54:in `require': /tmp/contents273973045/deps/0/ruby/lib/ruby/2.4.0/x86_64-linux/openssl.so: symbol EC_GROUP_new_curve_GF2m, version OPENSSL_1_1_0 not defined in file libcrypto.so.1.1 with link time reference - /tmp/contents273973045/deps/0/ruby/lib/ruby/2.4.0/x86_64-linux/openssl.so (LoadError)
   from /tmp/rubygems640991822/rubygems-3.0.6/lib/rubygems/core_ext/kernel_require.rb:54:in `require'
   from /tmp/contents273973045/deps/0/ruby/lib/ruby/2.4.0/openssl.rb:13:in `<top (required)>'
   from /tmp/rubygems640991822/rubygems-3.0.6/lib/rubygems/core_ext/kernel_require.rb:54:in `require'
   from /tmp/rubygems640991822/rubygems-3.0.6/lib/rubygems/security.rb:12:in `<top (required)>'
   from /tmp/rubygems640991822/rubygems-3.0.6/lib/rubygems/core_ext/kernel_require.rb:54:in `require'
   from /tmp/rubygems640991822/rubygems-3.0.6/lib/rubygems/core_ext/kernel_require.rb:54:in `require'
   from /tmp/rubygems640991822/rubygems-3.0.6/lib/rubygems/package.rb:44:in `<top (required)>'
   from /tmp/rubygems640991822/rubygems-3.0.6/lib/rubygems/core_ext/kernel_require.rb:54:in `require'
   from /tmp/rubygems640991822/rubygems-3.0.6/lib/rubygems/core_ext/kernel_require.rb:54:in `require'
   from /tmp/rubygems640991822/rubygems-3.0.6/lib/rubygems/commands/pristine_command.rb:3:in `<top (required)>'
   from /tmp/rubygems640991822/rubygems-3.0.6/lib/rubygems/core_ext/kernel_require.rb:54:in `require'
   from /tmp/rubygems640991822/rubygems-3.0.6/lib/rubygems/core_ext/kernel_require.rb:54:in `require'
   from /tmp/rubygems640991822/rubygems-3.0.6/lib/rubygems/commands/setup_command.rb:617:in `regenerate_binstubs'
   from /tmp/rubygems640991822/rubygems-3.0.6/lib/rubygems/commands/setup_command.rb:176:in `execute'
   from /tmp/rubygems640991822/rubygems-3.0.6/lib/rubygems/command.rb:321:in `invoke_with_build_args'
   from /tmp/rubygems640991822/rubygems-3.0.6/lib/rubygems/command_manager.rb:184:in `process_args'
   from /tmp/rubygems640991822/rubygems-3.0.6/lib/rubygems/gem_runner.rb:59:in `run'
   from setup.rb:41:in `<main>'
   **ERROR** Bundler 1.17.3 installed
   RubyGems 3.0.6 installed
   **ERROR** Unable to update rubygems: Could not install rubygems: exit status 1
   2019/12/30 03:56:48 failed to create droplet: Failed to run all supply scripts: exit status 225 - internal error: exit status 15
   2019/12/30 03:56:48 failed to create droplet: Failed to run all supply scripts: exit status 225 - internal error: exit status 15
Error staging application: Staging error: failed to create droplet: Failed to run all supply scripts: exit status 225 - internal error: exit status 15
FAILED

Environment

• SCF version: 2.19.1
• Modules enabled: eirini
• Other specifics

[jimmykarily]

Hi @EmilyEmily thank you for the detailed description of your issue. I don't have an answer to what the problem is but I got some hints that might help you figure it out.

First of all, on scf 2.19.1 when you are deploying with Eirini, the eirinifs you are getting is based on SLE because the upstream ubuntu based one is lacking some binaries needed for cf ssh to work. So depending on what ruby_buildpack is (custom built? built-in?) you could be running an app on a SLE based stack with a buildpack built for Ubuntu.

I also see that the dependencies pulled are those for cflinuxfs3 (e.g. https://buildpacks.cloudfoundry.org/dependencies/ruby/ruby-2.4.9-linux-x64-cflinuxfs3-b8eb2847.tgz) which means somehow the stack is set to cflinuxfs3. Maybe you modified the DEFAULT_STACK during the deployment of scf? (is should have been set to sle15 automatically here: https://github.com/SUSE/scf/blob/develop/container-host-files/etc/scf/config/scripts/override-default-stack.sh#L5)

Regarding eirinifs versions, how did you find out that you are using version 40.0.0 of eirinifs? Even if the sle one wasn't used, on 2.19.1 the eirini-bosh-release we consume, includes the upstream eirinifs 75.0.0. But anyway, unless you tweaked it while deploying scf you should be using that either.

[andreas-kupries]

Hello @EmilyEmily - You mentioned to have an approved PR waiting to be merged. No link to the PR was given however. Where would we find this PR ?

[EmilyEmily]

bits-service-release PR has just been merged: https://github.com/cloudfoundry-incubator/bits-service-release/pull/38.

[EmilyEmily]

@jimmykarily from https://github.com/SUSE/scf/blob/2.19.1/container-host-files/etc/scf/config/role-manifest.yml#L188

- name: "bits-service"
  version: "2.28.0"
  url: "https://bosh.io/d/github.com/cloudfoundry-incubator/bits-service-release?v=2.28.0"

And then from bits-service-release repo, find it's eirinifs version: https://github.com/cloudfoundry-incubator/bits-service-release/blob/2.28.0/helm/bits/values.yaml#L6

rootfs_version: v40.0.0

[jimmykarily]

The eirinifs we use in scf comes from the eirinifs-sle15-release (see here) and is copied to the appropriate place by a job in that release (this one). I don't recall exactly but I think one of the checks in this file is the reason why the upstream eirinifs is never downloaded since our eirinifs is found where it should be. In any case, if in doubt you can check which eirinifs you got but running a simpler app (e.g. with the staticfile buildpack).

You can skip all that and directly try to use the correct buildpack dependencies for sle15 but you need to find out why the ones for cflinuxfs3 are being downloaded (DEFAULT_STACK is wrong?). Maybe try to set the stack using the -s flag? If the testruby app you are using is public can you share it so we can give it a try?

[EmilyEmily]

Many thanks for your good suggestions @jimmykarily . You are right, the root cause is that I didn't use the correct eirinifs. I'm using ubuntu based image instead of sle15 image, it seems the default image doesn't work for ubuntu, after changing the docker image to the official eirini image, pushing ruby app passed.