search

opensearch-project / opensearch-build

🧰 OpenSearch / OpenSearch-Dashboards Build Systems
Apache License 2.0
136 stars 271 forks source link

[Bug]: Cannot release opensearch-ruby 3.2.0 because of Jenkins #4624

Closed nhtruong closed 5 months ago

nhtruong commented 5 months ago

Describe the bug

The release of Ruby 3.2.0 was first triggered in Feb 2024 and failed with the following error:

/var/jenkins/workspace/opensearch-ruby-gems-release@tmp/durable-aa0447d1/script.sh: line 3: /usr/share/opensearch/.rvm/scripts/rvm: No such file or directory
ERROR:  Error installing opensearch-ruby-3.2.0.gem:
    The last version of faraday-net_http (>= 2.0, < 3.2) to support your Ruby & RubyGems was 3.0.2. Try installing it with `gem install faraday-net_http -v 3.0.2` and then running the current command again
    faraday-net_http requires Ruby version >= 3.0.0. The current ruby version is 2.6.0.0.
Successfully installed multi_json-1.15.0
Successfully installed uri-0.13.0
Successfully installed net-http-0.4.1
Gem 'opensearch-ruby' is not installed
WARNING:  faraday-net_http-3.1.0 is not signed
ERROR:  Error installing opensearch-ruby-3.2.0.gem:
    The last version of faraday-net_http (>= 2.0, < 3.2) to support your Ruby & RubyGems was 3.0.2. Try installing it with `gem install faraday-net_http -v 3.0.2` and then running the current command again
    faraday-net_http requires Ruby version >= 3.0.0. The current ruby version is 2.6.0.0.
Gem 'opensearch-ruby' is not installed

We tried to upgrade the rubyVersion of Jenkins to 3.0.6 but Jenkins still failed with the exact error.

We notice in the error message that RVM could not pick up the correct Ruby version:

/var/jenkins/workspace/opensearch-ruby-gems-release@tmp/durable-aa0447d1/script.sh: line 3: /usr/share/opensearch/.rvm/scripts/rvm: No such file or director

To reproduce

Build and install the gem with Ruby 2.6

https://github.com/opensearch-project/opensearch-ruby/issues/238#issuecomment-2048228954

Additional context

Check out the discussion on the Ruby client repo.

gaiksaya commented 5 months ago

The docker image used in the release: opensearchstaging/ci-runner:release-centos7-clients-v4 (see code)

I tried pulling the image and looking at the installed ruby versions. This is the list:

[root@5df978f832bc opensearch]# rvm list
Warning! PATH is not properly set up, /usr/share/opensearch/.gem/bin is not available.
         Usually this is caused by shell initialization files. Search for PATH=... entries.
         You can also re-add RVM to your profile by running: rvm get stable --auto-dotfiles
         To fix it temporarily in this shell session run: rvm use .gem
         To ignore this error add rvm_silence_path_mismatch_check_flag=1 to your ~/.rvmrc file.
   jruby-9.3.0.0 [ x86_64 ]
 * ruby-2.6.0 [ x86_64 ]

# => - current
# =* - current && default
#  * - default

Looks like these are only versions installed on the docker file: https://github.com/opensearch-project/opensearch-build/blob/main/docker/ci/dockerfiles/current/release.centos7.clients.x64.arm64.dockerfile#L138-L141

@peterzhuamazon I believe for additional versions we either need to update the image or have something generic that would install it on the go?

Earlopain commented 5 months ago

That seems to be the way to go, yes. Can you try running /usr/share/opensearch/.rvm/scripts/rvm instead? That is the command actually being executed at the moment. I believe this path wasn't updated when https://github.com/opensearch-project/opensearch-build/commit/01613af16774cfe39bf41ab23bf2fdf27a78b95f was done.

So there needs to be made a change at https://github.com/opensearch-project/opensearch-build-libraries/blob/6d35e942e98e1acddd88633a2aa1bbe39ab5e002/vars/publishToRubyGems.groovy#L27 as well. Do these things actually run as root? My first instict is that they would run as whatever CONTAINER_USER is set to, so the command could just be ~/.rvm/scripts/rvm I think.

https://build.ci.opensearch.org/blue/organizations/jenkins/opensearch-ruby-gems-release/detail/opensearch-ruby-gems-release/29/pipeline/ which was run before that commit about changing the user doesn't show the rvm error. I've openend https://github.com/opensearch-project/opensearch-build-libraries/issues/410

So, two issues. A more recent ruby version needs to be added to the image anyways.

gaiksaya commented 5 months ago

This is what I see when I use above command:

[opensearch@0628d16dc11d ~]$ rvm list
Warning! PATH is not properly set up, /usr/share/opensearch/.gem/bin is not available.
         Usually this is caused by shell initialization files. Search for PATH=... entries.
         You can also re-add RVM to your profile by running: rvm get stable --auto-dotfiles
         To fix it temporarily in this shell session run: rvm use .gem
         To ignore this error add rvm_silence_path_mismatch_check_flag=1 to your ~/.rvmrc file.
   jruby-9.3.0.0 [ x86_64 ]
 * ruby-2.6.0 [ x86_64 ]

# => - current
# =* - current && default
#  * - default

[opensearch@0628d16dc11d ~]$ /usr/share/opensearch/.rvm/scripts/rvm list
[opensearch@0628d16dc11d ~]$ which rvm
~/.rvm/bin/rvm

Note that I am logged in as opensearch user in the docker image.

What version of ruby would need to be installed?

Earlopain commented 5 months ago

The release action currently uses 3.1, mirroring that would be ideal I think. See https://github.com/opensearch-project/opensearch-ruby/blob/bfd43cf44775981d719ebfa00fb420560c0d2dac/.github/workflows/release_drafter.yml#L29-L32

The latest patch release at this time is 3.1.4.

peterzhuamazon commented 5 months ago

I would need to update the ruby from 2.6.0 to 3.0.2+, need to check the rvm support and rebuild images with changes.

peterzhuamazon commented 5 months ago 

[ci-runner@88850ad06f30 ~]$ ruby --version
ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) [x86_64-linux]
[ci-runner@88850ad06f30 ~]$ rvm list
Warning! PATH is not properly set up, /home/ci-runner/.gem/bin is not available.
         Usually this is caused by shell initialization files. Search for PATH=... entries.
         You can also re-add RVM to your profile by running: rvm get stable --auto-dotfiles
         To fix it temporarily in this shell session run: rvm use .gem
         To ignore this error add rvm_silence_path_mismatch_check_flag=1 to your ~/.rvmrc file.
   jruby-9.3.0.0 [ x86_64 ]
 * ruby-3.1.2 [ x86_64 ]

# => - current
# =* - current && default
#  * - default

Send PR soon.

peterzhuamazon commented 5 months ago

Able to build:


[ci-runner@88850ad06f30 opensearch-ruby]$ gem build opensearch-ruby.gemspec
WARNING:  open-ended dependency on multi_json (>= 1.0) is not recommended
  if multi_json is semantically versioned, use:
    add_runtime_dependency 'multi_json', '~> 1.0'
WARNING:  See https://guides.rubygems.org/specification-reference/ for help
  Successfully built RubyGem
  Name: opensearch-ruby
  Version: 3.2.0
  File: opensearch-ruby-3.2.0.gem
peterzhuamazon commented 5 months ago

Hi @nhtruong please confirm if ruby version 3.1.2 is good for you. I try to build opensearch-rb main branch with that and it seems fine.

Thanks.

peterzhuamazon commented 5 months ago

The above PR merged and will have the image available in 5 hours or so due to the complexity of this image.

We can try to release next week.

Thanks.

peterzhuamazon commented 5 months ago

The opensearch-ruby workflow need to switch to 3.1.2 ruby before run the build. @gaiksaya to apply the corresponding changes as it defaults to 2.6.0 still.

Thanks.

Earlopain commented 5 months ago

That sound good, thank you for your work. Is https://github.com/opensearch-project/opensearch-build-libraries/issues/410 not an issue for the release?

Earlopain commented 5 months ago

I have openend PR https://github.com/opensearch-project/opensearch-build-libraries/pull/413 for the above issue.

nhtruong commented 5 months ago

Attempted to redo the release but still failed. With different errors this time: https://build.ci.opensearch.org/blue/organizations/jenkins/opensearch-ruby-gems-release/detail/opensearch-ruby-gems-release/33/pipeline

gaiksaya commented 5 months ago

Hi @nhtruong We are yet to make changes to your jenkinsFile. See attached PR above, that is yet to be merged and then another PR needs to be opened in ruby repo. If this is urgent we can merge above PR and additional changes I can create another one.

nhtruong commented 5 months ago

We got past the previous error but got another one: https://build.ci.opensearch.org/blue/organizations/jenkins/opensearch-ruby-gems-release/detail/opensearch-ruby-gems-release/34/pipeline

ERROR:  While executing gem ... (Gem::Security::Exception)
    root cert /CN=opensearch/DC=amazon/DC=com is not trusted
Gem 'opensearch-ruby' is not installed
Earlopain commented 5 months ago

Maybe the gem cert --add in opensearch-build has to happen after switching versions? I'm not able to test myself at the moment but the fact that this happens so early stands out to me.

gaiksaya commented 5 months ago

I think you are right @Earlopain I just tried reproducing the error. However forgot to use default ruby (2.6.0) to add cert. So basically did what you suggested:

docker run -u root -it opensearchstaging/ci-runner:release-centos7-clients-v4 /bin/bash
[root@e314dcfc5515 ci-runner]# git clone https://github.com/opensearch-project/opensearch-ruby.git
Cloning into 'opensearch-ruby'...
remote: Enumerating objects: 29553, done.
remote: Counting objects: 100% (932/932), done.
remote: Compressing objects: 100% (364/364), done.
remote: Total 29553 (delta 628), reused 793 (delta 553), pack-reused 28621
Receiving objects: 100% (29553/29553), 6.66 MiB | 3.08 MiB/s, done.
Resolving deltas: 100% (20467/20467), done.
[root@e314dcfc5515 ci-runner]# cd opensearch-ruby/
[root@e314dcfc5515 opensearch-ruby]# source /home/ci-runner/.rvm/scripts/rvm
[root@e314dcfc5515 opensearch-ruby]# rvm use 3.1.2
Using /home/ci-runner/.rvm/gems/ruby-3.1.2
[root@e314dcfc5515 opensearch-ruby]# ruby --version
ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) [x86_64-linux]
[root@e314dcfc5515 opensearch-ruby]# gem cert --add .github/opensearch-rubygems.pem
Added '/CN=opensearch/DC=amazon/DC=com'
[root@e314dcfc5515 opensearch-ruby]# curl -J -L -H 'Accept: application/octet-stream' -H 'Authorization: Bearer <token>' https://api.github.com/repos/opensearch-project/opensearch-ruby/releases/assets/162445549 -o artifacts.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  140k  100  140k    0     0   109k      0  0:00:01  0:00:01 --:--:-- 3783k
[root@e314dcfc5515 opensearch-ruby]# ls
ADMINS.md     CODE_OF_CONDUCT.md  CONTRIBUTING.md     Gemfile      MAINTAINERS.md  NOTICE.txt      README.md     Rakefile     UPGRADING.md   api_generator     bin     jenkins  opensearch-ruby.gemspec  samples  test
CHANGELOG.md  COMPATIBILITY.md    DEVELOPER_GUIDE.md  LICENSE.txt  Makefile        OpenSearch.svg  RELEASING.md  SECURITY.md  USER_GUIDE.md  artifacts.tar.gz  guides  lib      profile                  spec
[root@e314dcfc5515 opensearch-ruby]# tar -xvf artifacts.tar.gz
dist/
dist/opensearch-ruby-3.2.0.gem
[root@e314dcfc5515 opensearch-ruby]# cd dist/
[root@e314dcfc5515 dist]# gemNameWithVersion=$(ls *.gem)
[root@e314dcfc5515 dist]# echo $gemNameWithVersion
opensearch-ruby-3.2.0.gem
[root@e314dcfc5515 dist]# gem install $gemNameWithVersion
Fetching faraday-2.9.0.gem
Fetching multi_json-1.15.0.gem
Fetching faraday-net_http-3.1.0.gem
Successfully installed multi_json-1.15.0
Successfully installed faraday-net_http-3.1.0
Successfully installed faraday-2.9.0
Successfully installed opensearch-ruby-3.2.0
Parsing documentation for multi_json-1.15.0
Installing ri documentation for multi_json-1.15.0
Parsing documentation for faraday-net_http-3.1.0
Installing ri documentation for faraday-net_http-3.1.0
Parsing documentation for faraday-2.9.0
Installing ri documentation for faraday-2.9.0
Parsing documentation for opensearch-ruby-3.2.0
Installing ri documentation for opensearch-ruby-3.2.0
Done installing documentation for multi_json, faraday-net_http, faraday, opensearch-ruby after 26 seconds
4 gems installed
[root@e314dcfc5515 dist]# gemName=$(echo $gemNameWithVersion | sed -E 's/(-[0-9.]+-*[a-z]*.gem$)/g')
sed: -e expression #1, char 27: unterminated `s' command
[root@e314dcfc5515 dist]# gemName=$(echo $gemNameWithVersion | sed -E 's/(-[0-9.]+-*[a-z]*.gem$)//g')
[root@e314dcfc5515 dist]# echo $gemName
opensearch-ruby
[root@e314dcfc5515 dist]# gem uninstall $gemName
Remove executables:
    opensearch_ruby_console

in addition to the gem? [Yn]  Y
Removing opensearch_ruby_console
Successfully uninstalled opensearch-ruby-3.2.0
[root@e314dcfc5515 dist]# gem install $gemNameWithVersion -P MediumSecurity
Successfully installed opensearch-ruby-3.2.0
Parsing documentation for opensearch-ruby-3.2.0
Installing ri documentation for opensearch-ruby-3.2.0
Done installing documentation for opensearch-ruby after 23 seconds
1 gem installed
[root@e314dcfc5515 dist]# gem uninstall $gemName
Remove executables:
    opensearch_ruby_console

in addition to the gem? [Yn]  Y
Removing opensearch_ruby_console
Successfully uninstalled opensearch-ruby-3.2.0
[root@e314dcfc5515 dist]# gem install $gemNameWithVersion -P HighSecurity
Successfully installed opensearch-ruby-3.2.0
Parsing documentation for opensearch-ruby-3.2.0
Installing ri documentation for opensearch-ruby-3.2.0
Done installing documentation for opensearch-ruby after 32 seconds
1 gem installed

Will check if exact steps in the code to see if that is the issue.

gaiksaya commented 5 months ago

And our theory is right. Ruby version needs to be switched before the cert is added. Did not think that would make a difference

docker run -u root -it opensearchstaging/ci-runner:release-centos7-clients-v4 /bin/bash
[root@9d94be570f33 ci-runner]# git clone https://github.com/opensearch-project/opensearch-ruby.git
Cloning into 'opensearch-ruby'...
remote: Enumerating objects: 29553, done.
remote: Counting objects: 100% (932/932), done.
remote: Compressing objects: 100% (364/364), done.
remote: Total 29553 (delta 628), reused 793 (delta 553), pack-reused 28621
Receiving objects: 100% (29553/29553), 6.66 MiB | 5.72 MiB/s, done.
Resolving deltas: 100% (20467/20467), done.
[root@9d94be570f33 ci-runner]# cd opensearch-ruby/
[root@9d94be570f33 opensearch-ruby]# gem cert --add .github/opensearch-rubygems.pem
Added '/CN=opensearch/DC=amazon/DC=com'
[root@9d94be570f33 opensearch-ruby]# source /home/ci-runner/.rvm/scripts/rvm
[root@9d94be570f33 opensearch-ruby]# rvm use 3.1.2
Using /home/ci-runner/.rvm/gems/ruby-3.1.2
[root@9d94be570f33 opensearch-ruby]# ruby --version
ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) [x86_64-linux]
[root@9d94be570f33 opensearch-ruby]# curl -J -L -H 'Accept: application/octet-stream' -H 'Authorization: Bearer <token>' https://api.github.com/repos/opensearch-project/opensearch-ruby/releases/assets/162445549 -o artifacts.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  140k  100  140k    0     0   131k      0  0:00:01  0:00:01 --:--:--  131k
[root@9d94be570f33 opensearch-ruby]# tar -xvf artifacts.tar.gz
dist/
dist/opensearch-ruby-3.2.0.gem
[root@9d94be570f33 opensearch-ruby]# cd dist/
[root@9d94be570f33 dist]# gemNameWithVersion=$(ls *.gem)
[root@9d94be570f33 dist]# gem install $gemNameWithVersion
Fetching faraday-2.9.0.gem
Fetching faraday-net_http-3.1.0.gem
Fetching multi_json-1.15.0.gem
Successfully installed multi_json-1.15.0
Successfully installed faraday-net_http-3.1.0
Successfully installed faraday-2.9.0
Successfully installed opensearch-ruby-3.2.0
Parsing documentation for multi_json-1.15.0
Installing ri documentation for multi_json-1.15.0
Parsing documentation for faraday-net_http-3.1.0
Installing ri documentation for faraday-net_http-3.1.0
Parsing documentation for faraday-2.9.0
Installing ri documentation for faraday-2.9.0
Parsing documentation for opensearch-ruby-3.2.0
Installing ri documentation for opensearch-ruby-3.2.0
Done installing documentation for multi_json, faraday-net_http, faraday, opensearch-ruby after 22 seconds
4 gems installed
[root@9d94be570f33 dist]# gemName=$(echo $gemNameWithVersion | sed -E 's/(-[0-9.]+-*[a-z]*.gem$)/g')
sed: -e expression #1, char 27: unterminated `s' command
[root@9d94be570f33 dist]# gemName=$(echo $gemNameWithVersion | sed -E 's/(-[0-9.]+-*[a-z]*.gem$)//g')
[root@9d94be570f33 dist]# gem uninstall $gemName
Remove executables:
    opensearch_ruby_console

in addition to the gem? [Yn]  Y
Removing opensearch_ruby_console
Successfully uninstalled opensearch-ruby-3.2.0
[root@9d94be570f33 dist]# gem install $gemNameWithVersion -P MediumSecurity
ERROR:  While executing gem ... (Gem::Security::Exception)
    root cert /CN=opensearch/DC=amazon/DC=com is not trusted

Will create a PR to fix this.

peterzhuamazon commented 5 months ago

It is released: https://build.ci.opensearch.org/job/opensearch-ruby-gems-release/35/console

gaiksaya commented 5 months ago

Closing it as the issue is resolved now!