Build and install the gem during the release workflow

[Earlopain]

What are you proposing?

The release pipeline broke because the jenkins workflow used Ruby 2.6 by default. Most workflows in the repo use a more modern version like 3.1.

The release workflow should try to build and install the gem before asking for approval so that failures are quick to be noticed. In addition, jenkins should use the same ruby version as the github action. I bumped it to 3.0 since that was the first to work but it should have been 3.1. At the minimum it requires a comment to keep it in sync.

[nhtruong]

Thanks, @Earlopain. Are you also working on a fix?

[Earlopain]

I was kinda betting on #234 working and would have done the changes I explained here, yes.

However with the jenkins pipeline still using 2.6 (https://build.ci.opensearch.org/blue/organizations/jenkins/opensearch-ruby-gems-release/detail/opensearch-ruby-gems-release/32/pipeline/) I'm at a bit of a loss here. I checked if maybe the version used here doesn't support this argument but as far as I can tell 5.7.1 is more than enough.

I have never used jenkins, I'm much more familiar with github/gitlab CI. I don't think I will be able to figure this one out myself.

[nhtruong]

I could install faraday-net_http with ruby 2.6 locally:

bundle install
bundle info faraday-net_http

  * faraday-net_http (2.1.0)
        Summary: Faraday adapter for Net::HTTP
        Homepage: https://github.com/lostisland/faraday-net_http
        Source Code: https://github.com/lostisland/faraday-net_http
        Changelog: https://github.com/lostisland/faraday-net_http/releases/tag/v2.1.0
        Path: /Users/theotr/.rbenv/versions/2.6.10/lib/ruby/gems/2.6.0/gems/faraday-net_http-2.1.0
        Reverse Dependencies: 
                faraday (2.8.1) depends on faraday-net_http (>= 2.0, < 3.1)
                faraday-net_http_persistent (2.0.2) depends on faraday-net_http (< 3)

[Earlopain]

Yeah, that works fine. I can reproduce this problem by first doing gem build and then gem install with the created file:

$ ruby --version
ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-linux]
$ gem build
WARNING:  open-ended dependency on multi_json (>= 1.0) is not recommended
  if multi_json is semantically versioned, use:
    add_runtime_dependency "multi_json", "~> 1.0"
WARNING:  See https://guides.rubygems.org/specification-reference/ for help
  Successfully built RubyGem
  Name: opensearch-ruby
  Version: 3.2.0
  File: opensearch-ruby-3.2.0.gem
$ gem install opensearch-ruby-3.2.0.gem 
Successfully installed opensearch-ruby-3.2.0
Parsing documentation for opensearch-ruby-3.2.0
Installing ri documentation for opensearch-ruby-3.2.0
Done installing documentation for opensearch-ruby after 5 seconds
1 gem installed

$ ruby --version
ruby 2.6.10p210 (2022-04-12 revision 67958) [x86_64-linux]
$ gem build opensearch-ruby.gemspec 
WARNING:  open-ended dependency on multi_json (>= 1.0) is not recommended
  if multi_json is semantically versioned, use:
    add_runtime_dependency 'multi_json', '~> 1.0'
WARNING:  See http://guides.rubygems.org/specification-reference/ for help
  Successfully built RubyGem
  Name: opensearch-ruby
  Version: 3.2.0
  File: opensearch-ruby-3.2.0.gem
$ gem install opensearch-ruby-3.2.0.gem 
Fetching uri-0.13.0.gem
Fetching multi_json-1.15.0.gem
Fetching net-http-0.4.1.gem
Fetching faraday-2.9.0.gem
Fetching faraday-net_http-3.1.0.gem
Successfully installed multi_json-1.15.0
Successfully installed uri-0.13.0
Successfully installed net-http-0.4.1
ERROR:  Error installing opensearch-ruby-3.2.0.gem:
        The last version of faraday-net_http (>= 2.0, < 3.2) to support your Ruby & RubyGems was 3.0.2. Try installing it with `gem install faraday-net_http -v 3.0.2` and then running the current command again
        faraday-net_http requires Ruby version >= 3.0.0. The current ruby version is 2.6.10.210.

As you can see, for ruby 2.6 it is trying to install the latest faraday version (currently 2.9.0) which has a minimum required ruby version of 3.0. I do not know why it is trying to pull in that version.

There has been a bug at one point that makes bundle install ignore required_ruby_version from gemspec, perhaps ruby 2.6.0 ships a version that suffers from this? https://github.com/rubygems/rubygems/issues/4263

This does coincide with the general timeline. faraday 2.8.1 still supported ruby 2.6, faraday 2.9.0 was released only after the last successful release.

[nhtruong]

somehow works for me

gem build
WARNING:  open-ended dependency on multi_json (>= 1.0) is not recommended
  if multi_json is semantically versioned, use:
    add_runtime_dependency 'multi_json', '~> 1.0'
WARNING:  See https://guides.rubygems.org/specification-reference/ for help
  Successfully built RubyGem
  Name: opensearch-ruby
  Version: 3.2.0
  File: opensearch-ruby-3.2.0.gem

gem install ./opensearch-ruby-3.2.0.gem 
Successfully installed opensearch-ruby-3.2.0
Parsing documentation for opensearch-ruby-3.2.0
Installing ri documentation for opensearch-ruby-3.2.0
Done installing documentation for opensearch-ruby after 2 seconds
1 gem installed
theotr@88665a51fc3f opensearch-ruby % 

[nhtruong]

Tho Jenkins is using 2.6.0 while I'm using 2.6.10

[Earlopain]

Nah, I don't think that is it. I was also using 2.6.10 in my command above, with rbenv as well, same as you

[nhtruong]

This could be why https://github.com/opensearch-project/opensearch-ruby/blob/main/.github/workflows/release_drafter.yml#L32

It was built with Ruby 3.1. But it has always been like that 🤷

[Earlopain]

I don't think that would be revelant. A build gem doesn't contain resolved dependency versions, it just has the version ranges from the gemspec.

I noticed you were able to gem build without providing a gemspec path. Doing that on ruby 2.6 is an error for me, only on more recent rubies is the default rubygems able to infer the path. gem --version prints 3.0.3.1 for me, does that look similar?

[nhtruong]

% ruby -v
ruby 2.6.10p210 (2022-04-12 revision 67958) [x86_64-darwin22]
% gem -v
3.2.3

[Earlopain]

Yeah, so I think this is pretty much about the issue I linked above (https://github.com/rubygems/rubygems/issues/4263, and its linked issue https://github.com/rubygems/bundler/issues/2721)

The warning being shown by bundler the following:

Bundler.ui.warn \
    "Your RubyGems version (#{Gem::VERSION}) has a bug that prevents " \
    "`required_ruby_version` from working for Bundler. Any scripts that use " \
    "`gem install bundler` will break as soon as Bundler drops support for " \
    "your Ruby version. Please upgrade RubyGems to avoid future breakage " \
    "and silence this warning by running `gem update --system 3.2.3`"

So I think you just ran that at some point since the versions match exactly. I installed ruby 2.6 just now again and never did that.

Still I do not know why 2.6 is stil being used by jenkins.

[nhtruong]

Another thing to note is that Jenkins ran successfully last time with ruby 2.6.0. in Dec 2023

https://rubygems.org/gems/faraday-net_http/versions/3.1.0 was released Jan 2024

[Earlopain]

Yeah, I touched on that earlier with the faraday releases. At that time Dec 12 2023 the latest faraday version was 2.7.12 which had a ruby version requirement of >= 2.6. In faraday 2.9.0 from Jan 09 2024 this was bumped to >= 3.0 which is where I imagine most of the problems are now coming from (in combination with the broken rubygems version).

Similar things apply to faraday-net_http, with a release bumping the minimum ruby version between the last successful pipeline run and now. Don't know why I went to faraday in my example, the error clearly surfaces in faraday-net_http first.

[nhtruong]

Thanks for all of this work, @Earlopain. As of right now the opensearch-ruby client still supports Ruby 2. So I think locking faraday-net_http <= 3.0.2 is the better option. What do you think?

[Earlopain]

That is not necessary and blocks people from upgrading their own dependencies. There is no problem with how the gem itself dependens on faraday/faraday-net_http. If you run Ruby 2.5 it will simply pick the latest versions that match these constraint, that doesn't have to correspond to the latest version. For ruby 2.5, it would pick faraday 1.10.3 and resolve dependencies in a way to make that work.

I do think I have found the root cause for jenkins not picking the correct ruby version. It's quite simple and I'm a bit bad mad I haven't noticed it sooner:

/var/jenkins/workspace/opensearch-ruby-gems-release@tmp/durable-aa0447d1/script.sh: line 3: /usr/share/opensearch/.rvm/scripts/rvm: No such file or directory

It looks like rvm is just not available there

[nhtruong]

I'm not well versed in rvm. How did we get this error then if the current ruby version is not 2.6.0?

faraday-net_http requires Ruby version >= 3.0.0. The current ruby version is 2.6.0.0.

[nhtruong]

Previously successful run also didn't have rvm either https://build.ci.opensearch.org/blue/organizations/jenkins/opensearch-ruby-gems-release/detail/opensearch-ruby-gems-release/30/pipeline

[Earlopain]

I'm not well versed in rvm. How did we get this error then if the current ruby version is not 2.6.0?

I'm not quite sure what you mean here. The error message looks right to me and what I would expect in this situation.

Previously successful run also didn't have rvm either

Yes, previous run has no rvm, ruby 2.6 must be what is available on jenkins by default The script has no pipefail so it just ignores the failed command and continues on. But when wanting to use something else it must be there, otherwise we'll find us in the situation we are in currently.

Here is a workflow that presumably works: https://github.com/opensearch-project/logstash-output-opensearch/blob/d60a58f382ac3c772cd70ad52c79aecb0d085d25/jenkins/release.jenkinsFile

The only thing I can think of that might make a differencfe is the overrideDockerImage field. I have no idea what this docker image contains though, which one would be needed here, or if that is even relevant.

[nhtruong]

Thanks @Earlopain. I've created issue in opensearch-project/opensearch-build. Someone from the build team will give us a hand. So, to clarify, only Ruby 2.6 has this issue (because Ruby 2.5 works just fine)? And if we can get RVM to pick the correct rubyVersion defined in the Jenkins file, which is Ruby 3.0.2, this will likely be resolved?

[Earlopain]

I'm not sure in which version of rubygems this was resolved with, I skimmed the changelog but didn't really find anything. Ruby 2.5 might contain a rubygems version without this error, or it might. I don't really know. It's easy to check but also not really relevant.

The version of Ruby a gem is build with is not important. A gem may support rubies from 2.0 all the way to 3.3 and you could choose any version inbetween. It is really just very unfortunate that 2.6 contains this bug. Even just running gem update --system might resolve this issue, though it would still use 2.6 of course.

The important thing here is that a ruby environment without this rubygems error is available. You might even be able to use the jruby version I found earlier? Though I don't know anything about that, a "normal" ruby would certainly be more welcome to me at least. The latest patch version of 3.0 definitly works.

I'll follow that issue you openend and chime in if I've got something useful to add.

[Earlopain]

I'll go ahead and close this, now that a new version has successfully been released. After more understanding I don't think there's much value in doing what I initially proposed in this issue.

Thanks @nhtruong for all the conversation around this, got me learning about a bunch of new things (:

[nhtruong]

Thank you @Earlopain for all your help and I've learned quite a few things myself too. Also I don't see an email on your profile. Are you happen to be on our slack workspace?

[Earlopain]

I'm not really in the slack, no. I didn't even know that was a thing until you pointed that out to me just now. Are you asking because of something specific or just in general?

[nhtruong]

@Earlopain Hi sorry it took some time to go through the process to nominate a new maintainer. Please shoot me an email to theotr@amazon.com so that I can send you the invite :)

[Earlopain]

Ah ok, I see. I'll hit you up tomorrow