search

rubygems / rubygems

Library packaging and distribution for Ruby.
https://rubygems.org/
Other
3.64k stars 1.74k forks source link

'gem install' fails to find some gems on RubyGems 3.5.19 #8042

Open roberts1000 opened 2 days ago

roberts1000 commented 2 days ago

Describe the problem as clearly as you can

I have RubyGems configured to to use Nexus to serve private gems, It's also setup to act as caching proxy to rubygems.org. After updating to RubyGems 3.5.19 and Bundler 2.5.19, gem install fails to find some gems, and throws an error. gem install works consistently if I revert back to 3.5.18 and 2.5.18.

Did you try upgrading RubyGems?

Yes, the latest version causes the issue.

Post steps to reproduce the problem

I have some infrastructure that first runs:

gem update --system

Even though Bundler 2.5.19 is already installed after the above step runs, my infrastructure has a legacy step that runs:

gem install bundler -v=2.5.19

which initially caused the following error:

$ gem install bundler -v=2.5.19
ERROR: Could not find a valid gem 'bundler' (=2.5.19) in any repository
ERROR: Possible alternatives: "...."

I went into Nexus and invalidated the cache, and that got gem install bundler -v=2.5.19 working, however, I have similar issues when I attempt to install other gems. If I use the --verbose flag, I sometimes see very different output:

On RubyGems 3.5.18:

$ gem install some_gem -v=1.0.0
HEAD https://my-nexus-host/nexus/repository/all-gems-group/
400 Bad Request
GET https://my-nexus-host/nexus/repository/all-gems-group/prerelease_specs.4.8.gz
200 OK
GET https://my-nexus-host/nexus/repository/all-gems-group/specs.4.8.gz
200 OK
GET https://my-nexus-host/nexus/repository/all-gems-group/quick/Marshal.4.8/some_gem-1.0.0.gemspec.rz
200 OK
Download gem some_gem-1.0.0.gem
....(it goes on to download and install correctly)...

On RubyGems 3.5.19:

$ gem install some_gem -v=1.0.0
HEAD https://my-nexus-host/nexus/repository/all-gems-group/versions
200 OK
GET GET https://my-nexus-host/nexus/repository/all-gems-group/info/some_gem
404 Not Found
ERROR: Could not find a valid gem 'some_gem' (=1.0.0) in any repository
....

The RubyGems 2.5.19 output isn't always the same as above. When gem install works, it's looks more like the 3.5.18 output.

gem install doesn't fail on every gem.

Which command did you run?

gem install some_gem -v=x.y.z`

or

gem install some_gem

What were you expecting to happen?

See above.

What actually happened?

See above.

Run gem env and paste the output below

$ gem env
RubyGems Environment:
  - RUBYGEMS VERSION: 3.5.19
  - RUBY VERSION: 3.3.5 (2024-09-03 patchlevel 100) [x86_64-linux]
  - INSTALLATION DIRECTORY: /home/dev/.rvm/gems/ruby-3.3.5
  - USER INSTALLATION DIRECTORY: /home/dev/.gem/ruby/3.3.0
  - RUBY EXECUTABLE: /home/dev/.rvm/rubies/ruby-3.3.5/bin/ruby
  - GIT EXECUTABLE: /usr/bin/git
  - EXECUTABLE DIRECTORY: /home/dev/.rvm/gems/ruby-3.3.5/bin
  - SPEC CACHE DIRECTORY: /home/dev/.cache/gem/specs
  - SYSTEM CONFIGURATION DIRECTORY: /home/dev/.rvm/rubies/ruby-3.3.5/etc
  - RUBYGEMS PLATFORMS:
     - ruby
     - x86_64-linux
  - GEM PATHS:
     - /home/dev/.rvm/gems/ruby-3.3.5
     - /home/dev/.rvm/rubies/ruby-3.3.5/lib/ruby/gems/3.3.0
  - GEM CONFIGURATION:
     - :update_sources => true
     - :verbose => true
     - :backtrace => false
     - :bulk_threshold => 1000
     - :sources => ["https://my-nexus-host/nexus/repository/all-gems-group/"]
     - "gem" => "--no-document"
  - REMOTE SOURCES:
     - https://my-nexus-host/nexus/repository/all-gems-group/
  - SHELL PATH:
     - /home/dev/.rvm/gems/ruby-3.3.5/bin
     - /home/dev/.rvm/gems/ruby-3.3.5@global/bin
     - /home/dev/.rvm/rubies/ruby-3.3.5/bin
     - /home/dev/.rvm/bin
     - /home/dev/.nvm/versions/node/v20.15.0/bin
     - /home/dev/.gitbin
     - /usr/local/sbin
     - /usr/local/bin
     - /usr/sbin
     - /usr/bin
     - /sbin
     - /bin

The env output when using 3.5.18 is the exact same, except for the expected version number changes.

deivid-rodriguez commented 1 day ago

Hello! This is definitely caused by #8006.

We faced similar issues in Bundler when the dependency API was deprecated, as explained in this blog post: https://blog.rubygems.org/2023/04/07/dependency-api-deprecation-delayed.html

It turns out that Artifactory and Nexus (at least sometimes) get a 404 Not Found response from RubyGems.org, but then return a 200 OK response to Bundler or RubyGems. Unfortunately, that 200 OK response prevents Bundler from falling back to the full index, and breaks installing gems.

Well, RubyGems now has the same "problem", since we applied the same strategy used by Bundler.

According to sonatype release notes, this was fixed in Sonatype Nexus Repository 3.53.1.

Can you confirm which version your nexus host is running? Is it possible to upgrade it to >= 3.53.1? I'm also curious, can you check the contents of the https://my-nexus-host/nexus/repository/all-gems-group/versions response? Depending on how it looks, we may be able to provide some workaround.

roberts1000 commented 1 day ago

Thanks for taking a look.

  1. I'm using Nexus 3.68.1-02. (Nexus 3.72.0 is currently the latest, but I don't see any mention of Ruby in the release notes, between the two versions, so the latest Nexus probably doesn't contain anything that would help this issue.)

  2. The contents of the versions response looks like this:

    created_at 2024-09-19T14:32:00.478Z
---
- 1 05d011...
-A 0.0.0 8b1527...
.cat 0.0.1 631fd...
.omghi 1,2 7a67c0...
0mq 0.1.0,0.1.1,0.1.2,0.2.0,0.2.1,0.3.0,0.4.0,04.4.1,0.5.0,0.5.1,0.5.2,0.5.3 61561....
.... (many more gem entries follow; they seem to follow the above structure) ...

    There's about 190,000 lines in the file.

  3. A colleague is sending a support request to Nexus to have them take a look at a long term solution.

deivid-rodriguez commented 1 day ago

Interesting, that makes this more intriguing!

Does that file include a line for bundler? If the /versions endpoint responds fine info/<gem> for a gem included in the versions file should also respond succesfully.

roberts1000 commented 1 day ago

It does have a line for bundler, but 2.5.19 is missing from the list of versions. Here's the bundler line:

bundler 0.3.1,0.3.0,0.4.0,0.4.1,0.5.0,0.6.0,0.7.0,0.7.1,0.7.2,0.7.3.pre,0.7.3.pre2,0.8.0,0.8.1,0.9.0.pre1,0.9.0.pre2,0.9.0.pre3,0.9.0.pre4,0.9.0.pre5,0.9.0,0.9.1.pre1,0.9.1,0.9.2,0.9.3,0.9.4,0.9.5,0.9.6,0.9.7,0.9.8,0.9.9,0.9.10,0.9.11,0.9.12,0.9.13,0.9.14,0.9.15,0.9.16,0.9.17,0.9.18,0.9.19,0.9.20,0.9.21,0.9.22,0.9.23,0.9.24,0.9.25,0.9.26,1.0.0.beta.1,1.0.0.beta.2,1.0.0.beta.3,1.0.0.beta.4,1.0.0.beta.5,1.0.0.beta.8,1.0.0.beta.9,1.0.0.beta.10,1.0.0.rc.1,1.0.0.rc.2,1.0.0.rc.3,1.0.0.rc.5,1.0.0.rc.6,1.0.0,1.0.2,1.0.3,1.0.5,1.0.7,1.0.9,1.1.pre,1.0.10,1.1.pre.1,1.0.11,1.0.12,1.1.pre.2,1.1.pre.3,1.0.13,1.1.pre.4,1.0.14,1.0.15,1.1.pre.5,1.1.pre.7,1.0.17,1.1.pre.8,1.0.18,1.0.19.rc,1.0.20.rc,1.1.pre.9,1.1.pre.10,1.0.20,1.0.21.rc,1.0.21,1.1.rc,1.1.rc.2,1.1.rc.3,1.1.rc.5,1.1.rc.6,1.1.rc.7,1.0.22,1.1.rc.8,1.1.0,1.1.1,1.1.2,1.1.3,1.2.0.pre,1.1.4,1.2.0.pre.1,1.1.5,1.2.0.rc,1.2.0.rc.2,1.2.0,1.2.1,1.2.2,1.2.3,1.3.0.pre,1.3.0.pre.2,1.3.0.pre.3,1.3.0.pre.4,1.3.0.pre.5,1.3.0.pre.6,1.3.0.pre.7,1.2.4,1.3.0.pre.8,1.2.5,1.3.0,1.3.1,1.3.2,1.3.3,1.3.4,1.3.5,1.4.0.pre.1,1.4.0.pre.2,1.4.0.rc.1,1.5.0.rc.1,1.5.0.rc.2,1.5.0,1.5.1,1.3.6,1.5.2,1.6.0.pre.1,1.5.3,1.6.0.pre.2,1.6.0.rc,1.6.0.rc2,1.6.0,1.6.1,1.6.2,1.6.3,1.6.4,1.6.5,1.7.0,1.7.1.pre,1.7.1.pre.2,1.7.1.pre.3,1.7.1,1.7.2,1.6.6,1.7.3,1.6.7,1.7.4,1.6.8,1.7.5,1.6.9,1.7.6,1.7.7,1.7.8,1.7.9,1.7.10,1.7.11,1.7.12,1.8.0.pre,1.7.13,1.8.0,1.8.1,1.8.2,1.8.3,1.8.4,1.8.5,1.9.0.pre,1.9.0.rc,1.9.0.pre.1,1.9.0,1.9.1,1.7.14,1.8.6,1.9.2,1.8.7,1.9.3,1.9.4,1.7.15,1.8.8,1.9.5,1.8.9,1.9.6,1.10.0.pre,1.10.0.pre.1,1.10.0.pre.2,1.9.7,1.9.8,1.9.9,1.10.0.rc,1.10.0,1.10.1,1.10.2,1.10.3,1.10.4,1.9.10,1.10.5,1.10.6,1.11.0.pre.1,1.11.0.pre.2,1.11.0,1.11.1,1.11.2,1.12.0.pre.1,1.12.0.pre.2,1.12.0.rc,1.12.0.rc.2,1.12.0.rc.3,1.12.0.rc.4,1.12.0,1.12.1,1.12.2,1.12.3,1.12.4,1.12.5,1.13.0.pre.1,1.13.0.rc.1,1.13.0.rc.2,1.13.0,1.13.1,1.13.2,1.12.6,1.13.3,1.13.4,1.13.5,1.13.6,1.13.7,1.14.0.pre.1,1.14.0.pre.2,1.14.0,1.14.1,1.14.2,1.14.3,1.14.4,1.14.5,1.14.6,1.15.0.pre.1,1.15.0.pre.2,1.15.0.pre.3,1.15.0.pre.4,1.15.0,1.15.1,1.15.2,1.15.3,1.15.4,1.16.0.pre.1,1.16.0.pre.2,1.16.0.pre.3,1.16.0,1.16.1,1.16.2,1.16.3,1.16.4,1.16.5,1.17.0.pre.1,1.16.6,1.17.0.pre.2,1.17.0,1.17.1,2.0.0.pre.1,2.0.0.pre.2,1.17.2,1.17.3,2.0.0.pre.3,2.0.0,2.0.1,2.0.2,2.1.0.pre.1,2.1.0.pre.2,2.1.0.pre.3,2.1.0,2.1.1,2.1.2,2.1.3,2.1.4,2.2.0.rc.1,2.2.0.rc.2,2.2.0,2.2.1,2.2.2,2.2.3,2.2.4,2.2.5,2.2.6,2.2.7,2.2.8,2.2.9,2.2.10,2.2.11,2.2.12,2.2.13,2.2.14,2.2.15,2.2.16,2.2.17,2.2.18,2.2.19,2.2.20,2.2.21,2.2.22,2.2.23,2.2.24,2.2.25,2.2.26,2.2.27,2.2.28,2.2.29,2.2.30,2.2.31,2.2.32,2.2.33,2.3.0,2.3.1,2.3.2,2.3.3,2.3.4,2.3.5,2.3.6,2.3.7,2.3.8,2.3.9,2.3.10,2.3.11,2.3.12,2.3.13,2.3.14,2.3.15,2.3.16,2.3.17,2.3.18,2.3.19,2.3.20,2.3.21,2.3.22,2.3.23,2.3.24,2.3.25,2.3.26,2.4.0,2.4.1,2.4.2,2.4.3,2.4.4,2.4.5,2.4.6,2.2.34,2.4.7,2.4.8,2.4.9,2.4.10,2.4.11,2.4.12,2.4.13,2.4.14,2.4.15,2.4.16,2.4.17,2.4.18,2.4.19,2.4.20,2.4.21,2.4.22,2.3.27,2.5.0,2.5.1,2.5.2,2.5.3,2.5.4,2.5.5,2.5.6,2.5.7,2.5.8,2.5.9,2.5.10,2.5.11,2.5.12,2.5.13,2.5.14,2.5.15,2.5.16,2.5.17,2.5.18

With the above versions response in place, I just:

  1. Uninstalled Ruby 3.3.5 and all the gems I had installed (I'm using RVM).
  2. Reinstalled Ruby 3.3.5.
  3. Ran gem update --system, which installed gem 3.5.19 and Bundler 2.5.19.
  4. Ran gem install bundler -v=2.5.19.

  5. Bundler installed correctly (even though it's missing Bundler 2.5.19 in the versions response). Here's the verbose output:

    HEAD https://my-nexus-host/nexus/repository/all-gems-group/versions
200 OK
GET https://my-nexus-host/nexus/repository/all-gems-group/info/bundler
200 OK
Downloading gem bundler 2.5.19.gem
GET https://my-nexus-host/nexus/repository/all-gems-group/gems/bundler-2.5.19.gem
Fetching bundler-2.5.19.gem
200 OK
... (a list of bundler files) ...

This was failing yesterday and we had to manually rebuild the Nexus cache to get it to find Bundler 2.5.19, so Nexus isn't in the same state as it was when I had trouble installing bundler.

I'll see if I can find another gem and version that's causing problems. It'll probably need to be a version that's been pushed since our Nexus cache was refreshed.