RubyGems.org v1 dependency API deprecation

[headius]

We learned recently that the v1 version of the dependency-lookup API on RubyGems.org was deprecated on February 23, slated for complete removal on (at time of writing) May 10.

This API is used by the mavengem plugin to fetch a list of dependencies for gem artifacts, and is critical to this plugin. When the API goes away–if we are unable to update, test, and release a new version of mavengems–JRuby's build and any downstream builds that fetch Ruby gems as Maven dependencies will cease to function.

I have joined an issue here, basically begging for more time to make this transition: https://github.com/rubygems/rubygems.org/issues/3698#issuecomment-1531812315

This came to our attention, finally, due to the intermittent "brown outs" that RubyGems.org has deployed in the last month. That led to a revival of this jruby-maven-plugins issue: https://github.com/torquebox/jruby-maven-plugins/issues/124#issuecomment-1530093898

Had we noticed the original deprecation blog post two months ago we might have had time to make this transition. Now, with only 8 days left and a major Ruby conference coming up (RubyKaigi on the 11th), I don't see how we can fix this before the API shuts down altogether.

I am begging the RubyGems.org maintainers for additional time to make this transition. Meanwhile we need to move quickly to try to fix our uses of the deprecated API (and make sure we're not using any others).

[Axel-1]

Hello, Do you have any updates about the migration to API V2? We use mavengem for our software and we can no longer build our project.

Thank!

[headius]

@Axel-1 Sorry for the late reply. The RubyGems servers have whitelisted our use of this API from around mid-May until mid-August. We still need to update the plugin for the new API and get a release out (ASAP).

[headius]

An update on this:

I have in #9 managed to wire up the v2/rubygems API and part of the compact index API used by bundler to replace most of the uses of the v1/dependencies API.

The changes are nearly all in rubygems-tools, which is where the bulk of the integration code lives. All tests in mavengem-protocol and mavengem-wagon pass with the new code. Most tests pass in rubygems-tools; the known failures are due to the "bundler API" features and other API endpoints still dependent on getting multiple gems' worth of data using the form dependencies?gems=rails,sequel.

I confirmed that the new mavengem works with the disabled v1/dependencies API by doing the following:

• Run with an unmodified build of the updated and cleaned-up mavengem 1.0.4-SNAPSHOT. Gems were retrieved successfully.
• Run with a modified build that removes "Java" from the request User-Agent, preventing our requests from getting whitelisted. Gems failed to be retrieved with the "not found" errors users reported this spring during the API brownouts.
• Run with a build of the PR that both removes "Java" from the User-Agent and includes the new code for v2 and compact API. Gems were retrieved successfully.

So in a pinch, we could do a release with the known issue that the "bundler API" feature is temporarily broken. That would ensure JRuby and other mavengem users can still resolve dependencies past the August 8 shut-down of the v1/dependencies API.

Note we have moved this library from the "org.torquebox.mojo" group ID to the "org.jruby.maven" group ID. Torquebox is defunct now and we do not have push rights to that group anyway. Going forward, mavengem maintenance is done by the JRuby team.

[headius]

Resolved by #9.