Optional dependencies with fallback

[alexevanczuk]

Summary

This is request to allow bundler to be able to support optional dependencies with an opinionated syntax and behavior.

Use case

A creator of a gem might want to declare that their package has some additional functionality if the client wants to include an additional dependency. For example, a string formatter gem might want to optionally add string coloring functionality if the client is okay with including the rainbow gem.

Another use case is sorbet, a type-checker for ruby. More info on my specific use case here.

What we'd like is for a gem to include sorbet-runtime (method type annotations for sorbet). However, we do not want to burden the client with this gem if they do not want to install it. Instead, we'd like to fall back to a separate gem, which is a shim of this runtime.

Desired syntax

This is the desired syntax for this feature. Very open to other options here.

Take no action of the primary dependency is unavailable

spec.add_optional_dependency 'sorbet-runtime', '~> 0.5.5585'

Sub in a separate dependency if the primary dependency is unavailable

spec.add_optional_dependency 'sorbet-runtime', '~> 0.5.5585', else_add_dependency: 'sorbet-runtime-stub', '~> 0.14'

Desired behavior

Case 1: The user does not include the optional dependency

Behavior: If the gem uses else_add, it will include the given argument (sorbet-runtime-stub above) and reconcile the fall back against the client's other dependencies, and include it in Gemfile.lock of the client (i.e. the fallback is required and behaves as if its any other required gem).

Case 2: The user does include the optional dependency

Behavior: The fallback is ignored, and the optional dependency is required, and the versions are reconciled against the client's version, and the standard reconciliation process applies (i.e. bundler errors if there are irreconcilable dependencies).

Optional additions

Allow the client of the gem to specify the optionality inline.

For example, the client might have:

gem 'my_special_gem', include_optional_dependencies: 'sorbet'

This would allow the client to explicitly tell the gem to use that dependency. This is essentially syntactical sugar over:

gem 'sorbet'
gem 'my_special_gem'

It makes it explicit that the dependency is specifically to satisfy the optionality of the sorbet dependency.

If the client uses include_optional_dependencies, it uses the version specified by the gem. If the client wants to specify the version of sorbet, they use the second approach (i.e. simply declare the two separate gems with the desired version numbers).

Allow the gem to declare a fallback require

spec.add_optional_dependency 'sorbet-runtime', '~> 0.5.5585', else_require: '../lib/sorbet-runtime-stub'

This would allow us to explicitly handle the case where the client did not include the optional dependency.

Other options

This behavior is today able to be emulated, but the emulation is fraught with concerns. Here is my best attempt at emulating this behavior.

There are many issues with this: 1) The client's Gemfile.lock will not reflect that the optional dependency is actually a dependency of the gem. 2) Bundler will not reconcile versions at the time the user executes bundle install. Instead errors come only during runtime, which conflicts with the overarching philosophy of bundler of being able to reconcile dependencies statically in a separate process (i.e. before runtime). 3) This relies on possibly unstable API (specific errors raised by the gem command and the specific messages in those errors). 4) Depending on the load order, it might be possible that the fallback overrides the optional dependency in an unexpected way. 5) In general, we are moving dependency management away from bundler and into the client. This can potentially have a lot of undesirable downstream effects. One simple example is if you are constructing a dependency tree of your application, it would not reflect these optional dependencies using this workaround.

[kbrock]

Thanks for writing this up @alexevanczuk

I just nailed by this from Dependabot suggesting sqlite < 3 but my rails version requires ~> 1.4. I can run the tests using pg, but once I use sqlite3 it blows up.

Declaring the optional dependencies

As a starter just declaring the version number of both would be great:

spec.add_optional_dependency 'sorbet-runtime', '~> 0.5.5585'
spec.add_optional_dependency 'sorbet-runtime-stub', '~> 0.14'

This can be posted on rubygems and provide documentation if nothing else.

Reference optional dependencies

I do like the ability to declare that an optional dependency is used:

gem 'my_special_gem', include_optional_dependencies: 'sorbet-runtime'
gem 'activerecord', include_optional_dependencies: 'sqlite'

# Shorter key (name for the group to decide)

gem 'my_special_gem', uses: 'sorbet-runtime'
gem 'activerecord', uses: 'sqlite'

Since this is opt-in behavior, there is no behavior change without the developer actively seeking it. So no surprises.

It is similar to just including the sqlite gem, but you are deferring to activerecord to set the guidelines that works for them.

Does this resolve your concern @matthewd ?

Fallback

I feel the fallback dependency makes this a bit complicated.

For activerecord, it declares optional dependencies as pg, sqlite, and mysql. If there were a fallback, the developer may be inclided to fallback to sqlite, since activerecord is only useful with a database. But then when someone wants to use oracle, which is not on the gem developer's radar, there is that outstanding sqlite dependency.

I appreciate what we gain by having the fallback, but this adds a layer of complexity that may be better handled in a v2.