Support requiring Gemfiles inside of Gemfiles (i.e. Gemfileception)

[dtognazzini]

Background

Gems, by themselves, only declare dependencies; they do not inform how to satisfy those dependencies. Bundler declares gem dependencies for package management and also specifies how to satisfy the dependencies (e.g. source, path, gem(git:), etc.)

It would be nice to reuse Gemfile declarations (both dependency specification and satisfaction) in other Gemfiles without copying and pasting.

Proposal

Provide a require_gemfile method in Bundler::DSL that takes a path to another Gemfile that can hold common dependency resolution declarations. For example:

<root>/Gemfile

require_gemfile "common/Gemfile"
gem "some_gem"

<root>/common/Gemfile

source "https://my_fav_gemserver.net"
path "gems" do
  gem "my_fav_gem"
end

The above would be identical to the following single file:

<root>/Gemfile

source "https://my_fav_gemserver.net"
path "common/gems" do
  gem "my_fav_gem"
end
gem "some_gem"

Notice: the resulting Gemfile nests the relative paths of the required Gemfile (common/Gemfile) under the relative path from the requiring Gemfile (common/).

Use Case 1: Sharing common gems

This is very similar to bundler/bundler#3102, which uses eval_gemfile. In addition to sharing dependency declaration, require_gemfile allows sharing all supported DSL methods.

What require_gemfile offers over eval_gemfile is that it rebases relative paths from the required Gemfile to relative paths from the requiring Gemfile.

Use Case 2: Sharing source code, repository local gem paths

It's become a common pattern amongst Rails projects to decompose the system into an ecosystem of inline gems containing Rails Engines. The main Rails Application pulls in the inline gems by declaring dependencies in the application-level Gemfile. For example:

_<systemroot>/Gemfile

path "gems" do
  gem "subsystem_one"
  gem "subsystem_two"
  gem "subsystem_three"
end

If subsystem_one grows crazily, it could be further decomposed into many gems, perhaps:

_<system_root>/subsystem_one/subsystemone.gemspec

 s.add_runtime_dependency("subsystem_one-authorization")
 s.add_runtime_dependency("subsystem_one-billing")
 s.add_runtime_dependency("subsystem_one-analytics")

_<system_root>/subsystemone/Gemfile

# path "gems" here would be <system_root>/subsystem_one/gems
path "gems" do
  gem "subsystem_one-authorization"
  gem "subsystem_one-billing"
  gem "subsystem_one-analytics"
end

To support this decomposition, the main application's Gemfile would have to be updated:

_<systemroot>/Gemfile

# Declare dependencies on subsystems
path "gems" do
  gem "subsystem_one"
  gem "subsystem_two"
  gem "subsystem_three"
end

# Add the path to find gems required by subsystem_one
path "subsystem_one/gems"

Updating the top level Gemfile for subsystems undergoing structural refactors like this is somewhat painful but reasonably maintainable provided there is only one, conventional "gems" repository for each layer.

When it becomes unmaintainable

This strategy becomes unmaintainable with one more level of layering.

For example, consider a super_system project that pulls in the root of the above project (named here as system_1) and the root of another project (system_2):

_<super_system>/systems/system1.gemspec

 s.add_runtime_dependency("subsystem_one")
 s.add_runtime_dependency("subsystem_two")
 s.add_runtime_dependency("subsystem_three")

_<supersystem>/Gemfile

path "systems" do
  gem "system_1"
  gem "system_2"
end

The super_system project would have to include paths in its Gemfile for the gems required by system_1:

_<supersystem>/Gemfile

# Declare dependencies on the systems
path "systems" do
  gem "system_1"
  gem "system_2"
end

# Add paths to find gems required by system_1 and its subsystems
path "systems/system_1/gems"
path "systems/system_1/gems/subsystem_one/gems"

The last line couples super_system to the internal structure of system_1.

Contrived?

This example may seem a bit contrived, but I work on something similar to this whereby a system is comprised of many Rails Applications. Each Rails Application supplies its own inline gem that the system depends on in its Gemfile. The inline gems provided by the Rails Applications are used by the system to interact with the applications. In the usage above, every restructuring internal to system_1 would need to be handled by the Gemfile for super_system.

With require_gemfile, using a convention whereby systems specify their path repositories in a Gempaths file, the above would be simpler:

_<supersystem>/Gemfile

# Reuse gem paths defined by system_1
require_gemfile "systems/system_1/Gempaths"

path "systems" do
  gem "system_1"
  gem "system_2"
end

_<super_system>/systems/system1/Gemfile

# Use gem paths defined for system_1
require_gemfile "Gempaths"

_<super_system>/systems/system1/Gempaths

# Reuse gem paths defined by the subsystems
require_gemfile "gems/subsystem_one/Gempaths"
require_gemfile "gems/subsystem_two/Gempaths"
require_gemfile "gems/subsystem_three/Gempaths"

_<super_system>/systems/system_1/gems/subsystemone/Gemfile

# Use gem paths defined for subsystem_one
require_gemfile "Gempaths"

_<super_system>/systems/system_1/gems/subsystemone/Gempaths

path "gems"

With the above, no layer knows about the internals of the layer beneath it.

[dtognazzini]

@johnnyshields Per bundler/bundler/issues/2688 and rubygems/rubygems/issues/702, you may find the proposal presented in this issue interesting.

I too work on Rails apps that are composed of many engines and I've felt the pain of having to copy-and-paste git: and path: declarations across Gemfiles.

With require_gemfile, you could put common declarations in a shared Gemfile to be reused in other Gemfiles. Although, I still use gemspecs for declaring library dependencies.

[indirect]

In the general case, gems do in fact declare how to satisfy dependencies: via gem servers, which are declared in the Gemfile using source. If you have application specific needs (a patch that isn't upstream yet, a vendored and unpacked internal library) you can use git or path in your Gemfile as a sort of escape hatch.

While Bundler supports gems from paths or git repos, the happy path is still (and is intended to be) installing gems directly from servers. Most users don't have more than one Gemfile in a single repository, since most users only have one application in each git repo. For repos that have many gems, like Rails, there is a single unified gemfile that declares the Rails gem in the current directory. Bundler is smart enough to automatically find all of the Rails sub-dependencies inside the same directory, and they don't have to be explicitly spelled out, path by path.

To zoom way, way out here... what exactly is it that you want to do with Bundler that led you to this feature request? Maybe it can be solved in a way that doesn't requiring adding something this complex to Bundler.

[dtognazzini]

With respect to dependency resolution, I mean that gemspecs by themselves do not specify how their dependencies are satisfied, only that they have dependencies.

I took a look at the Rails repo and I see what you mean: there is a single top level Gemfile that uses the gemspecDSL method and the related active* gems are found locally.

I've described 2 uses cases above. I realize they're a bit difficult to read. One of the use cases is very similar to the Rails layout, although, I store the repo-local gems under a sub-directory, which means I have to specify path "sub-directory" in my Gemfile.

The Rails engine guide advises creating mountable engines wrapped in repo-local gems and suggests pulling in these gems into the main Rails application via: gem 'blorgh', path: "/path/to/blorgh". This is not the general case, but it is becoming fairly common for Rails projects. Using repo-local gems in this fashion is not a workaround, but intentional, as the repo-local gems will never be upstreamed to a gem server.

Rails engines come with a nested dummy Rails application to run tests against. Development for the engine uses a Gemfile local to the Rails engine; it's not the same as the main Rails application in the repo. One use of require_gemfile would be to reuse common Gemfile setup across engine-local Gemfiles.

I'll pull together a sample Rails project to exhibit what I mean. I won't have any time to do that until next week, though.

[johnnyshields]

This proposal seems good to me.

Like @dtognazzini, I am also requiring internal gems via the :path to their gemspec. The main limitation I face is that the gemspec syntax doesn't allow :git gem references. Requiring these internal gems via their Gemfile instead of gemspec would would be a reasonable way to resolve this limitation (since Gemfiles can use git directives). An alternative would be to support :git directives inside gemspecs, but as per previous conversations with the Rubygems authors this looks unlikely to happen. Another alternative is to host your own gem server, but that's quite a bit of extra work (pushing updates for each modification, etc).

[indirect]

The local trick works automatically on subdirectories. You don't need to do anything special to get the sub-gems, even if they are in subdirectories.

For projects that all share a lot of git dependencies, I'm not really clear of what is easier about git push than rake release to your private gem server. Gemfury offers private servers with a couple of clicks, and then you gain all the benefits of releases and versions that git lacks.

On Thu, Oct 30, 2014 at 9:55 AM, Johnny Shields notifications@github.com wrote:

This proposal seems good to me.

Like @dtognazzini, I am also requiring internal gems via the :path to their gemspec. The main limitation I face is that the gemspec syntax doesn't allow :git gem references. Requiring these internal gems via their Gemfile instead of gemspec would would be a reasonable way to resolve this limitation (since Gemfiles can use git directives). An alternative would be to support :git directives inside gemspecs, but as per previous conversations with the Rubygems authors this looks unlikely to happen. Another alternative is to host your own gem server, but that's quite a bit of extra work (pushing updates for each modification, etc).

Reply to this email directly or view it on GitHub: https://github.com/bundler/bundler-features/issues/65#issuecomment-61127309

[johnnyshields]

Suppose a sub-gem is dependent on a Github-only (unreleased) version of a gem, there's no way to maintain this reference in the sub-gem's gemspec currently. The best you can do is specify the Github reference in the parent Gemfile, but this is not ideal.

As far as using Gemfury, I have over 20 internal gems in my project and growing. It would be to much work to have to release every change for each gem.

[indirect]

You already have to git push each change for each gem; rake release is the same amount of work. I guess I’m pretty unclear on your workflow. Can you explain it in detail, like Donnie did for his use-case?

On Oct 30, 2014, at 11:53 AM, Johnny Shields notifications@github.com wrote:

Suppose a sub-gem is dependent on a Github-only (unreleased) version of a gem, there's no way to maintain this reference in the sub-gem's gemspec currently. The best you can do is specify the Github reference in the parent Gemfile, but this is not ideal.

As far as using Gemfury, I have over 20 internal gems in my project and growing. It would be to much work to have to release every change for each gem.

— Reply to this email directly or view it on GitHub https://github.com/bundler/bundler-features/issues/65#issuecomment-61149531.

[johnnyshields]

I have all my gems inside one project. Each gem is inside the /engines/ folder. In my Gemfile, I have:

%w( subgem1 subgem2 subgem3 ... ).each do |engine|
  gem engine, path: File.expand_path("../engines/#{engine}", __FILE__)
end

[indirect]

@johnnyshields You don't need to do that. You can just do this:

# Gemfile
gemspec

# app.gemspec
Gem::Specification.new do |s|
  [...]
  s.add_dependency "subgem1"
  s.add_dependency "subgem2"
  s.add_dependency "subgem3"
end

[johnnyshields]

@indirect even using this more concise syntax, the problem remains that it references the gemspecs of the sub-gems, and gemspecs can't do things like reference Github dependencies.

[indirect]

@johnnyshields I may not be fully understanding you, but it sounds like you're saying that Bundler should add a very complicated feature that is likely to cause a lot of bugs so that you can run git push instead of rake release when you update sub-dependencies. Is that true? Is there something I'm missing?

[johnnyshields]

@indirect firstly I wish to be respectful for the great work you and others have done on Rubygems/Bundler--the "Extreme Makover" work you have done in the past 2 years has been a lifesaver for the community at large.

My issue is best explained by example. Suppose I have a "Parent" Rails project and "Child A" and "Child B" as subgems in a subdir of the project. I wish to declare the following:

1) Parent depends on Child A 2) Child A depends on Child B 3) Child B depends on a git-only version of a commonly used gem, e.g. "devise"

Conceptually I'd like to represent this as something like:

# Parent Gemfile
gem 'child_a', path: '/subgems/child_a'

# Child A gemspec (or Gemfile)
gem 'child_b', path: '../child_b'  # depends on Child B

# Child B gemspec (or Gemfile)
gem 'devise', git: 'plataformatec/devise', branch: 'not_yet_released'

However, the syntax in Child A (:path) and Child B (:git) are not supported in gemspecs. So the closest I can do is:

# Parent Gemfile
gem 'child_a', path: '/subgems/child_a'
gem 'child_b', path: '/subgems/child_b'
gem 'devise', git: 'plataformatec/devise', branch: 'not_yet_released'

# Child A gemspec
# reference to Child B omitted, since Child B doesn't exist on Rubygems

# Child B gemspec
Gem::Specification.new do |s|
  s.add_dependency 'devise'    # no version/git constraint
end

In addition, in order to test Child B isolation against the repo-version of devise, Child B must have a Gemfile which has the git devise reference. This is a duplication of the reference in the Parent Gemfile. This is non-DRY and becomes cumbersome when working with ~150 gem dependencies across the project.

Two possible solutions are:

1) Rubygems could support :git / :path dependencies in its gemspecs, OR

2) Bundler could support embedding Gemfiles within other Gemfiles (a la @dtognazzini's require_gemfile proposal).

It is true that I could host all my internal gems a private gem server, and rake release each time I make a change to a given gem, but I feel this is too cumbersome when the number of subgems (~25) greatly outnumbers the number of people in the organization (~5).

[johnnyshields]

One last note, even if I were to use a private gem server, it would not solve the issue of having to reference the repo version of devise in my Parent Gemfile (when I'd rather reference it from Child B's Gemfile / gemspec). Granted I could also release that repo version of devise as a private gem on my gem server, but all this becomes a time burden when managing 25 internal and 150+ external gems.

[indirect]

Okay, I think I understand what you're saying now. I have doubts about this entire scheme of organization as being overly complex. :)

Using gems from git is slower, adds complexity, and honestly is a (functional and helpful, but still a) hack. Using released gems greatly increases the speed of Bundle install, bundle exec and everything else Bundler does. Git gems are intended to be temporary when they are absolutely needed, and should not be standard operating procedure.

Again, it seems like your described situation requires git push and bundle update every time you update a sub-gem, so I'm not sure why you're objecting to rake release and bundle update, since it's literally zero more steps.

[johnnyshields]

My usage above does not require a git push / bundle update for my sub-gems. I reference sub-gems via :path, not :git. I do also think it's a valid use case to reference sub-gems via :git--if I could I might do that for certain gems which I've hired vendors to build--but it's not what I do currently.

The main issue I have with :git is that I cannot a reference a Git version of a commonly used library in the sub-gem gemspec. As a workaround I put this :git reference in my parent Gemfile, but this is unnatural since it's actually the sub-gem which has the dependency.

I agree with your statement that git references are a hack, but given the reality of version upgrades, bugs, gem maintainers gone AWOL, etc. they a necessary evil.

[johnnyshields]

Regarding the "complexity" of the scheme, the ultimate goal is use internal gems to reduce the complexity of the app. Gems are a very useful paradigm to "isolate" code into loosely-coupled components with well-defined APIs and dependencies. I often to delegate tasks to my team and vendors by requesting them to build a subsystem as a gem.

if we could resolve the :git / :path limitations I think many more people/teams would do things in this manner--it would even be a "best practice" for scaling out Ruby apps using loose-coupling IMHO.

[segiddins]

@johnnyshields I think the solution here is to use a gem server to enforce loose coupling, rather than using ':git/:path` as a permanent solution.

[johnnyshields]

I still think a gem server is overkill. Why should I have to setup a server and roundtrip it when I have the code right there in a subdir of my project? Why shouldn't we have other options to best suit the team's workflow?

[TimMoore]

@johnnyshields can you talk more about the team's workflow? Do people work on these gems independently of the parent project? Are the gems shared between projects (it sounds like no)? Are all of the gems in the same git repo as the parent project (it sounds like yes)?

If it's the case that these gems are specific to a project and not reusable outside of it, what is the benefit you get from structuring them as gems rather than simple subdirectories on your load path?

What is the current process that a new developer would use to set up the project and start working on it? How do you imagine an ideal process would be different?

What problems do you encounter by specifying all of the git gem overrides in the top-level Gemfile that would be solved by splitting them up?

[johnnyshields]

There are two types of sub-gems I use:

• Standalone Gems (in my /gems/ dir) are private gems that are completely isolated. Structure-wise they look like the gems on RubyGems, but I don't make them public.
• Rails Engines (in my /engines/ dir) are mountable rails apps as described here. Each Rails Engine has a gemspec so it is technically a "gem", but structure-wise these look like Rails apps (e.g. with /app/, /config/, etc dirs)

Do people work on these gems independently of the parent project?

Yes. I've had several cases where I've asked vendors to build specific gems based on an API spec and have not given them access to the main project repo, with great results.

Are the gems shared between projects (it sounds like no)?

Yes. I have multiple apps on different domains which share internal gems, for example tablesolution.com which is restaurant mgmt app and tablecheck.jp which is a public-facing reservation booking app, the core reservation engine shared by both is packaged as a gem. (Currently I have these two apps as separate "engines" in the same project, but I'd like to split them into separate projects if I could use :git references in my gemspecs.) In addition, I've had third parties contact me about licensing some of my components, and since they are already packaged as gems it makes it very easy to share the code.

Are all of the gems in the same git repo as the parent project (it sounds like yes)?

Currently yes, but I would like to move some gems to separate private git repos if I could get around the aforementioned limitations.

[TimMoore]

If they are all in the same git repo, then how do people work on them independently of the main project or share them between projects?

[johnnyshields]

I've put them in a separate repo for the vendor, then copy-pasted periodically :(

(Again this is due to the aforementioned limitations; allowing :git reference would save this headache!)

[TimMoore]

@johnnyshields so what would the ideal configuration & workflow look like for you? Let's say you had each gem/engine in a separate git repo... how would that work?

It sounds to me like the only solution that actually works in that case would be to release the gems (including forked gems) to a gem server, because as soon as the Gemfile for the child gem is not in the same repo as the parent project, the hypothetical require_gemfile wouldn't be an option.

[johnnyshields]

Let's say you had each gem/engine in a separate git repo... how would that work?

• I would reference most of my "engine gems" by path and most of my "standalone gems" by git.
• For my "engine gems", I would want to declare dependencies to both git versions of public gems (e.g. gem "devise", github: "plataformatec/devise", branch: "not-yet-released) and also to my other engines/gems.
• For my "standalone gems", I'd reference them by git tag or branch and use git as if it were gem server (minus the need to setup/maintain an extra server and run rake release each time).

If I only had "standalone gems" I'd probably be OK to use a gem server. It's really the "engines" that are the pain point, since:

• I have a lot of them
• it's far more expedient to reference them by path
• I can't declare dependencies from one engine to another
• I can't declare dependencies from an engine to a repo version of public gems.

as soon as the Gemfile for the child gem is not in the same repo as the parent project, the hypothetical require_gemfile wouldn't be an option.

As per above, this would not really be an issue for "engines" since they're referenced by path. But in order to support this, the require_gemfile could have a :git option itself which instructs to load from git but using the Gemfile in the repo instead of the gemspec.

But I think a better/cleaner solution would be to allow :git and :path references in the gemspec files instead, perhaps with an internal_mode! directive which enables these options but prevents release to Rubygems. I proposed this here: https://github.com/rubygems/rubygems/issues/702

[segiddins]

Specifications should never declare how dependencies are fulfilled, only what they are.

[johnnyshields]

@segiddins the "what" which I wish to reference is the code at a certain system path or git tag. Unfortunately it can't be referenced without also specifying "how". Allowing this for private gems (not released to Rubygems) seems valid to me, because the "what" is not in the public domain and is directly under your control. In order to release to Rubygems one wouldn't be allowed to do this--all of the "whats" for a public gem must also be in the public domain.

[segiddins]

And that's what gem servers are for :)

-Samuel E. Giddins

On Nov 4, 2014, at 10:02 PM, Johnny Shields notifications@github.com wrote:

@segiddins the "what" which I wish to reference is the code at a certain system path or git tag. Unfortunately it can't be referenced without specifying "how". Allowing this for private gems (not released to Rubygems) seems valid to me, because often times this "what" is not in the public domain and is directly under your control. In order to release to Rubygems one wouldn't be allowed to do this--all of the "whats" in a public gem must also be in the public domain.

— Reply to this email directly or view it on GitHub.

[johnnyshields]

Gem servers are not an ideal solution for the engines (path-based) use case as I and @dtognazzini have outlined above.

[johnnyshields]

I will put up a $3,000 bounty to resolve this limitation, i.e. allowing :git and :path references inside nested gems. I'm agnostic as to whether the issue is resolved in Bundler (Gemfiles) or Rubygems (gemspecs), so long as it is merged into master of either library.

[indirect]

@johnnyshields wow, is that a serious offer? While Bundler team is unlikely to endorse your style of dependency management as recommended, I have an idea of a way that it might be possible to do what you want without the significant changes that are part of this pull request.

[johnnyshields]

The offer is serious, under the assumption that significant development is required in the either the Bundler or Rubygems gem as we've been discussing in this thread. I'll pay $500 for a low-effort but non-brittle workaround solution that meets my requirements (both git and path references without a gem server.)

[johnnyshields]

(Just to be clear the two different bounties are for "first-class support" versus a "clever hack". I think we all know the difference between the two.)

[indirect]

First-class support is way, way more than $3000 worth of effort. But I can offer you a clever hack. :)

[johnnyshields]

If it meets my requirements, the $500 is yours.

[indirect]

In extremely short form, here it is: Any place that you would want to use require_gemfile, use eval File.read. So a clever hack that implements the require_gemfile method from the beginning of this pull request without any changes to Bundler would look like this:

# ./Gemfile
def require_gemfile(name)
  dir = File.dirname(name)
  eval File.read(name).gsub(/(:?path(?: ?| ?=> ?).*?)"(.+)"/, '\1"' + dir + '/\2"')
end

require_gemfile "common/Gemfile"
gem "some_gem"

# ./common/Gemfile
source "https://my_fav_gemserver.net"
path "gems" do
  gem "my_fav_gem"
end

With this (effective) result:

source "https://my_fav_gemserver.net"
path "common/gems" do
  gem "my_fav_gem"
end
gem "some_gem"

The gsub takes all the ways you can call path in a Gemfile and prepends the directory of the required gemfile onto them. Git gems don't need any modification. That definitely does what you've described.

[johnnyshields]

Apologies was just writing my requirements and you beat me to the punch. Does this solve all of the following:

1) Each of the four cases below (AA, AB, BA, BB) should be supported:

Parent project Gemfile

• Nested gem A referenced by path
  • 2x nested gem AA referenced by path
  • 2x nested gem AB referenced by git
• Nested gem B referenced by git
  • 2x nested gem BA referenced by path
  • 2x nested gem BB referenced by git

2) Nesting should be possible for n-levels.

3) Gem conflicts:

• If nested gems A and B reference two different git tags or paths of gem C, there must be a bundler error thrown.
• If nested gem A references gem C via git, and gem B references C via path, it is considered a conflict.
• If nested gems A and B reference the same git tag or path of gem C, the bundle must succeed.
• "paths" in this context should be aware of relative paths.

4) Git references to private github repos should work as they currently do in Bundler (http://user:pass@github.com/xxx/xxx.git is fine)

5) Bundle install, etc. should work as normal, there should not be:

• major caveats that prevent normal Bundler usage
• extra manual steps each time we bundle
• gem code which must be manually maintained/copied to local dirs

6) No private/dedicated gem server should be required (source 'http://rubygems.org' is fine obviously)

[indirect]

(To be super clear to anyone reading this ticket later: the above is a clever but dirty hack, it could break in future versions of Bundler, and the Bundler team only provides help with things like this if you provide monetary compensation.)

[indirect]

@johnnyshields yes. I'm not going to build test cases for each individual point because that's more than $500 worth of my time, but the hack supports arbitrary depth nesting, git gems work as usual, and conflict errors will be raised the same as if the conflicting requirements were inside the same file. You will probably need to add the def require_gemfile at the top of each Gemfile that requires another Gemfile.

[johnnyshields]

@indirect thanks and let's take this offline for now, I've emailed you at the address in your GH profile.

[dtognazzini]

@indirect I've finally gotten around to pulling together an example Rails project illustrating my setup. Although, after reading through all these comments, I'm fairly convinced that you understand both the setup and workflow.

Here's the example: https://github.com/dtognazzini/require_gemfile_example

@johnnyshields's comment here is the same as this commit.

Another example, which I did not implement in my example, is illustrated in bundler/bundler#3102. In this case, their Gemfile "requires" a common Gemfile.devtools file via eval_gemfile. Their Gemfile.devtools uses the group method from Bundler::DSL. If all Gemfile.devtools did was just pull in other gems, you could create a gem instead and have the Gemfile pull it in. But, in this case, they're using eval_gemfile to share the use of the group method from Bundler::DSL.

To be clear, I do use a private gem server for gems that are used across projects. In my example project I could have forked-and-released a one-off gem to my private gem server. And, I'd probably do just that if my fork was reusable across other projects.

That said, sharing dependency resolution for remote gems (e.g. using git, source, etc.) is just one use case that require_gemfile would solve. In other cases, you might want to share uses of group, path, etc.

bundler/bundler#3102 provides an example of group. I'll pull together an example for path, but I probably won't be able to get to it until next week (again) ;-)

Finally, to reiterate some of what's been said here and other places, I think the separation between dependencies declaration and dependencies resolution is good. The goal of require_gemfile is to enhance the resolution side provided by Bundler with a way to reuse resolution details, but also with the ability to reuse all the methods provided by Bundler::DSL; basically, require for Gemfiles.

Perhaps a different implementation whereby you could use plain-old require in a Gemfile would be sufficient. Rake supports this by extending the main object with the Rake DSL. Whereas Bundler instance_evals the Gemfile. I wonder if it would be possible to rework some of the internals to allow for using plain-old require.

[indirect]

@dtognazzini thanks for the additional writeup! I definitely understand the pain of having to copy and paste git and path dependency declarations between Gemfiles. A big part of the reason that we've been hesitant to support something like require_gemfile is that it encourages coupling between libraries: if your application only works when a child dependency of a child dependency is coming from git, then your application actually has a direct dependency on the git version of that child dependency, and it should be declared in that application's Gemfile.

After emailing with @johnnyshields, it turns out that his desired version of this feature includes even more, like the ability to require gemfiles from the git repos of other gems. That version of this feature is even less likely to ever be merged into Bundler.

I think the best way out of this kind of mess is by both decoupling your libraries (so they can be released independently) and by releasing actual gems with actual version numbers that can be used to create a dependency tree. Ultimately, as evidenced by both the pull request and my "clever hack" implementation, it's possible, but I don't think it's a good practice.

Ultimately, of course, it's Ruby, and you can do amazing hacky things with Ruby. Anyone who wants to do this can implement require_gemfile themselves, knowing that they are taking on the cost of maintaining it. In the future, we'd like to support a plugin system that allows new methods to be added to the Gemfile DSL. This is a great candidate for that, since you would be able to maintain a plugin that is outside Bundler's core, but provides the functionality that you want inside your own Gemfiles.

I'm going to close this ticket, since we've decided that this feature isn't a good fit for Bundler itself, but I'm happy to talk about the plugin system and how to make it easier for you to maintain this feature on your own for yourself and others who want to use it.

[dtognazzini]

@indirect thanks for spending the time on this discussion; it's been very helpful!

A big part of the reason that we've been hesitant to support something like require_gemfile is that it encourages coupling between libraries

That is a valid concern.

if your application only works when a child dependency of a child dependency is coming from git, then your application actually has a direct dependency on the git version of that child dependency, and it should be declared in that application's Gemfile.

In this case, I'd say that whatever has the requirement on the specific dependency coming from git is the thing that actually has the dependency, not the application; the application has an indirect dependency.

As I alluded to in my previous comment, in this case I'd probably fork-and-release a gem to a private gem server with a pre-release version. In fact, I've done this with great success. I typically do this if the timeline for an official release of the gem with the fix is unknown or is just too long to wait and if the maintenance burden is getting annoying. Another option would be to look at using a git submodule, but I haven't run that experiment yet.

Reusing git sources would not be my primary use of require_gemfile. My primary use case is to reuse repo-local sources via path. A few of us are continuing that discussion here for those interested. This is only necessary because I'm using gems to implement repo-local/project-local libraries as described by the Rails Engines guide.

Anyone who wants to do this can implement require_gemfile themselves, knowing that they are taking on the cost of maintaining it

True, but it's a bit more difficult with a gem like Bundler as it's usually installed on the system. An implementation for require_gemfile would have to bind to some public interface of some version of Bundler. require_gemfile makes use of eval_gemfile which isn't part of the public API, so at least in the current implementation it would be difficult to implement require_gemfile outside of official support because there's no way to say at the project level "use Bundler version X" for this Gemfile. Hmmmm... or is there?. All that said, this alone isn't reason enough to add a general feature to Bundler.

Some final thoughts on require_gemfile:

• Perhaps using gems for implementing project-local libraries isn't a good practice. Doing so has some advantages:
  • Explicit versioned, dependency declaration.
  • Isolation
  • Conventions: naming, testing, packaging, etc.
• Perhaps Bundler isn't the tool for a feature like this since I really only want to use path.
• Perhaps Bundler is the right tool but there's a way to accomplish this without something as general and broad reaching as require_gemfile.

In my previous life working with C++ in MSDev (sshhhh), creating an application from an ecosystem of project-local libraries was trivial. What's the Ruby/Rails analog?

I'm happy to talk about the plugin system and how to make it easier for you to maintain this feature on your own for yourself and others who want to use it.

I'm interested. Where can I find more information?

Thanks!

[johnnyshields]

A big part of the reason that we've been hesitant to support something like require_gemfile is that it encourages coupling between libraries

"Coupling" is the degenerative case of "dependency" which occurs when dependencies are too narrowly defined. I agree that we wouldn't want git and path dependencies on public (Rubygems-based) gems, but why not allow for private usage? I still think something like private_mode! in gemspecs which would enable git and path, but disallow pushing to Rubygems is the way to go here.

[TimMoore]

I still think something like private_mode! in gemspecs which would enable git and path, but disallow pushing to Rubygems is the way to go here.

I think this would be a far better solution than to try to work around its absence in Bundler. My main worry is that I don't want the Gemfile to try to become an alternative gemspec. It's already difficult enough for people to understand how all of the different pieces interact as it is.

[dtognazzini]

private_mode! sounds interesting. However, it doesn't solve the use case of wanting to reuse group or other Bundler::Dsl methods. Additionally, I'm not convinced that Rubygems should have this complexity for all the same reasons that require_gemfile may add too much complexity to Bundler.

My main worry is that I don't want the Gemfile to try to become an alternative gemspec.

That's a valid concern. It's interesting that a project's dependencies are defined in a Gemfile. Alternatively, the project could define its dependencies in a gemspec and use the Gemfile solely for resolving dependencies. The Gemfile is fulfilling multiple roles:

• expressing dependencies
• resolving dependencies
• specifying semantic groups that mean something in the context of the project.
• providing some convenient methods for pulling in the dependencies for the runtime (e.g. require).

I wonder if there are other tools that are a better fit for defining a project composed of project-local gems. To me, Bundler is positioned to be that tool as it's already in the business of defining the project, whereas gemspecs are in the business of defining independent libraries.

[ccutrer]

Wow, a long conversation. I completely understand the bundler team's apprehensiveness at making public/supporting a require_gemfile method. I just want to present how we use our own "clever hack" of it (which is basically what @indirect wrote for @johnnyshields). I'm also in complete agreement that gemspecs should not specify the source.

We use sub-gemfiles for two important reasons:

1. To break up and maintain the Gemfile. Canvas (https://github.com/instructure/canvas-lms) is a massive Rails project, and depends on 300+ gems (not all of them directly, but many many of them are). To that end, we have a Gemfile.d, and put each bundler group in its on .rb file. This also inherently (partially) solves the use case of developer-specific gems, by just creating your own untracked file in that directory. This wasn't a main goal of creating Gemfile.d, but it's been a happy side effect for some developers. It's also only a partial solution, because developer-specific gems would end up in the lockfile. But that's not a problem, because we don't commit the lock file, because of the next major reason we use sub-gemfiles.
2. Plugins. Canvas is an open source project, but Instructure also maintains proprietary plugins that extend its functionality for us. There are other organizations that also maintain their own Canvas extensions, and by keeping them in plugins, they don't have to worry about merge conflicts keeping the "shared" main Canvas up to date. There are also potential legal benefits, because Canvas is AGPL (being able to keep their own code out of Canvas). Historically, Canvas was a Rails 2 application, and plugins lived in vendor/plugins, and any directory in there automatically had its Gemfile pulled in so that it had its dependency. In Rails 4, vendor/plugins is no longer supported, and we've been migrating both private and public plugins to become real (non-released; referenced by :path) gems, that are simply in the correct location. Either way however, we cannot commit the Gemfile.lock because it would contain references to the proprietary plugins that not everyone has access to. Sadly, this also means we tend to be excessively specific about dependent versions in our actual Gemfile.

Our "clever hack" system has worked well for us, and we're okay maintaining custom monkey-patches of Bundler in our Gemfile when necessary (and removing them if/when they are accepted upstream; https://github.com/bundler/bundler/pull/3171 being one of them). Recently, I ran into an issue that one of our proprietary plugins needs to use a custom fork of a gem we do not maintain. This is not the first time that it has happened, and it's normally easily resolved by using a Gemfile that's not in the public Canvas repo (but included by it), that either points to a private gem server, or points to the git source. Unfortunately, this gem happens to be one that is also used by the main part of Canvas, so having the separate declaration of it in the private Gemfile fragment conflicted with the public Gemfile, which motivated https://github.com/bundler/bundler/pull/3421 (which was rejected, which makes sense because that feature makes no sense in a single-gemfile world; and then I was pointed here)

In conclusion, I'm not pushing to have bundler or rubygems include first class support for either of our scenarios. We would use it if it was available. I'm simply trying to raise awareness that there are use cases out there that would benefit from better support from Bundler, and would be open to any suggestions that are less hacky (especially around not being able to commit the Gemfile.lock).

[johnnyshields]

@ccutrer thanks for your awesome comment. I totally get it.

Recently I'm turning my attention towards node.js. Their package manager NPM fully supports the :git and :file recursive dependency use cases I've outlined above. Granted, NPM has it's own issues, but at least the Node community does not get hung up on these silly semantics of "what" versus "how".

[indirect]

@ccutrer The entire point of a lock file is to guarantee that the exact same gem versions are available in development and production. What you're describing is the opposite of that: you want to be able to run different versions of the same gems on different machines or different production environments. The closest we can allow to that use-case is using a built gem, and then supplementing one of that gem's dependencies by declaring a git or path gem in the Gemfile. That still produces a single logical outcome—one set of gems for all machines. How do you imagine the lock could possibly work in cases like the one you are asking for?

[ccutrer]

I know, and I wish we could do that (natively). Right now we essentially lock down our versions so tight in the Gemfile that the lockfile nearly matches the Gemfile. I think in an ideal world, there would be a lockfile corresponding to each Gemfile that all contribute to the master Gemfile, locking the gems in that specific file (and a final runtime check that none of those lockfiles conflict).

In practice, when we build a new release package, we include the lockfile in it, so we don't have strange problems later if a dependency changed (we also package the gems).

Another not-so-perfect solution is one suggested for per-user gemfile inclusion - having a separate lockfile. Then we could commit the generic lockfile excluding any additional plugins not included in the base project. On deploy, or for developers working with additional plugins, they get a separate lockfile of their own that's not commited (it would be nice if it was "bootstrapped" with the main lockfile, so they're not too far off, but that seems complicated to impossible to keep up to date).

[johnnyshields]

@ccutrer supposing gemspecs could specify :git and :file dependencies, which would be evaluated at higher priority than version numbers >= 0.3.0 (i.e. Rubygems), does that solve your problem?