Discussion

[TimMoore]

Hi @dtognazzini I'm still interested in understanding this use case better. I'm raising an issue here to avoid blowing out the conversation on bundler/bundler-features#65 any further. I can post a summary there when we reach a good point.

In this specific case, I probably would not create Gemfiles for the child gems at all. I generally find that one Gemfile per git repo is a good rule of thumb. Bundler searches up through the directory tree for the closest Gemfile, so something like cd engines/blorgh && bundle exec rake still works.

Is there a down side that I'm missing?

[dtognazzini]

Hi @TimMoore. Thanks for carrying on the discussion!

The Gemfile in the child gems is not for the gems themselves, but for the embedded dummy Rails application in test/dummy. See the Rails Engine guide for more details on this setup.

While having a 1-to-1 relationship between a Gemfile and a git repo is common, I'm not sure it's a rule of thumb per se. Rather, you need a Gemfile to package together an environment for something to run. With a typical (i.e. flat) Rails app that doesn't use repo-local gems, you have one environment (from a gem perspective): the Rails application. With Rails Engines packaged in repo-local gems, you have many different environments. Each embedded dummy application in each Rails Engine packaged in each repo-local gem has a Gemfile that declares the environment for the dummy application.

The outer application pulls in the repo-local gems via their gemspecs ala Bundler::DSL#path. There was some discussion, and I've seen a few examples online, of having the outer application reference the Gemfiles from the repo-local gems directly instead of the gemspecs. I am not doing this.

There are some downsides to your proposed setup of having the dummy Rails apps local to the Rails Engines use the outer application's Gemfile. For one, it would result in each dummy app effectively "being" the outer Rails application thereby removing any isolation for the engines, which is the whole point of moving from a flat Rails app to one composed of an ecosystem of self-contained and provably independent subsystems. Secondly, in my case, the outer Rails application is quite large, and naturally, some subsystems have 'owners' who spend a lot of their time developing the engine. In this type of decentralized environment, it scales better to allow Gemfiles across the dummy applications to vary according to the owners wishes.

Does that make sense?

[TimMoore]

Yes, it does. Thanks for the additional detail... I'll give this some thought.

[TimMoore]

So... in your example, where would you put the require_gemfile calls? Can you add in a commit that shows what this project would look like if that existed?

[dtognazzini]

Per my comment in the original feature request, I probably wouldn't do this for the git case. Nonetheless, here is that example.

I would do this for uses of path I'd want to share. Or, if I wanted to share uses of group. I don't have an example of that at the moment. I was going to get to that next week as it's a bit more complicated to setup.

[johnnyshields]

@dtognazzini sounds like you and I are in the same boat. Here's my project tree, with one engine and one gem expanded to illustrate their internal structure:

[dtognazzini]

@johnnyshields, yep, we're doing exactly the same thing.

[dtognazzini]

@TimMoore @johnnyshields Here is an example of using require_gemfile for sharing path sources. The idea is that the project (require_gemfile_example) declares common sources to be used by all "gem environments" within the application in a sources.gemfile.

Each "gem environment" pulls in those sources by specifying the following in their respective Gemfile:

require_gemfile 'sources.gemfile'

This setup makes it easy to make project-global changes for the gem sources to use. In bigger applications like Johnny's, managing this manually without require_gemfile is a pain. With the project-global sources file in sources.gemfile, it's trivial to add/remove new sources without visiting all of Gemfiles for the repo-local gems.

Some notes about the require_gemfile implementation in the commit: it only supports path for Bundler 1.3.5. You can run it yourself by cloning and running bundle install in each of the engines/gems and at the top, project level.

[johnnyshields]

Am I correct to say the reasons for require_gemfile (or similar solution) are:

1) Specify external git dependencies within each engines? (Since non-git deps can be done in gemspecs already)

2) Specify dependencies between engines (since gemspecs cannot reference path). This is useful for testing engines among other reasons.

3) Ability to move engines/internal gems outside your main project and into a separate git repo, yet still track their git dependencies. (If require_gemfile can reference a git repo)

Are there other fundamental reasons?

[dtognazzini]

I just commented on the original feature request discussing some of this.

I'll elaborate here.

1) Specify external git dependencies within each engines? (Since non-git deps can be done in gemspecs already)

It could be used for this. However, my primary use case is to reuse repo-local sources via path.

2) Specify dependencies between engines (since gemspecs cannot reference path). This is useful for testing engines among other reasons.

This is my primary use case. To be clear, the dependencies are specified in the gemspec; they're resolved in the Gemfile.

3) Ability to move engines/internal gems outside your main project and into a separate git repo, yet still track their git dependencies. (If require_gemfile can reference a git repo)

I don't have this use case. In this case, I'd either use require_gemfile to reuse git source declarations (like your 1) or I'd release to a private gem server.

Another use for require_gemfile would be to reuse common Bundler setup like group as explained in bundler/bundler#3102.

[dtognazzini]

@TimMoore @johnnyshields dtognazzini/require_gemfile_example#8 shows some changes along with documentation on how I'd like to use require_gemfile to layout Rails application built from an ecosystem of self-contained subsystems.

I'm interested in hearing your thoughts.

[johnnyshields]

@dtognazzini great example. Some comments:

• Interesting how you use sub-gems inside engines (e.g. "yayaya" gem inside "lorde" engine). I do not do this, but in theory it should be valid to do. The drawback to this IMHO is the relative paths in your Gemfiles get more complex, e.g. doing lots of ../../../../../../root.gemfile to reference stuff at the top-level.

• I find it a bit cumbersome that you need 5 different files to define "lorde"'s dependencies:

  • /engines/lorde/Gemfile which contains gemspec and require_gemfile "sources.gemfile"
  • /engines/lorde/gemspec
  • /engines/lorde/sources.gemfile which contains require_gemfile "../../sources.gemfile"
  • /sources.gemfile which contains require_gemfile "engines/lorde/paths.gemfile"
  • /engines/lorde/paths.gemfile which contains path "gems"

  This seems like a lot of definition and path mangling. This is why I was originally proposing that :git and :path should be supported in gemspecs not Gemfiles. Couldn't you then achieve this entirely in /engines/lorde/gemspec:

  Gem::Specification.new do |s|
  s.name        = "lorde"
  s.add_dependency "yayaya", :path => "gems"
  end

• Suppose "lorde" depends a third-party gem "randy_marsh" on Rubygems which you do not own. What happens if "randy_marsh" breaks and you need to specify a :git => reference to an unreleased fixed version of "randy_marsh" in "lorde"'s Gemfile as a workaround? In this case, wouldn't case # 1 in my previous post apply to you?

[dtognazzini]

Interesting how you use sub-gems inside engines (e.g. "yayaya" gem inside "lorde" engine). I do not do this, but in theory it should be valid to do. The drawback to this IMHO is the relative paths in your Gemfiles get more complex, e.g. doing lots of ../../../../../../root.gemfile to reference stuff at the top-level.

The depth of the relative paths are constrained by each aggregate (i.e. require_gemfile_example, lorde) defining a "relative top level" sources.gemfile.

I find it a bit cumbersome that you need 5 different files to define "lorde"'s dependencies

Yes, that is a lot of files. Still, only the gemspec defines the dependencies.

Suppose "lorde" depends a third-party gem "randy_marsh" on Rubygems which you do not own. What happens if "randy_marsh" breaks and you need to specify a :git => reference to an unreleased fixed version of "randy_marsh" in "lorde"'s Gemfile as a workaround? In this case, wouldn't case # 1 in my previous post apply to you?

Yes. I've discussed this case in this comment. I understand what you're saying, but I think dependency resolution is more in the realm of Bundler's Gemfile declarations than in Rubygems' gemspec. Here is how I'd handle this case.

Much of the overhead here is simply unwarranted for simple projects and is intended for large projects where it's likely you have different "owners" maintaining the various subsystems.

The sources.gemfile and paths.gemfile paradigm could be simplified by using some conventions. See https://github.com/dtognazzini/require_gemfile_example/compare/embeddedGem...usingPathConvention for an example of replacing the use of paths.gemfile with a convention that sub gems keep their internal gems in a gems/ directory. This simplification wouldn't work if one were to aggregate the top level application (require_gemfile_example) with other applications using the same structure since require_gemfile_example stores its internal gems in both gems/ and engines/. Although, obviously, you could just make that a convention as well.

The structure could be further simplified by not supporting recursive composition by moving lorde's yayaya gem to the top level gem repository for the project. There are advantages and disadvantages to this approach. The big disadvantage to me is that it busts encapsulation and information hiding by spilling lorde's internals into the project level gem repository.

The point with these conventions (paths.gemfile, sources.gemfile, gems/, engines/, etc.) is to sustain simple maintainability as the project grows. Before the changes in the PR there was duplication in many places as shown by these changes: https://github.com/dtognazzini/require_gemfile_example/compare/embeddedGem...noReuse. Adding more gem sources (remote or local) would require changes to many Gemfiles. Similarly, if I were to require a certain unreleased version of some gem, I'd like that all of my sub gems are tested against the version used in the outer project.

As I explained here, I'm not sure using gems in this fashion or using the tools for managing gems (Bundler or Rubygems) in this fashion is the best way at decomposing/modularizing a complex project.

[dtognazzini]

@johnnyshields @TimMoore

Hey guys,

I wanted to share with you guys what we ended up doing. Basically, we removed all the Gemfiles used by the engines and had the engines use the top level Gemfile.

We used Bundler groups to achieve isolation between engines. We created one group for the development environment of the engine's dummy app and one for the test environment.

We created a little utility to be used in the Gemfile for defining dependencies on gems inlined in our Rails' project repository.

We updated our CI suite to infer the dependencies of the engines from the single top level Gemfile.lock. CI will kick off builds for engines when any of their code or their dependencies non-test code changes.

This has been working fantastic for us.

The README from our utility gem is below.

The inline_gem method expands to:

      gemsemble.dev_group(gem.development_group, gem.test_group) do
        @gemfile.gem gem_name, gem_params
      end

Gemsemble

Provides various utilities for helping with inline gem development whereby the inline gems share a Gemfile.

Gemfile Usage

Inside of your Gemfile, create a Gemsemble::Gemsembler:

gemsembler = Gemsemble::Gemsembler.new(...)

Use gemsembler to declare dependencies on inline gems:

gemsembler.inline_gem 'too_much_tuna'

inline_gem will:

1. add the inline gem to 2 Bundler groups: _too_much_tunadevelopment and _too_much_tunatest.
2. Add _too_muchtuna/gems to the Gemfile's source paths if it exists.

Use gemsembler to add other gems into the generated groups:

gemsembler.dev_group :too_much_tuna_development, :too_much_tuna_test do
  gem 'hpricot'
end

gemsembler.dev_group :too_much_tuna_test do
  gem 'mocha'
end

The groups created by dev_group can also be used to create reusable groups that can be mixed into other groups. For example, say you have many inline gems that have Selenium tests. You can create a reusable selenium_gems group and mix that group into the test groups of the inline gems.

gemsembler.inline_gem 'too_much_tuna'
gemsembler.inline_gem 'so_much_tuna'

gemsembler.dev_group :selenium_gems do
  gem 'capybara', '2.2.0'
  gem 'selenium-webdriver'
end

gemsembler.compose_group :too_much_tuna_test, :selenium_gems
gemsembler.compose_group :so_much_tuna_test, :selenium_gems

In the above example, all the gems in the selenium_gems will be added to both the too_much_tuna_test and so_much_tuna_test groups.

It's common to have default, development, and test Bundler groups. Suppose you'd like to create a set of groups to hold gems common to all other groups per environment:

gemsembler.dev_group :common do
  gem 'rake'
  gem 'passenger'
end

gemsembler.dev_group :common_development do
  gem 'annotate'
end

gemsembler.dev_group :common_test do
  gem 'mocha'
end

Using compose_group you can add the gems in these groups to the appropriate group of inline gems:

gemsembler.compose_group :too_much_tuna_development, :common
gemsembler.compose_group :too_much_tuna_development, :common_development

gemsembler.compose_group :too_much_tuna_test, :common
gemsembler.compose_group :too_much_tuna_test, :common_test

gemsembler.compose_group :so_much_tuna_development, :common
gemsembler.compose_group :so_much_tuna_development, :common_development

gemsembler.compose_group :so_much_tuna_test, :common
gemsembler.compose_group :so_much_tuna_test, :common_test

You can use compose_env_group to get the same results:

gemsembler.compose_env_group :too_much_tuna, :common
gemsembler.compose_env_group :so_much_tuna, :common

Code Usage

To use the inline gem groups declared in the Gemfile, call Bundle.require with the appropriate group for your gem:

require 'rubygems'
require 'bundler'

begin
  Bundler.require(:too_much_tuna_test)
rescue Bundler::BundlerError => e
  $stderr.puts e.message
  $stderr.puts "Run `bundle install` to install missing gems"
  exit e.status_code
end

If your inline gem is a Rails engine, you can use Gemsemble.gem_env_group:

require 'gemsemble'
Bundler.require(Gemsemble.gem_env_group("so_much_tuna", Rails.env))