Test gem with multiple dependencies on Travis CI

Intro

Sometimes you wrote a gem, you need to test it with multiple dependencies, and you use Travis CI. You don't know where to get started, this article is for you!

Gemfile for RubyGem

A RubyGem typically contains a .gemspec and a Gemfile. Personally I specified all runtime dependencies in .gemspec, and all non-runtime dependencies (gems for development, test environments) in Gemfile.

source "https://rubygems.org"

# Specify your gem's dependencies in your-gem.gemspec
gemspec

group :development do
  # development environment dependencies
end

group :test do
  # test environment dependencies
end

Suppose you wrote a gem foo that need to test from Ruby 2.0 to trunk Ruby, and from Rails 3 to Rails 5.

.travis.yml

Test with Multiple Ruby Versions

# .travis.yml

rvm:
  - 2.3.1
  - 2.2.5
  - 2.1
  - 2.0
  - ruby-head

This will run your build with Ruby:

Ruby 2.3.1 (2.3.1)
Ruby 2.2.5 (2.2.5)
Ruby 2.1.x (2.1)
Ruby 2.0.x (2.0)
Trunk Ruby (ruby-head)

The build will run according to this order. So it is important you put the version you care the most on top (cannot fail), and the version you care the least at bottom (those can fail).

Some builds are impossible to run on Rails 5, like unsuppoted Ruby version (<= 2.0). Some builds are safe to fail, like build with trunk Ruby; You can specify which Ruby versions in .travis.yml that can exclude or can fail:

# .travis.yml

matrix:
  fast_finish: true
  exclude:
    - rvm: 2.0
  allow_failures:
    - rvm: ruby-head

With fast_finish: true in place, as soon as these builds finish:

Ruby 2.3.1 (2.3.1)
Ruby 2.2.5 (2.2.5)
Ruby 2.1.x (2.1)

The whole build will be considered as PASSED / FAILED.

OK. So you now know how to run your build with different Ruby on Travis CI.

How about run the build with different dependencies (gems)?

Test with Multiple Dependencies

Thanks to Bundler, we can specify our dependencies in Gemfile. And Travis CI let you run your build with multiple Gemfile, if we want to test our build from Rails 3 to Rails 5:

gemfile:
  - gemfiles/rails_5.gemfile
  - gemfiles/rails_4.gemfile
  - gemfiles/rails_3.gemfile

Create a folder gemfiles on project root and put rails_3.gemfile, rails_4.gemfile, and rails_5.gemfile files. And you specify your dependencies in each file. Travis will now run your build with Ruby versions you specified combined with these gemfiles:

2.3.1 Rails 3
2.2.5 Rails 3
2.1.x Rails 3
2.0.x Rails 3
Trunk Ruby Rails 3
2.3.1 Rails 4
2.2.5 Rails 4
2.1.x Rails 4
2.0.x Rails 4
Trunk Ruby Rails 4
2.3.1 Rails 5
2.2.5 Rails 5
2.1.x Rails 5
2.0.x Rails 5
Trunk Ruby Rails 5

And you can also specify which Ruby with certain Rails version can fail, for example, Rails 5 required Ruby version 2.2.2, so Ruby version < 2.2.2 will fail:

# .travis.yml

matrix:
  fast_finish: true
  allow_failures:
    - gemfile: gemfiles/rails_5.gemfile
      rvm: 2.1
    - gemfile: gemfiles/rails_5.gemfile
      rvm: 2.0
    - rvm: ruby-head

OK how do we create these gemfiles:

Gemfile
gemfiles/rails_3.gemfile
gemfiles/rails_4.gemfile
gemfiles/rails_5.gemfile

We will use thoughtbot's Appraisal tool to do that.

thoughtbot/appraisal

Appraisal runs your tests across configurable, reproducible scenarios that describe variations in dependencies. For example, if you need to test with versions of Rails

First install appraisal gem to development environment:

# Gemfile
source "https://rubygems.org"

gemspec

group :development do
  gem "appraisal"
end

Then creates a Appraisals file with following content:

appraise "rails-3" do
  group :development do
    gem "bundler"
    gem "rake"
  end

  gem "rack", "< 2"
  gem "rails", "3.2.22.2"
end

appraise "rails-4" do
  group :development do
    gem "bundler"
    gem "rake"
  end

  gem "rack", "< 2"
  gem "rails", "~> 4.2.6"
end

appraise "rails-5" do
  group :development do
    gem "bundler"
    gem "rake"
  end

  gem "rails", "~> 5.0.0"
end

And put common dependencies in Gemfile:

source "https://rubygems.org"

gemspec

group :development do
  gem "bundler"
  gem "rake"
  gem "appraisal"
end

then Appraisal file can just be:

appraise "rails-3" do
  gem "rack", "< 2"
  gem "rails", "3.2.22.2"
end

appraise "rails-4" do
  gem "rack", "< 2"
  gem "rails", "~> 4.2.6"
end

appraise "rails-5" do
  gem "rails", "~> 5.0.0"
end

Add:

require "rubygems"
require "bundler/setup"

to the top of your Rakefile.

Run $ appraisal install command, appraisal command-line program will read your Appraisals file and generate multiple Gemfiles to gemfiles folder.

With everything in place, you can now run tests for Rails 3, 4, 5 with appraisal:

$ appraisal rails-3 rake
$ appraisal rails-4 rake
$ appraisal rails-5 rake

Or run tests with all Rails:

$ appraisal rake

To learn more about how appraisal works, please refer to the Appraisal README.md

Wrap Up

You learned...

Role of Gemfile and .gemspec for RubyGem
How to test with multiple Ruby versions on Travis CI
How to test with multiple Gemfiles on Travis CI
How to exclude certain build / allow to fail
How to use appraisal to generate Gemfiles
How to run tests with appraisal

See a real-life example here: https://github.com/jch/html-pipeline/pull/257.

jollygoodcode / jollygoodcode.github.io