search

heroku / heroku-buildpack-ruby

Heroku's buildpack for Ruby applications.
MIT License
9 stars 3 forks source link

Gem sourced from git when added to development group still counts towards slug size #312

Closed jackturnbull closed 9 years ago

jackturnbull commented 9 years ago

It seems that gems built from a git repo and then later added to the development group are not properly removed by bundler. I had the 'refills' gem built from the git repo and noticed that my slug size was huge (>130mb) for an empty rails project. I then added this gem to the development group and the slug size remains unchanged.

Here is the Gemfile:

source 'https://rubygems.org'

# Bundle edge Rails instead: gem 'rails', github: 'rails/rails'
gem 'rails', '4.1.6'
# Use sqlite3 as the database for Active Record
gem 'sqlite3', group: :development
gem 'pg', group: :production
# Use SCSS for stylesheets
gem 'sass-rails', '~> 4.0.3'
# Use Uglifier as compressor for JavaScript assets
gem 'uglifier', '>= 1.3.0'
# Use CoffeeScript for .js.coffee assets and views
gem 'coffee-rails', '~> 4.0.0'
# See https://github.com/sstephenson/execjs#readme for more supported runtimes
# gem 'therubyracer',  platforms: :ruby

# Use jquery as the JavaScript library
gem 'jquery-rails'
# Turbolinks makes following links in your web application faster. Read more: https://github.com/rails/turbolinks
gem 'turbolinks'
# Build JSON APIs with ease. Read more: https://github.com/rails/jbuilder
gem 'jbuilder', '~> 2.0'
# bundle exec rake doc:rails generates the API under doc/api.
gem 'sdoc', '~> 0.4.0',          group: :doc

# Spring speeds up development by keeping your application running in the background. Read more: https://github.com/rails/spring
gem 'spring',        group: :development

# Use ActiveModel has_secure_password
# gem 'bcrypt', '~> 3.1.7'

# Use unicorn as the app server
# gem 'unicorn'

# Use Capistrano for deployment
# gem 'capistrano-rails', group: :development

# Use debugger
# gem 'debugger', group: [:development, :test]

# Bourbon front-end framework. See: http://bourbon.io
gem 'bourbon'
gem 'neat'
gem 'bitters'

### Problem Gem
gem 'refills', git: 'git://github.com/thoughtbot/refills.git'

Now push to heroku master:

Fetching repository, done.
Counting objects: 3, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 312 bytes | 0 bytes/s, done.
Total 3 (delta 2), reused 0 (delta 0)

-----> Ruby app detected
-----> Compiling Ruby/Rails
-----> Using Ruby version: ruby-2.0.0
-----> Installing dependencies using 1.6.3
       Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment
       Fetching gem metadata from https://rubygems.org/..........
       Fetching git://github.com/thoughtbot/refills.git
       Using rake 10.3.2
       Using i18n 0.6.11
       Using minitest 5.4.2
       Using thread_safe 0.3.4
       Using builder 3.2.2
       Using erubis 2.7.0
       Using rack 1.5.2
       Using arel 5.0.1.20140414130214
       Using mime-types 2.4.3
       Using sass 3.2.19
       Using thor 0.19.1
       Using coffee-script-source 1.8.0
       Using execjs 2.2.2
       Using hike 1.2.3
       Using multi_json 1.10.1
       Using pg 0.17.1
       Using bundler 1.6.3
       Using tilt 1.4.1
       Using tzinfo 1.2.2
       Using rack-test 0.6.2
       Using mail 2.6.3
       Using coffee-script 2.3.0
       Using bourbon 3.2.3
       Using sprockets 2.11.3
       Using bitters 0.10.1
       Using neat 1.5.1
       Using json 1.8.1
       Using activesupport 4.1.6
       Using rdoc 4.1.2
       Using uglifier 2.5.3
       Using actionview 4.1.6
       Using activemodel 4.1.6
       Using jbuilder 2.2.4
       Using activerecord 4.1.6
       Using actionpack 4.1.6
       Using sdoc 0.4.1
       Using actionmailer 4.1.6
       Using railties 4.1.6
       Using sprockets-rails 2.2.0
       Using coffee-rails 4.0.1
       Using jquery-rails 3.1.2
       Using rails 4.1.6
       Using sass-rails 4.0.4
       Using turbolinks 2.5.1
       Using refills 0.0.2 from git://github.com/thoughtbot/refills.git (at master)
       Your bundle is complete!
       Gems in the groups development and test were not installed.
       It was installed into ./vendor/bundle
       Bundle completed (17.74s)
       Cleaning up the bundler cache.
-----> Preparing app for Rails asset pipeline
       Running: rake assets:precompile
       Asset precompilation completed (2.42s)
       Cleaning assets
       Running: rake assets:clean

###### WARNING:
       Include 'rails_12factor' gem to enable all platform features
       See https://devcenter.heroku.com/articles/rails-integration-gems for more information.

###### WARNING:
       You have not declared a Ruby version in your Gemfile.
       To set your Ruby version add this line to your Gemfile:
       ruby '2.0.0'
       # See https://devcenter.heroku.com/articles/ruby-versions for more information.

###### WARNING:
       No Procfile detected, using the default web server (webrick)
       https://devcenter.heroku.com/articles/ruby-default-web-server

-----> Discovering process types
       Procfile declares types -> (none)
       Default types for Ruby  -> console, rake, web, worker

-----> Compressing... done, 133.9MB
-----> Launching... done, v7
       https://vast-crag-3598.herokuapp.com/ deployed to Heroku

The last line in the Gemfile is then changed to

gem 'refills', git: 'git://github.com/thoughtbot/refills.git', group: :development

Once this is commited, the response from push heroku master is:

Fetching repository, done.
Counting objects: 3, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 317 bytes | 0 bytes/s, done.
Total 3 (delta 2), reused 0 (delta 0)

-----> Ruby app detected
-----> Compiling Ruby/Rails
-----> Using Ruby version: ruby-2.0.0
-----> Installing dependencies using 1.6.3
       Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment
       Using rake 10.3.2
       Using json 1.8.1
       Using minitest 5.4.2
       Using thread_safe 0.3.4
       Using builder 3.2.2
       Using erubis 2.7.0
       Using rack 1.5.2
       Using mime-types 2.4.3
       Using arel 5.0.1.20140414130214
       Using thor 0.19.1
       Using sass 3.2.19
       Using coffee-script-source 1.8.0
       Using hike 1.2.3
       Using execjs 2.2.2
       Using multi_json 1.10.1
       Using pg 0.17.1
       Using bundler 1.6.3
       Using tilt 1.4.1
       Using rdoc 4.1.2
       Using i18n 0.6.11
       Using tzinfo 1.2.2
       Using rack-test 0.6.2
       Using bourbon 3.2.3
       Using mail 2.6.3
       Using coffee-script 2.3.0
       Using uglifier 2.5.3
       Using sprockets 2.11.3
       Using sdoc 0.4.1
       Using bitters 0.10.1
       Using neat 1.5.1
       Using activesupport 4.1.6
       Using actionview 4.1.6
       Using activemodel 4.1.6
       Using jbuilder 2.2.4
       Using actionpack 4.1.6
       Using activerecord 4.1.6
       Using actionmailer 4.1.6
       Using railties 4.1.6
       Using sprockets-rails 2.2.0
       Using coffee-rails 4.0.1
       Using jquery-rails 3.1.2
       Using turbolinks 2.5.1
       Using rails 4.1.6
       Using sass-rails 4.0.4
       Your bundle is complete!
       Gems in the groups development and test were not installed.
       It was installed into ./vendor/bundle
       Bundle completed (0.64s)
       Cleaning up the bundler cache.
-----> Preparing app for Rails asset pipeline
       Running: rake assets:precompile
       Asset precompilation completed (3.18s)
       Cleaning assets
       Running: rake assets:clean

###### WARNING:
       Include 'rails_12factor' gem to enable all platform features
       See https://devcenter.heroku.com/articles/rails-integration-gems for more information.

###### WARNING:
       You have not declared a Ruby version in your Gemfile.
       To set your Ruby version add this line to your Gemfile:
       ruby '2.0.0'
       # See https://devcenter.heroku.com/articles/ruby-versions for more information.

###### WARNING:
       No Procfile detected, using the default web server (webrick)
       https://devcenter.heroku.com/articles/ruby-default-web-server

-----> Discovering process types
       Procfile declares types -> (none)
       Default types for Ruby  -> console, rake, web, worker

-----> Compressing... done, 133.9MB
-----> Launching... done, v8
       https://vast-crag-3598.herokuapp.com/ deployed to Heroku

This is on a blank Rails 4.1.6 project.

It may also be worth noting that if a push using the second version of the Gemfile is done without first pushing the version where it is not in development then this error does not occur.

Cheers

geoffharcourt commented 9 years ago

@jackturnbull, I think the reason that the development gems count toward the slug size is that they are considered by Bundler when building your production bundle. Bundler does this to ensure that your production bundle doesn't end up with different gem versions (and with them potentially unanticipated behavior that differs from development or test environments) when you deploy.

Here's a post explaining this a bit: http://www.codeography.com/2010/05/20/exclude-bundler-gems-from-heroku-deployment.html

geoffharcourt commented 9 years ago

@jackturnbull, from the Heroku docs, it looks like you can cut out development and test gems with some environment configuration (see "Specifying gems and groups"): https://devcenter.heroku.com/articles/bundler

jackturnbull commented 9 years ago

Thanks for the links and the replies it's much appreciated, although I'm still not 100% convinced. When using non-git deployed Gems, heroku removes these by default. I carried out the same test with the HAML gem downloaded from Ruby Gems.

At least on my buildpack defaults, bundler is ran with the following flags, key one being without development:test

Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment

I had initially deployed HAML to production, then moved the gem to development and pushed the changed Gemfile. This resulted in the following output line from bundler:

Removing haml (4.0.5)

This then reduced the slug size from 26.0MB to 25.8MB but this 'Removing xxxx' line only seems to appear with pre-packaged gems.

Here are the two outputs from the git push. HAML in production:

Initializing repository, done.
Counting objects: 63, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (52/52), done.
Writing objects: 100% (63/63), 15.48 KiB | 0 bytes/s, done.
Total 63 (delta 2), reused 0 (delta 0)

-----> Ruby app detected
-----> Compiling Ruby/Rails
-----> Using Ruby version: ruby-2.0.0
-----> Installing dependencies using 1.6.3
       Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment
       Fetching gem metadata from https://rubygems.org/..........
       Installing i18n 0.6.11
       Installing rake 10.3.2
       Installing minitest 5.4.2
       Installing builder 3.2.2
       Installing erubis 2.7.0
       Installing json 1.8.1
       Installing mime-types 2.4.3
       Installing rack 1.5.2
       Installing arel 5.0.1.20140414130214
       Installing coffee-script-source 1.8.0
       Installing execjs 2.2.2
       Installing thor 0.19.1
       Installing tilt 1.4.1
       Installing hike 1.2.3
       Installing multi_json 1.10.1
       Using bundler 1.6.3
       Installing thread_safe 0.3.4
       Installing sass 3.2.19
       Installing mail 2.6.3
       Installing rack-test 0.6.2
       Installing rdoc 4.1.2
       Installing uglifier 2.5.3
       Installing coffee-script 2.3.0
       Installing haml 4.0.5
       Installing tzinfo 1.2.2
       Installing sprockets 2.11.3
       Installing sdoc 0.4.1
       Installing activesupport 4.1.6
       Installing activemodel 4.1.6
       Installing actionview 4.1.6
       Installing jbuilder 2.2.4
       Installing actionpack 4.1.6
       Installing activerecord 4.1.6
       Installing actionmailer 4.1.6
       Installing sprockets-rails 2.2.0
       Installing railties 4.1.6
       Installing coffee-rails 4.0.1
       Installing jquery-rails 3.1.2
       Installing sass-rails 4.0.4
       Installing rails 4.1.6
       Installing turbolinks 2.5.1
       Installing pg 0.17.1
       Your bundle is complete!
       Gems in the groups development and test were not installed.
       It was installed into ./vendor/bundle
       Post-install message from rdoc:
       Depending on your version of ruby, you may need to install ruby rdoc/ri data:
       <= 1.8.6 : unsupported
       = 1.8.7 : gem install rdoc-data; rdoc-data --install
       = 1.9.1 : gem install rdoc-data; rdoc-data --install
       >= 1.9.2 : nothing to do! Yay!
       Post-install message from haml:
       HEADS UP! Haml 4.0 has many improvements, but also has changes that may break
       your application:
       * Support for Ruby 1.8.6 dropped
       * Support for Rails 2 dropped
       * Sass filter now always outputs <style> tags
       * Data attributes are now hyphenated, not underscored
       * html2haml utility moved to the html2haml gem
       * Textile and Maruku filters moved to the haml-contrib gem
       For more info see:
       http://rubydoc.info/github/haml/haml/file/CHANGELOG.md
       Bundle completed (24.91s)
       Cleaning up the bundler cache.
-----> Preparing app for Rails asset pipeline
       Running: rake assets:precompile
       I, [2014-11-06T09:32:08.845596 #1065]  INFO -- : Writing /tmp/build_6258e4cbe62d21ed91db4d6d59d791c3/public/assets/application-ebd72bf780f7ac08d5059d4ee9d0b508.js
       I, [2014-11-06T09:32:08.869015 #1065]  INFO -- : Writing /tmp/build_6258e4cbe62d21ed91db4d6d59d791c3/public/assets/application-933259d6df80d47b0b16c28abf326b5e.css
       Asset precompilation completed (6.03s)
       Cleaning assets
       Running: rake assets:clean

###### WARNING:
       Include 'rails_12factor' gem to enable all platform features
       See https://devcenter.heroku.com/articles/rails-integration-gems for more information.

###### WARNING:
       You have not declared a Ruby version in your Gemfile.
       To set your Ruby version add this line to your Gemfile:
       ruby '2.0.0'
       # See https://devcenter.heroku.com/articles/ruby-versions for more information.

###### WARNING:
       No Procfile detected, using the default web server (webrick)
       https://devcenter.heroku.com/articles/ruby-default-web-server

-----> Discovering process types
       Procfile declares types -> (none)
       Default types for Ruby  -> console, rake, web, worker

-----> Compressing... done, 26.0MB
-----> Launching... done, v6
       https://limitless-castle-5869.herokuapp.com/ deployed to Heroku

After moving HAML to development:

Fetching repository, done.
Counting objects: 3, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 299 bytes | 0 bytes/s, done.
Total 3 (delta 2), reused 0 (delta 0)

-----> Ruby app detected
-----> Compiling Ruby/Rails
-----> Using Ruby version: ruby-2.0.0
-----> Installing dependencies using 1.6.3
       Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment
       Using rake 10.3.2
       Using minitest 5.4.2
       Using thread_safe 0.3.4
       Using builder 3.2.2
       Using erubis 2.7.0
       Using json 1.8.1
       Using i18n 0.6.11
       Using rack 1.5.2
       Using arel 5.0.1.20140414130214
       Using coffee-script-source 1.8.0
       Using mime-types 2.4.3
       Using execjs 2.2.2
       Using thor 0.19.1
       Using multi_json 1.10.1
       Using hike 1.2.3
       Using pg 0.17.1
       Using bundler 1.6.3
       Using tilt 1.4.1
       Using tzinfo 1.2.2
       Using sass 3.2.19
       Using rack-test 0.6.2
       Using mail 2.6.3
       Using rdoc 4.1.2
       Using coffee-script 2.3.0
       Using sprockets 2.11.3
       Using uglifier 2.5.3
       Using activesupport 4.1.6
       Using sdoc 0.4.1
       Using actionview 4.1.6
       Using activemodel 4.1.6
       Using actionpack 4.1.6
       Using activerecord 4.1.6
       Using actionmailer 4.1.6
       Using railties 4.1.6
       Using jbuilder 2.2.4
       Using sprockets-rails 2.2.0
       Using jquery-rails 3.1.2
       Using rails 4.1.6
       Using coffee-rails 4.0.1
       Using turbolinks 2.5.1
       Using sass-rails 4.0.4
       Your bundle is complete!
       Gems in the groups development and test were not installed.
       It was installed into ./vendor/bundle
       Bundle completed (0.86s)
       Cleaning up the bundler cache.
       Removing haml (4.0.5)
-----> Preparing app for Rails asset pipeline
       Running: rake assets:precompile
       Asset precompilation completed (1.93s)
       Cleaning assets
       Running: rake assets:clean

###### WARNING:
       Include 'rails_12factor' gem to enable all platform features
       See https://devcenter.heroku.com/articles/rails-integration-gems for more information.

###### WARNING:
       You have not declared a Ruby version in your Gemfile.
       To set your Ruby version add this line to your Gemfile:
       ruby '2.0.0'
       # See https://devcenter.heroku.com/articles/ruby-versions for more information.

###### WARNING:
       No Procfile detected, using the default web server (webrick)
       https://devcenter.heroku.com/articles/ruby-default-web-server

-----> Discovering process types
       Procfile declares types -> (none)
       Default types for Ruby  -> console, rake, web, worker

-----> Compressing... done, 25.8MB
-----> Launching... done, v7
       https://limitless-castle-5869.herokuapp.com/ deployed to Heroku

I'm aware that this could very well be an issue with bundler itself but it doesn't seem like expected behaviour that prepackaged gems are removed, although those build from git aren't.

schneems commented 9 years ago

We use a default BUNDLE_WITHOUT. You can see the exact command we run in the output of your build log. I haven't verified the nature of this bug, but if it's reproducible then it is a bug in bundler. Can you please open up an issue there?