git "<repo url>" , :depth => 1

[collin]

git-clone takes a --depth option.

This would be nice for large repositories like rails where history stacks up to 30MB.

[morgoth]

This would be very usefully i. e. on Heroku, where the repo size is quite important.

[wildchild]

Trivial, but extremely useful feature. +1.

[sferik]

It almost seems like this should be the default.

[hedgehog]

This would become very useful in chef cookbook repos where things can get quite 'deep'

[jeroenvandijk]

We are on Rails 3 edge on Heroku and we have a slugsize of 50mb.. So I'm thinking this would be a very nice feature :)

+1

[jamesarosen]

+1 and agree 1 should be the default depth. In fact, while I can imagine a reason for wanting a history, it seems like it would be very rarely used.

[josevalim]

Yes, this would rock. Have any of guys tried to implement it? A patch would be awesome!

[jeroenvandijk]

Haven't tried to implement it yet, but I will try to come with something shortly.

Any hints? I'll go into IRC and ask as well if necessary.

[collin]

http://github.com/carlhuda/bundler/blob/master/lib/bundler/source.rb#L573-598

I believe both clone and fetch take the depth option.

Personally I had difficulty getting the Bundler tests to run.

[jeroenvandijk]

@collin Yeah i tried to run individual specs, but didn't succeed. I also didn't get autotest to work with it. The whole test suite takes a bit too long to really do TDD. It seems that the test suite depends on the rake task.

Will look into it more.

[collin]

I couldn't even get them to run with the rake task. Maybe I didn't have all the development dependencies installed.

Confirmed, I've started up a new issue related to the unfortunate problem. Hopefully bundler can solve this problem by eating its own dogfood.

http://github.com/carlhuda/bundler/issues/issue/456

[collin]

Now that I've got the specs running I'm working on this tonight.

[collin]

changes: http://github.com/collin/bundler/commit/c736f9739e5f7bb54b1f3fa7ce1956e25b4c866e

I've gotten the current specs running with some --depth 1 changes. I'm not sure what to do about properly testing this, anybody have thoughts?

I get errors if I add --depth 1 to this line: http://github.com/collin/bundler/blob/feature/gitfetchdepth/lib/bundler/source.rb#L599

These are the sort of is the errors I get when adding --depth 1 to that line:

spec spec/install/git_spec.rb .......fatal: attempt to fetch/clone from a shallow repository fatal: The remote end hung up unexpectedly F..fatal: attempt to fetch/clone from a shallow repository fatal: The remote end hung up unexpectedly /Users/collin/Code/bundler/lib/bundler/source.rb:550:in load_spec_files': /Users/collin/Code/bundler/tmp/libs/foo-1.0 (at 248d5ff) is not checked out. Please runbundle install(Bundler::PathError) from /Users/collin/Code/bundler/lib/bundler/source.rb:370:inlocal_specs' from /Users/collin/Code/bundler/lib/bundler/source.rb:532:in specs' from /Users/collin/Code/bundler/lib/bundler/definition.rb:119:inresolve' from /Users/collin/Code/bundler/lib/bundler/definition.rb:117:in each' from /Users/collin/Code/bundler/lib/bundler/definition.rb:117:inresolve' from /Users/collin/Code/bundler/lib/bundler/definition.rb:151:in to_lock' from /Users/collin/Code/bundler/lib/bundler/definition.rb:147:ineach' from /Users/collin/Code/bundler/lib/bundler/definition.rb:147:in to_lock' from /Users/collin/Code/bundler/lib/bundler/environment.rb:39:inlock' from /Users/collin/Code/bundler/lib/bundler/runtime.rb:9:in initialize' from /Users/collin/Code/bundler/lib/bundler.rb:113:innew' from /Users/collin/Code/bundler/lib/bundler.rb:113:in load' from /Users/collin/Code/bundler/lib/bundler.rb:97:insetup' from -e:1 F..fatal: attempt to fetch/clone from a shallow repository fatal: The remote end hung up unexpectedly ..fatal: attempt to fetch/clone from a shallow repository fatal: The remote end hung up unexpectedly F......

1)
'bundle install with git sources handles repos that have been force-pushed' FAILED
expected: #<Gem::Version "1.1">,
     got: #<Gem::Version "1.0"> (using ==)
/Users/collin/Code/bundler/spec/support/matchers.rb:37:in `should_be_installed'
/Users/collin/Code/bundler/spec/support/matchers.rb:32:in `each'
/Users/collin/Code/bundler/spec/support/matchers.rb:32:in `should_be_installed'
./spec/install/git_spec.rb:314:

2)
'bundle install with git sources handles implicity updates when modifying the source info' FAILED
expected: "WIN",
     got: "" (using ==)
./spec/install/git_spec.rb:386:

3)
'bundle install with git sources when floating on master complains if pinned specs don't exist in the git repo' FAILED
expected "Updating /Users/collin/Code/bundler/tmp/libs/foo-1.0\nAn error has occurred in git. Cannot complete bundling." to include "Source contains 'foo' at: 1.0"
./spec/install/git_spec.rb:67:

Finished in 25.684657 seconds

22 examples, 3 failures

[indirect]

Sounds like the problem is that --depth 1 causes git errors when bundler tries to clone the shallow repo at a specific revision.

[indirect]

So guys, as nice as shallow clones sound, bundler allows you to pick any ref from the remote repository and install that. That's not compatible with shallow cloning, and never will be. So unless someone submits a patch that reconciles --depth 1 with the :ref option on gem, I think we're at a stalemate on this ticket.

[xn]

Couldn't we error out if :ref and :depth were both present?

[indirect]

what about if you use :depth but then remove it and use :ref instead?

[xn]

Reclone the repo?

[indirect]

When you run bundle install, the Gemfile.lock records the ref that you are using. From that point forward, Bundler must check out that ref, or it cannot satisfy the lockfile. That means :depth becomes useless as soon as there are X commits to the repo, where X is :depth+1. Does that make sense?

[xn]

Not completely, nor does it have to. You know this codebase and understand the implications, I've just poked around it. The setup is this: 1) We were devving against 3.1 edge on heroku and our slug sizes were way over the top. 2) Apparently after using git for the past 6 years I missed the --depth option. 3) I wondered if bundler supported it so I found this thread and after reading it, it seemed I was missing something as to why this was closed. Thanks for taking the time to humor me.

[indirect]

Honestly, I think the best way for Heroku to cut slug sizes would be to exclude the git repositories from the slugs. I'm pretty sure that once they've checked out the files at the correct ref, the rails repo itself is unnecessary. @hone, c/d?

[xn]

I've only just started working with heroku deploys(in Feb), so I can't say for sure, but I definitely see your point.

[sferik]

Why was this issue closed? Has this been implemented?

[maxwell]

Yeah, we could really use this @diaspora

[indirect]

Hmm... I didn't close it. I blame github. Reopening...

@maxwell, did you read the ticket? Unless there's a way to clone a shallow repo that somehow includes a specific sha, I don't see how this is possible to implement. :\

[stevenh512]

@indirect I agree that it might not be possible to do a shallow clone and still satisfy Gemfile.lock, but your other idea a few comments up is interesting. I wonder how hard it would be to modify Heroku's Ruby buildpack to get rid of the git repos after Bundler has checked out the right version of the files. Haven't looked at the code for Bundler in quite some time, but I'm also curious how hard it would be to make Bundler do that for us when the --deployment flag is used.

Either way would definitely be an improvement over the current situation, where adding one git dependency (in my case, rails_admin) adds up to 37mb to my slug size.. but I imagine getting rid of the git repo would also make each deploy take longer, so it's probably something that should be done in the Heroku buildpack if possible instead of potentially making deploys take longer for everyone (regardless of whether they're using Heroku, their own VPS or Dedicated, or whatever else). :grin:

Maybe a better option (for Heroku, specifically) might be to have the git repos cached somewhere outside the slug where they're only stored once and available to every deploy during slug compilation.. similar to the way Bundler+RVM with globalcache enabled keeps everything in my $HOME/.rvm/gems/cache/bundler/git directory. That, combined with modifying the buildpack to get rid of the git repo history in vendor/bundle during slug compile, should keep the repo out of the slug without affecting deploy/slug compile times too much, but I'm sure that's not something I could achieve just by hacking the buildpack, it would have to be done on Heroku's end.

[collin]

Maybe something could go into the heroku .slugignore file?

https://devcenter.heroku.com/articles/slug-compiler#ignoring_files_with_

On Mon, Apr 16, 2012 at 3:30 AM, Steven Hancock < reply@reply.github.com

wrote:

@indirect I agree that it might not be possible to do a shallow clone and still satisfy Gemfile.lock, but your other idea a few comments up is interesting. I wonder how hard it would be to modify Heroku's Ruby buildpack to get rid of the git repos after Bundler has checked out the right version of the files. Haven't looked at the code for Bundler in quite some time, but I'm also curious how hard it would be to make Bundler do that for us when the --deployment flag is used.

Either way would definitely be an improvement over the current situation, where adding one git dependency (in my case, rails_admin) adds up to 37mb to my slug size.. but I imagine getting rid of the git repo would also make each deploy take longer, so it's probably something that should be done in the Heroku buildpack if possible instead of potentially making deploys take longer for everyone (regardless of whether they're using Heroku, their own VPS or Dedicated, or whatever else). :grin:

Maybe a better option (for Heroku, specifically) might be to have the git repos cached somewhere outside the slug where they're only stored once and available to every deploy during slug compilation.. similar to the way Bundler+RVM with globalcache enabled keeps everything in my $HOME/.rvm/gems/cache/bundler/.git directory. This should keep the repo out of the slug without affecting deploy/slug compile times too much, but I'm sure that's not something I could achieve just by hacking the buildpack, it would have to be done on Heroku's end.

---

Reply to this email directly or view it on GitHub: https://github.com/carlhuda/bundler/issues/228#issuecomment-5147900

[stevenh512]

@collin Interesting, I hadn't thought of that.. I'm always really careful about what I put in my .slugignore since I've broken more than a few deploys that way (thankfully all small apps that didn't really matter), but since we're talking about files that aren't really needed by the deployed app that might actually be the simplest solution.

[collin]

@github really wish I could @ reference an organization in an issue so I could ask @heroku to chime in on this one and give some advice.

edit: nm, it seems like it may have worked, created links. wonder if anybody will see those?

[stevenh512]

Unfortunately, adding anything in vendor/bundle to the .slugignore makes no difference, slug size is exactly the same.

Hacking the buildpack reduces the slug size of my Diaspora deploy by almost 11mb but I'm not sure yet if there are any side-effects other than the multiple "fatal: Not a git repository (or any of the parent directories): .git" errors I see during assets:precompile, even with the errors the app still runs, assets are still precompiled and the git dependencies (or rails_admin, at least) are still working as they should. :grin:

I think that might be my best bet, but I also think we might be drifting a little off-topic (while it's still related, hacking a Heroku buildpack isn't exactly a Bundler issue lol) so maybe we should move this discussion somewhere else?

[hone]

hey guys, sorry I've been traveling around Europe and won't be back until RailsConf. Feel free to open an issue on the ruby buildpack for the heroku specific stuff. I'd be more than happy to chat at RailsConf though. I'd need to look into it more to see what side effects there are for not having the git repo there.

[collin]

go open source! 2 year old ticket still gets action :)

[edrex-janrain]

I don't see any problem with making :depth and :ref incompatible (throw an exception). Am I missing something?

[indirect]

@edrex-janrain as soon as you have run "bundle install" one time, every git repo is :ref. That makes them all incompatible with :depth. :)

[xaviershay]

Moved to https://github.com/bundler/bundler-features/issues/1 as part of The Great Bundler Triage.