unnecessary git clone when updating package commit sha

[liskin]

Steps to reproduce:

1. put this in some project's stack.yaml:

   packages:
   - '.'
   - location:
      git: git://github.com/tfausak/strive.git
      commit: cfbd995dacafb1242482c0b6ce23826fad505338
    extra-dep: true

2. stack build
3. change commit to c39704d5f008af42f5c3c2d5319476c7aa12e52d
4. stack build again

Expected results:

• the second stack build does just git remote update; git reset --hard

Actual results:

2016-01-07 14:06:07.661449: [debug] Run process: git clone git://github.com/tfausak/strive.git /home/tomi/src/strava-gear/.stack-work/downloaded/5727e3e6b7fab4ac5241b42c7d83482fbddc6f05b1b698540df4ab681225a8ee.git.tmp/ @(stack_1RV4odVDW2e8SkMJA1Z8Zb:System.Process.Read src/System/Process/Read.hs:255:3)
2016-01-07 14:06:13.978009: [debug] Run process: git reset --hard c39704d5f008af42f5c3c2d5319476c7aa12e52d @(stack_1RV4odVDW2e8SkMJA1Z8Zb:System.Process.Read src/System/Process/Read.hs:255:3)

This means whenever I want to update the dep, the entire git repository is cloned over the network again. That's not very good. :-(

I may use git submodules as a workaround, but it's not nice.

[mgsloan]

Hmm, yeah, I'm surprised it doesn't already do it this way. Looks like it's naming the package's directory based on the SHA. Since we don't know the previous SHA, we'll need to change this naming schema in order to reuse the prior git repo.

[mgsloan]

Fixed on master!

[mgsloan]

Re-opening this, because my change only fixed the case where the commit was already in the repo.

To fix properly we'll need to use git fetch

[sjakobi]

Re-opening this, because my change only fixed the case where the commit was already in the repo.

Oh, and could we also get some logging? Depending on repo size and internet connection the wait can be a surprising.

[da-x]

About git fetch, maybe you can do something like git fetch origin -f <git-commit-hash>:sync-for-build && git reset --hard sync-for-build (which is agnostic to local branch name), to ensure 'fetch' only brings the requested commit from the remote and not all changes from all remote branches.

[mgsloan]

I think it also makes sense to use --shallow to avoid downloading the full history, I see only upsides, no downsides.

[da-x]

For fetch? It only has --unshallow. you mean --update-shallow? At any case, it would the most efficient for the shallow clone invocation to bring only the requested commit. So overall, history would not be brought either for fetch or clone. Should also verify that these operations work well for repos with submodules.

[da-x]

I'm missing something - when you edit the git-hash for a dependency in the stack.yaml and re-run Stack, how does Stack knows to reuse the previous clone? What prevents the previous clone to sit idly in a directory bearing the previous hash?

[mgsloan]

I meant git clone --depth 1 to avoid downloading full history. I just found out why we don't do that, though http://docs.anybox.fr/anybox.recipe.openerp/trunk/devnotes/git-sparse-shallow.html / http://thread.gmane.org/gmane.comp.version-control.git/115811

Summary is that it's too computationally intensive to allow fetching specific SHAs, as it requires determining if it's accessible via a ref. Skipping the reachability check is considered to be a potential security issue and it's a non-default repo option.

We could have heuristics to determine if something looks like a tag, and do a shallow clone. Dunno if this complication is worth it.

[mgsloan]

I'm missing something - when you edit the git-hash for a dependency in the stack.yaml and re-run Stack, how does Stack knows to reuse the previous clone? What prevents the previous clone to sit idly in a directory bearing the previous hash?

It doesn't, actually. I made some changes towards doing this, but currently it just deletes the old repo and reclones.

At least changing things to use git fetch instead of deleting the repo would be good.

[mgsloan]

This turns out to be tricky to get right! Downgrading to P3

[liskin]

Does it really ever remove the old repo?

(Does stack ever remove anything? :-) My experience is that it basically expects an unlimited space and breaks in unexpected ways if someone manually tries to remove old things.)

[mgsloan]

No, the old repo isn't removed, and yes that could build up garbage. Removing .stack-work will clear it out.