Add a migrate command for submodule and subtree

[ingydotnet]

Write a command to convert old usage to subrepo

[erlandsona]

Please, with a cherry on top! I'm trying to update my dotfiles organization since I recently dual booted my 2010 MBP with Arch Linux. I figure it's time to really dig into an organized dotfiles with symlinks and all... Next step is to write bootstrapping scripts for each OS. Or inherit one of the bajillion others.

Granted in my newness to the whole submodules / subtree / now subrepo movement... I mistakenly added eveything as a submodule then read a bunch of reasons why submodules will get hairy soon.

So I adopted subrepo and got it installed and completions hooked up. But I deinit'd my .bash_it submodule thinking I could somehow get back my local changes to that repo and well you can guess how that turned out... Anyways. :+1: if we could get a submodule migration command? Something like...

$ git subrepo init Arch/.bash_it/`
 > This directory is currently a submodule. Would you like to convet it into a subrepo?[Y/n]:

[ingydotnet]

@erlandsona, I like your idea of bundling this under the init command.

We should start by having the init command at least recognize that the subdir is already a submodule, subtree or stree.

Any takers? :)

[erlandsona]

I'd be happy to dig into the code to see if I can't figure it out... Is all the code for the executable in that one bash file?

[linzuzeng]

This function would be very helpful when subrepo clone a repository alreadly containing subtrees or submodules.

[rbellamy]

In #3 @ingydotnet mentions that git subrepo clone -f $url $subdir is a possible solution for dealing with a current subtree - is this the accepted method for "migrating" from git subtree to git subrepo?

I keep getting dirty history with subtree with duplicate commits as a result of push/pull changes in commit hashes between repositories, and would love to have a better solution. From everything I've seen, git-subrepo is that solution - but I've got to address my current subtrees before I can make the leap.

I know this issue is for a migrate command - but maybe a bit of guidance on the best path forward until such a command is available?

[ingydotnet]

@rbellamy are the repos in question, public repos? If so, it would be easy for me to test out the steps.

Else can you locate a public repo that uses subtree?

[rbellamy]

@ingydotnet GREAT question - unfortunately, the public repo I'm working on is a subtree of a private repo, so it's not really an option here.

Some quick searches of closed GitHub issues produced this: https://github.com/spotify/ios-ci/pull/11

Apparently they converted ios-ci/ci from submodule to subtree just recently, so maybe not the best example, since the only commit on that subtree is the original subtree merge.

Perhaps a better sandbox: https://github.com/FALCONN-LIB/FALCONN/pull/53

[ingydotnet]

As I understand it, a simple approach would be just to delete the subtree subdir, commit that, and then clone the subrepo into the same dir.

You could try this on a branch, and see how it works for you, and report back.

[rbellamy]

I did try that, and the result is that the commit history ends up with a large-scale delete followed by a large-scale add - and my attempts at graft/replace to fix-up the history ended in tears. Mostly because even after years of using git, my skills are still journeyman-level, at best.

However, that's the best path I've found so far - and the example I pointed to seems to be the best yet. My problem is that as examples go, it was missing some key pieces for me - my history didn't look like it "should", and I couldn't figure out how to synchronize what I was seeing with what I was "supposed" to see since the example was so terse.

[noidi]

I had the same problem. I removed my submodule with git rm and created a subrepo with the same name with git subrepo clone. The first git subrepo push pushed both those commits to the subrepo, creating one commit that deletes all files and another that adds them back.

As far as I can tell, the problem is caused by git subrepo merge-base thinking that the commit removing the submodule (made before git subrepo clone) is a common ancestor with the subrepo. I haven't delved into git-subrepo's inner workings, so this may be a dumb idea, but maybe git-subrepo merge base should stop at the "git subrepo clone" commit and not look further back for common ancestors?

[zhimsel]

I'm getting the same issue as @noidi and @rbellamy. After removing a previous subtree, and cloning a subrepo in the same place, git subrepo merge-base thinks the previous history from before the subtree was removed is a common ancestor. This causes any subrepo pull to fail. Should this be turned into a new issue?

[zhimsel]

The following output is the git history from a freshly clone subrepo, after git rm -rf'ing the subtree, and subrepo cloneing where it once was:

f1a948d 2016-08-08 14:46 Zach Himsel ● [refs/subrepo/lib/fzf/branch] [subrepo/lib/fzf] git subrepo clone git@github.com:junegunn/fzf.git lib/fzf
8700682 2016-08-08 14:41 Zach Himsel ● Remove existing subtrees in prep for subrepos
8f6b7b2 2016-08-02 11:26 Zach Himsel ● Merge commit '28f5fdc9329c8b851e4fc25754b27e93b2a8b11e'
87756b6 2016-07-26 12:30 Zach Himsel ● Merge commit '3e58886b736c6ca7f34b999b94bf360fc88b2b7a'
c27a336 2016-07-19 12:02 Zach Himsel ● Merge commit '05b553f20e820f24ab928fce9aa27cc7230f5e25'
bb83818 2016-07-11 12:24 Zach Himsel ● Merge commit '4c1add8315573835d160dddaa3813b773049f82d'
9e8bb8f 2016-07-05 13:50 Zach Himsel ● Merge commit '409013a5a648f928f7639cbac949ccdec2804eab'
7322880 2016-06-21 11:37 Zach Himsel ● Merge commit '130eb810f04de855a90cc35aa974716c2775980a'
7a6138a 2016-06-14 12:47 Zach Himsel ● Merge commit '565c405ecfc5a83d586273ba4fbf4751f1ca2fcb'
3e1497c 2016-06-06 15:35 Zach Himsel ● Merge commit '4209bc4f6eeab752d2485f02cfdd9ed1cf4d86a5'
d87fd8f 2016-05-29 20:31 Zach Himsel ● Merge commit '1323ecfcd57dd77b5acd763790f0b39f58391fe5'
055d43c 2016-05-17 11:32 Zach Himsel ● Merge commit 'efa903e6787c8e15930fb5163822ffd73b9f337b' into refactor
7a06c53 2016-05-05 01:31 Zach Himsel ◎ Merge commit '5447dedef8e69b5b7bf555775573e0dd9f86afcf' as 'lib/fzf'

It should look like this (cloned the same repo into a different folder):

f9c4185 2016-08-17 14:41 Zach Himsel ◎ [refs/subrepo/lib/fzf1/branch] [subrepo/lib/fzf1] git subrepo clone git@github.com:junegunn/fzf.git lib/fzf1

It seems that subrepo is picking up on the old subtree history and preventing a clean rebase when pulling upstream changes.

This history is available for playing around with at https://github.com/zhimsel/dotfiles (go ahead and clone it and try it yourself)

[zhimsel]

Ok, I was able to find a way to fix this, at least on a per-subrepo basis (this will need to be done any subrepo that is affected by this issue). You'll need to download the git-reparent script.

# Reparent the subrepo branch:
$ git subrepo fetch <subdir>
$ git subrepo branch <subdir>
$ git checkout subrepo/<subdir>
$ git-reparent --no-parent
$ git checkout -

# At this point, you can follow the usual pull steps:
$ git subrepo merge-base subrepo/<subdir>/fetch subrepo/<subdir>
$ git rebase --onto <new_parent> <old_parent> subrepo/<subdir>
$ git rebase subrepo/<subdir>/fetch subrepo/<subdir>
$ git checkout ORIG_HEAD
$ git subrepo commit <subdir>

I tested this working for the "next" pull by manually changing the <subdir>/fetch ref to a commit before the actual commit at the remote's HEAD. I then did a regular git subrepo pull and it worked fine.

[drewm1980]

I can confirm this issue; I converted a bunch of git submodules to subrepos months ago, finally got around to peeking at their histories, only to discover they were broken/polluted by this issue.

At the time of the conversion I was (and still am since everything seemed great) on git-subrepo version:

$ git log -1
commit a7ee886e0260e847dea6240eaa6278fb2f23be8a
Author: James Z.M. Gao <gaozhiming@360.cn>
Date:   Mon Feb 6 09:55:08 2017 +0800

    Try to skip 'tr' processes while normalizing or encoding

I know this is pre 1.0 software and there will be issues, but it seems like this would impact almost every potential git-subrepo user. Any idea what it is about the way we upgraded our subtrees/subrepos that triggered this problem, but presumably didn't for most of the other git-subrepo users?