We need a utility that can transfer history from one repo to another, including file renames

[samreid]

In https://github.com/phetsims/vegas/issues/101#issuecomment-1093479022 I wrote:

I observed that the command ../perennial/bin/copy-history-to-different-repo.sh js/StatusBar.ts ../scenery-phet/ failed to get the history prior to the file rename: Here is the history in vegas:

Here is the history in scenery-phet after the copy:

@zepumph or @pixelzoom any idea how to get the history across that file rename?

Later, I added:

I tried making up my own multi-script using the patches idea, like so:

git log --pretty=email --patch-with-stat --reverse --full-index --binary -- js/StatusBar.ts > ../patches/patch1.txt
git log --pretty=email --patch-with-stat --reverse --full-index --binary -- js/StatusBar.js > ../patches/patch2.txt

cd ../scenery-phet
git am --whitespace=fix --reject < ../patches/patch1.txt
git am --whitespace=fix --reject < ../patches/patch2.txt

However, this failed and I also tried applying patch2 then patch1 and it failed for a different reason. In one case, I ended up with a TS file and a JS file, where the JS file had some history, but still ended with define() implementations.

[samreid]

In https://github.com/phetsims/vegas/issues/101#issuecomment-1093659193 I said:

I have a proposed plan that I think will work well for this issue and future cases where we want to move history from one repo to another more robustly.

1. Install the command git filter-repo. It is recommended by the git documentation as an improvement over git filter-branch. https://git-scm.com/docs/git-filter-branch#_warning. I used git --exec-path to see the path for auxiliary git commands. On my platform it was /Library/Developer/CommandLineTools/usr/libexec/git-core

2. Clone another copy of the repo we are copying from. In this case, it is vegas.

   git clone https://github.com/phetsims/vegas.git vegas-backup

3. Filter the repo so it only contains the files of interest:

cd vegas-backup
git filter-repo --path js/StatusBar.js --path js/StatusBar.ts

4. Remove unwanted branches from the source repo. For example, remove all branches except master.

git branch | grep -v "master" | xargs git branch -D 

5. Inspect the filtered source repo to make sure it is what we want to merge to the target repo.
6. Merge the source repo into the target repo. This can be done fully locally, using a local directory for git remote add. Follow directions for doing a local repo merge, like: https://stackoverflow.com/questions/56051560/how-to-merge-two-git-repositories-without-loosing-the-history-for-either-reposit which is basically the following. But please use a sensible name for the remote nickname, since it will show in the commit history.

cd ../scenery-phet
git remote add repo-b ../vegas-backup
git fetch repo-b
git merge repo-b/master --allow-unrelated
git remote remove repo-b

8. Inspect the target repo and make sure it now has the desired commits/files/history. Update the namespace and registry statement, if we are still using that. See https://github.com/phetsims/phet-core/issues/100. Type check and test the new code. Preview what files are about to be pushed with git diff --stat --cached origin/master . Then git push.
9. Remove the filtered clone, so you don't accidentally push it later on: rm -rf vegas-backup.
10. Remove the old copy of the source file, commit and push.

[samreid]

I used the strategy proposed in the preceding comment to migrate StatusBar and its history, including file renames. @pixelzoom confirmed the history transfer seemed correct.

This issue is about reviewing the process above and either adding tooling or documentation to simplify that process. @zepumph can you please review and comment?

[zepumph]

Review:

In general this is awesome, good job finding this. I think it should go into documentation in copy-history....sh so that we can use it in the future. A few notes about the steps:

• Can we be explicit and name the different repos instead of using pronouns. It seems like we land on source and target repos eventually, but it takes a bit. It would be helpful to have a name for source and source copy also . (2) says "copy of the repo", (3) says "the repo" but you mean the copy this time, (4) says the "source repo" but doesn't have a cd in the command so I don't really know which repo it is, (9) says "filtered clone".

• For (1) can we add a link to the npm or github, and include the actual npm name git-filter-repo.

• Is there any chance we could be confident enough about this to create a script to do this? I feel like we could add git-filter-repo to perennials package.json without too much worry, and then it could be a new script that implements the above. Perhaps copy-history-for-file-through-rename.sh StatusBar.ts StatusBar.js vegas scenery-phet, and the script can name the remote something general like copy-history-remote.

I'm glad you were able to find this out. This is really awesome for the project and I would love to be able to use this again. I also think we will need it again quite often because of the typescript conversion.

[samreid]

Thanks, @chrisklus and I adapted this procedure into a script. I'll test out the script for https://github.com/phetsims/mean-share-and-balance/issues/5.

The script has not been well vetted and not tested on Windows. In particular, I wonder if forward and backward slashes will cause mixups.

[samreid]

In https://github.com/phetsims/mean-share-and-balance/issues/5#issuecomment-1100471805 I used this script to transfer QuestionBar from center-and-variability to scenery-phet. It appears to be working OK. Leaving self-assigned for testing on Windows.

Also, this script should be reviewed by @chrisklus (who collaborated on the first half) or @zepumph with the caveat that Windows hasn't been tested at all.

[samreid]

Today @zepumph volunteered to review after it's working on Windows. When this is ready, we'll delete the patch-oriented script.

[samreid]

I ran this script on Windows today for moving customizeWrapperTemplateForAction as part of https://github.com/phetsims/phet-io-wrappers/issues/435. I'll add some notes and reassign to @zepumph

[zepumph]

I have not had a chance to run these steps just yet. It is still on my radar for my next repo history move though.

[zepumph]

Working on this as part of https://github.com/phetsims/number-suite-common/issues/64.

[zepumph]

• This line was really scary to me in the boolean prompt:

Please inspect files/history and see if ready for push.

It made it sound like this script would be responsible for pushing if I pressed yes, but it is manual. I originally just ran this script as a test, and was prepared to just blow away the repo if I didn't like the results. I definitely got scared when I saw that prompt, especially since the result just changes console output afterwards.

• For a whole directory, I don't think it would work, and I didn't try, because of conflicts with things like Grunfile and package.json, so we were just doing top level directories at a time. Does that seem right to you?

We ended up not proceeding with https://github.com/phetsims/number-suite-common/issues/64, but I still was able to take this for a test drive.

It ended up working great though for a directory. Here are the commands I would have done if we proceeded

node perennial/js/scripts/copy-history-to-different-repo counting-common/js/ number-suite-common
node perennial/js/scripts/copy-history-to-different-repo counting-common/images/ number-suite-common
node perennial/js/scripts/copy-history-to-different-repo counting-common/mipmaps/ number-suite-common
node perennial/js/scripts/copy-history-to-different-repo counting-common/assets/ number-suite-common

I executed on js/ and it was perfect.

Thanks for implementing this, I will certainly use it again in the future.

I'm ready to close after the above thoughts.

[samreid]

This line was really scary to me in the boolean prompt:

Good idea, I revised that text.

For a whole directory, I don't think it would work, and I didn't try, because of conflicts with things like Grunfile and package.json, so we were just doing top level directories at a time. Does that seem right to you?

I'm not sure what would happen on a conflict. Your idea to go by subdirectory seems reasonable. I never tried merging 2 complete repos.

Want to review my commit? I feel this issue can be closed afterwards.

[zepumph]

I'm not really sure how https://github.com/phetsims/perennial/issues/269#issuecomment-1496556996 works, as I don't seem to have git filter-repo installed on my device now. Perhaps I did and then a new version of git blew it away? Or that comment actually didn't work as well as I thought.

I followed steps in the doc, and then running git filter-repo results in:

 mjkauzmann ~/PHET/git/density-buoyancy-common (main)
 $ git filter-repo
Python was not found; run without arguments to install from the Microsoft Store, or disable this shortcut from Settings > Manage App Execution Aliases.

But just running python and python3 from git bash work as expected into a REPL.

Not sure how to proceed here. Perhaps we can pair on it?

[zepumph]

I got here from https://github.com/phetsims/density-buoyancy-common/issues/364, where I want to move BalloonInteractionCueNode to scenery-phet while keeping history. I'm going to proceed with the shell script for now to unblock me.