Sage-Bionetworks / syndccutils

code for managing data coordinating operations through Synapse
7 stars 8 forks source link

Move GEO metadata script to this repo #58

Closed sgosline closed 6 years ago

jaeddy commented 6 years ago

Turns out that merging an entire repo into another repo (and preserving history) is easier than (1) moving a directory from one repo into another, which is easier than (2) moving an individual file from one repo to another...

Here are the steps I used to copy a Python file from another repo: https://github.com/CRI-iAtlas/synapse-utils/blob/master/python/data_manifest.py

... into the scripts folder of this repo.

(Might be worth turning into a script at some point. @kdaily might have better ideas how to do this.)

  1. Starting in the folder for this repo, move up to the parent directory and create a temp folder; cd into that folder.

    jaeddy:~/code/github/projects/syndccutils$ cd ..
    jaeddy:~/code/github/projects$ mkdir temp
    jaeddy:~/code/github/projects$ cd temp/
  2. Clone the CRI-iAtlas/synapse-utils repo into a new local folder under temp; cd into synapse-utils.

    jaeddy:~/code/github/projects/temp$ git clone https://github.com/CRI-iAtlas/synapse-utils
    Cloning into 'synapse-utils'...
    remote: Counting objects: 35, done.
    remote: Total 35 (delta 0), reused 0 (delta 0), pack-reused 35
    Unpacking objects: 100% (35/35), done.
    jaeddy:~/code/github/projects/temp$ cd synapse-utils/
  3. Locate the file I want to move (python/data_manifest.py).

    jaeddy:~/code/github/projects/temp/synapse-utils$ ls
    README.md            nbs/                 python/              synapse-utils.Rproj
    jaeddy:~/code/github/projects/temp/synapse-utils$ ls python/
    data_manifest.py
  4. Filter the branch to only keep files matching the one I want to move (this is where things get ugly...); note: good to remove the origin remote to avoid accidentally pushing changes to the CRI-iAtlas/synapse-utils repo.

    
    jaeddy:~/code/github/projects/temp/synapse-utils$ git remote rm origin # 4 safety
    jaeddy:~/code/github/projects/temp/synapse-utils$ git filter-branch --index-filter 'git read-tree --empty; git reset $GIT_COMMIT -- python/data_manifest.py' -- --all -- python/data_manifest.py
    Rewrite cd06a2e77d805cf151ebb7a766fd2b20557e2d76 (1/3) (0 seconds passed, remaining 0 predicted)    Unstaged changes after reset:
    D   python/data_manifest.py
    Rewrite 4e5451395b5c5b79c00442a0620b0c785a6b56ec (2/3) (1 seconds passed, remaining 0 predicted)    Unstaged changes after reset:
    D   python/data_manifest.py
    Rewrite 1efbede4c9ece810514436d220dbf6638e1cc52e (2/3) (1 seconds passed, remaining 0 predicted)    Unstaged changes after reset:
    D   python/data_manifest.py

Ref 'refs/heads/master' was rewritten Ref 'refs/remotes/origin/master' was rewritten WARNING: Ref 'refs/remotes/origin/master' is unchanged


5. Check the results... File is still within the `python` folder; let's fix that too.
```console
jaeddy:~/code/github/projects/temp/synapse-utils$ ls
python/
  1. Filter again: only keep files (just one because of step 4) in the python folder and move them to top level.

    jaeddy:~/code/github/projects/temp/synapse-utils$ git filter-branch -f --subdirectory-filter python/ -- --all
    Rewrite 504cc9d7cf40321c19c25f0732f4abaf242896c7 (3/3) (0 seconds passed, remaining 0 predicted)
    Ref 'refs/heads/master' was rewritten
    Ref 'refs/remotes/origin/master' was rewritten
    WARNING: Ref 'refs/remotes/origin/master' is unchanged
  2. Check results... Success!

    jaeddy:~/code/github/projects/temp/synapse-utils$ ls
    data_manifest.py
  3. Switch back to the syndccutils folder.

    jaeddy:~/code/github/projects/temp/synapse-utils$ cd ../../syndccutils/
  4. Set up a remote named temp pointing to the relative path of synapse-utils; fetch the remote.

    jaeddy:~/code/github/projects/syndccutils$ git remote add temp ../temp/synapse-utils/
    jaeddy:~/code/github/projects/syndccutils$ git fetch temp
    warning: no common commits
    remote: Counting objects: 9, done.
    remote: Compressing objects: 100% (6/6), done.
    remote: Total 9 (delta 2), reused 0 (delta 0)
    Unpacking objects: 100% (9/9), done.
    From ../temp/synapse-utils
    + fed86c7...8bcae2e master     -> temp/master  (forced update)
  5. Create a new branch named temp that tracks our new remote "repo".

    jaeddy:~/code/github/projects/syndccutils$ git branch temp remotes/temp/master
    Branch temp set up to track remote branch master from temp.
  6. Merge the temp branch into `master.

    jaeddy:~/code/github/projects/syndccutils$ git merge --allow-unrelated-histories temp
    Merge made by the 'recursive' strategy.
    data_manifest.py | 123 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
    1 file changed, 123 insertions(+)
    create mode 100644 data_manifest.py
  7. Check results... Success!

    jaeddy:~/code/github/projects/syndccutils$ ls
    DESCRIPTION           README.md             data/                 mock_datatable_files/ scripts/
    NAMESPACE             Rmd/                  data_manifest.py      portal-dashboard/     syndccutils.Rproj
    R/                    config/               mock_chart_files/     python/               tests/
    ~/code/github/projects/syndccutils$ git status
    On branch master
    Your branch is ahead of 'origin/master' by 4 commits.
    (use "git push" to publish your local commits)
  8. Move the new file to the scripts folder and push changes.

    jaeddy:~/code/github/projects/syndccutils$ mv data_manifest.py scripts/
    jaeddy:~/code/github/projects/syndccutils$ git add scripts/data_manifest.py
    jaeddy:~/code/github/projects/syndccutils$ git commit -m "stash data_manifest.py in scripts dir"
    [master 7be09c2] stash data_manifest.py in scripts dir
    1 file changed, 123 insertions(+)
    create mode 100644 scripts/data_manifest.py
    jaeddy:~/code/github/projects/syndccutils$ git push
    Counting objects: 14, done.
    Delta compression using up to 8 threads.
    Compressing objects: 100% (11/11), done.
    Writing objects: 100% (14/14), 2.83 KiB | 966.00 KiB/s, done.
    Total 14 (delta 4), reused 0 (delta 0)
    remote: Resolving deltas: 100% (4/4), completed with 1 local object.
    To https://github.com/Sage-Bionetworks/syndccutils
    cfa7e2b..7be09c2  master -> master
  9. Clean up: remove the temp remote.

    jaeddy:~/code/github/projects/syndccutils$ git remote rm temp