Megan Remains at War with Git

megpritch commented 2 months ago

Mapgen Modularize Update: Megan Remains at War with Git

I believe I have fixed the errors I was getting with mapping the labels in the new fns_mapgen.R, but I am still unable to push my progress to GitHub for others to see.

Here's the type of error I keep getting:

$ git push origin modularize
Enumerating objects: 150, done.
Counting objects: 100% (150/150), done.
Delta compression using up to 16 threads
Compressing objects: 100% (60/60), done.
Writing objects: 100% (139/139), 286.03 MiB | 10.42 MiB/s, done.
Total 139 (delta 39), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (39/39), completed with 8 local objects.
remote: warning: File HARP-2023-Summer/Mapping/Functions/.RData is 84.36 MB; this is larger than GitHub's recommended maximum file size of 50.00 MB
remote: error: Trace: 8b0afcfeb5611269b0e62dbb958ac64f2dace1f2d81ffcf0d7e9142f8602edd0
remote: error: See https://gh.io/lfs for more information.
remote: error: File HARP-2023-Summer/Mapping/.RData is 170.79 MB; this exceeds GitHub's file size limit of 100.00 MB
remote: error: GH001: Large files detected. You may want to try Git Large File SRewrite 91d859433dd1032a1b91b0e43e2c1c0455f17071 (1/2449) (0 seconds passed, remRewrite efdac9c44cbdf9d8dd1e90f1d00ff4f3614ac3c8 (2/2449) (0 seconds passed, remRewrite 431034047b28e8fa926c8687da4779390f3d9922 (3/2449) (1 seconds passed, remRewrite c87e39b27875d859328a9e1f9dc3aab02accf49e (3/2449) (1 seconds passed, remRewrite 5f6008b238002f50fc6a8afec95ead3511812927 (3/2449) (1 seconds passed, remRewrite 3667c3b825c2cb09b7e2e0c07283fac13d1b7bd5 (3/2449) (1 seconds passed, remRewrite cdce19e02c22ed7783bb37785b4cf50c50d9c383 (7/2449) (5 seconds passed, remRewrite 3208b19cfc5894da55bcb54a5aa7a32166973409 (7/2449) (5 seconds passed, remRewrite 77673ed62a812f4746a5c3a6be8f3d5857df3b7d (9/2449) (7 seconds passed, remRewrite 5732ec4f4f193763806fab2c4715ceb419f38653 (9/2449) (7 seconds passed, remRewrite ff48fe689825f0200b6c72fc83019498a05a5c2c (11/2449) (9 seconds passed, reRewrite 37da19771a95cfac22dd20b800e170325511aa86 (11/2449) (9 seconds passed, reRewrite 62a29b36db5a12ab274e08f3b60f284aa161cb79 (13/2449) (10 seconds passed, rRewrite d2048231ed84bc9d8afd72e5a5b201d59ecdffe5 (13/2449) (10 seconds passed, rRewrite d5821a8b0a25f3e7b909fa2980dd5081d8642746 (15/2449) (12 seconds passed, rRewrite e5ace91020c37ee16517e1b437c91f4c6eb97096 (15/2449) (12 seconds passed, rRewrite f9d5c2b433bb082924008e223925f1c431cd3cb3 (17/2449) (14 seconds passed, rRewrite 6c997eefb3ec2c023caf054f1e301dbc4eb693a6 (17/2449) (14 seconds passed, rRewrite f4b6557f5569264cc35f775700c8720ac80618c0 (19/2449) (16 seconds passed, rRewrite 0fbd2d0b8116415a2eb40d2579908ecbce18fbc8 (19/2449) (16 seconds passed, rRewrite 331d0b098f43b2761748d770fc48f815b5ac24c8 (21/2449) (18 seconds passed, rRewrite 6bbea87d36bb744a2fcd96e858e272dedd29537e (21/2449) (18 seconds passed, rRewrite 2813a6336dfa110ba60c71c9d8d3712f378c64c6 (23/24Rewrite 2d592d2c4bb30ff3f72e0aab0ce0c25fe406b107 (1589/2449) (1709 seconds passed, remaining 924 predicted)    error: invalid path 'HARP-2022-Summer/AutomatedScripts/C:/Workspace/tmp/fig.location_map_rivanna8.png'
Could not initialize the index

And here is what I've tried:

1. Deleting the .RData file on my computer and trying to `git add -A` `git commit -a -m` again

2. Both of the commands from the Git Tips and tricks:

git rm --cached /HARP-2023-Summer/Mapping/Functions/.RData
git commit --amend -CHEAD
git push origin modularize

And again with a full file path:

git rm --cached /c/aa_HARP/aa_GitHub/HARParchive/HARP-2023-Summer/Mapping/Functions/.RData
git commit --amend -CHEAD
git push origin modularize

I did the above commands for both of the large files that GitHub was unhappy with (i.e. for HARP-2023-Summer/Mapping/.RData as well). These commands would run for a long time and then ultimately would not work when I tried to push.

Then I tried the other command:

git filter-branch --force --index-filter  
 'git rm -r --cached --ignore-unmatch /HARP-2023-Summer/Mapping/Functions/.RData'   
--prune-empty --tag-name-filter cat -- --all

and this gives the error:

WARNING: git-filter-branch has a glut of gotchas generating mangled history
         rewrites.  Hit Ctrl-C before proceeding to abort, then use an
         alternative filtering tool such as 'git filter-repo'
         (https://github.com/newren/git-filter-repo/) instead.  See the
         filter-branch manual page for more details; to squelch this warning,
         set FILTER_BRANCH_SQUELCH_WARNING=1.
Proceeding with filter-branch...

Cannot rewrite branches: You have unstaged changes.

3. Then I did some googling and tinkered with `git rebase`:

Source: git rebase initial info Source: git rebase Source: git reset

 # to look at last 3 commits:
git rebase -i HEAD~3

The last 3 commits were all replicas of my failed 'modularize progress' commit attempts, so I did this:

 # to delete my last 3 failed commits:
git reset --soft HEAD~3

I thought this git reset would only delete the problematic commits that were in transit to GitHub, and NOT my local edits, but I was wrong. I lost all of my most recent fixes, so I had to go back and re-implement them. Then, I made sure to delete any .RData files before trying to add/commit/push again, but I still received the same push error as before.

Additionally,

I cannot test to make sure I re-implemented all of the edits I lost, because my REST login is now coming back unsuccessful. I made sure to re-update my auth.private with the most recent password, but this did not solve the issue.

The Good News:

After using git reset --soft HEAD~3, the warning for the large .RData file in the /Functions folder is gone:

$ git push origin modularize
Enumerating objects: 49, done.
Counting objects: 100% (49/49), done.
Delta compression using up to 16 threads
Compressing objects: 100% (38/38), done.
Writing objects: 100% (38/38), 202.08 MiB | 10.24 MiB/s, done.
Total 38 (delta 27), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (27/27), completed with 8 local objects.
remote: error: Trace: f37306169175dab6400d0aac6c756eb2f130864c524e390572e87bbb93421db9
remote: error: See https://gh.io/lfs for more information.
remote: error: File HARP-2023-Summer/Mapping/.RData is 170.79 MB; this exceeds GitHub's file size limit of 100.00 MB
remote: error: GH001: Large files detected. You may want to try Git Large File Storage - https://git-lfs.github.com.
To https://github.com/HARPgroup/HARParchive.git
 ! [remote rejected] modularize -> modularize (pre-receive hook declined)
error: failed to push some refs to 'https://github.com/HARPgroup/HARParchive.git'

gmahadwar commented 2 months ago

While this method is not ideal, it has always worked for me and kleiner in a pinch:

1) Commit your existing changes to a branch so that git doesn't complain at step 4 2) Delete .RData (as you've done) and/or 2) Check .RData is in your your .gitignore file as I've pushed up to the master here https://github.com/HARPgroup/HARParchive/blob/master/.gitignore (you may have already done this in the last meeting as well)

3) Copy the files that you've changed to somewhere safe outside of the GitHub folder structure, ex. Desktop. You only need to save the files that you are trying to push the changes for.

4) In git: git fetch origin master git checkout master git -c pull origin master

5) checkout a new branch 6) copy your files saved from your Desktop back into the git repository folder structure on your harddrive. 7) The usual: git add --all git commit -m "[your message]" git -c push origin [branchname]

Please note, this method only works if you have confidence in which files you changed. I think there's a trick to identifying those so if you're unsure, say so, and we can look for that tip.

rburghol commented 2 months ago

@megpritch if you follow @gmahadwar suggestion I am sure this will work. Also, let's all make sure that we have .RData in our .gitignore file!

megpritch commented 2 months ago

@gmahadwar @rburghol Thank you for your help--my updates are finally pushed to GitHub! I wouldn't merge anything just yet, but I'm hoping to have the modularized mapgen functional within the latest summaries script by later today.

HARPgroup / HARParchive