lbryio / lbry-android

The LBRY Android app
MIT License
2.43k stars 113 forks source link

Remove large binaries from history #977

Open foooorsyth opened 4 years ago

foooorsyth commented 4 years ago
## The Issue A bunch of large binaries from previous versions of the app are in the packfile. Most of these are from pre-native rewrite builds -- I assume that their value in the git history is pretty low, and they slow down your CI feedback loop considerably because a fresh clone takes quite a while due to file size (The .git folder is half a gig!) ### Steps to reproduce Run this: ``` git rev-list --objects --all \ | git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' \ | sed -n 's/^blob //p' \ | sort --numeric-sort --key=2 \ | cut -c 1-12,41- \ | $(command -v gnumfmt || echo numfmt) --field=2 --to=iec-i --suffix=B --padding=7 --round=nearest ``` See screenshot below of large files If you're on MacOS and do not have numfmt, install coreutils via brew ``` brew install coreutils ``` ### Expected behaviour Large binaries not in source control ### Actual behaviour Large binaries in source control ## System Configuration N/A ## Anything Else Here is a step-by-step on how to clean up the history: 1. Do a fresh clone ``` git clone https://github.com/lbryio/lbry-android.git cd lbry-android ``` 2. Run this script from the project root to track all branches ``` #!/bin/bash for branch in `git branch -a | grep remotes | grep -v HEAD | grep -v master`; do git branch --track ${branch##*/} $branch done ``` 3. filter-branch on the unnecessary binaries from older versions ``` # Old SDK dependencies git filter-branch --tag-name-filter cat --index-filter 'git rm -r --cached --ignore-unmatch app/libs/*.aar' --prune-empty -f -- --all # index.android.bundle (two locations) git filter-branch --tag-name-filter cat --index-filter 'git rm -r --cached --ignore-unmatch src/main/assets/index.android.bundle' --prune-empty -f -- --all git filter-branch --tag-name-filter cat --index-filter 'git rm -r --cached --ignore-unmatch app/src/main/assets/index.android.bundle' --prune-empty -f -- --all # Node package lock git filter-branch --tag-name-filter cat --index-filter 'git rm -r --cached --ignore-unmatch app/package-lock.json' --prune-empty -f -- --all # Old blockchain headers git filter-branch --tag-name-filter cat --index-filter 'git rm -r --cached --ignore-unmatch src/main/assets/blockchain/headers' --prune-empty -f -- --all ``` 4. Clean up the packfile ``` rm -rf .git/refs/original/ git reflog expire --expire=now --all git gc --prune=now git gc --aggressive --prune=now ``` 5. Force push all branches and all tags (need to turn branch protection off temporarily to do this) ``` git push origin --force --all git push origin --force --tags ``` This takes the packfile size down from **537 MB to 11 MB** **All other contributors will need to do a fresh clone of the repo after you force push. If they push from their old local copy, they will re-add all the files that were deleted from the history.** ## Screenshots Screen Shot 2020-08-16 at 5 23 49 PM ## Internal Use ### Acceptance Criteria 1. 2. 3. ### Definition of Done - [ ] Tested against acceptance criteria - [ ] Tested against the assumptions of the user story - [ ] The project builds without errors - [ ] Unit tests are written and passing - [ ] Tests on devices/browsers listed in the issue have passed - [ ] QA performed & issues resolved - [ ] Refactoring completed - [ ] Any configuration or build changes documented - [ ] Documentation updated - [ ] Peer Code Review performed
akinwale commented 4 years ago

Sounds good. Will get this done.

kauffj commented 4 years ago

@aimkey Thanks a lot, please drive by again :grin:. If you email an address to jeremy@lbry.com I will send you some LBC too.