search

OWASP / owasp-mastg

The Mobile Application Security Testing Guide (MASTG) is a comprehensive manual for mobile app security testing and reverse engineering. It describes the technical processes for verifying the controls listed in the OWASP Mobile Application Security Verification Standard (MASVS).
https://mas.owasp.org/
Creative Commons Attribution Share Alike 4.0 International
11.48k stars 2.26k forks source link

Clean Git Repo #1050

Open sushi2k opened 5 years ago

sushi2k commented 5 years ago

Describe the issue The github repo is using already 374MB. We should clean it:

https://stackoverflow.com/questions/2100907/how-to-remove-delete-a-large-file-from-commit-history-in-git-repository

sushi2k commented 5 years ago

There are a lot of MSTG releases checked in (script from https://stackoverflow.com/a/42544963). This would need to be removed

$ git rev-list --objects --all \
| git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' \
| sed -n 's/^blob //p' \
| sort --numeric-sort --key=2 \
| cut -c 1-12,41- \
| $(command -v gnumfmt || echo numfmt) --field=2 --to=iec-i --suffix=B --padding=7 --round=nearest | grep "MiB"
eea7fd0b4d29  1.1MiB Generated/MSTG-eng.mobi
662dc0b9f336  1.2MiB Tools/base_images/mstg-cover.png
c2f81fda048a  1.2MiB Tools/base_images/mstg-cover.png
fb88a530efe0  1.2MiB Tools/base_images/mstg-cover.png
3c18aa8f1d3a  1.2MiB Tools/base_images/mstg-cover.png
113607a570d1  1.3MiB Crackmes/Android/Level_03/UnCrackable-Level3.apk
fd56af789bfa  1.3MiB Crackmes/Android/Level_03/UnCrackable-Level3.apk
e1009063fbc9  1.3MiB Crackmes/Android/Level_03/UnCrackable-Level3.apk
294b7b6347c0  1.4MiB Crackmes/Android/Level_03/UnCrackable-Level3.apk
c9e11b6748de  1.4MiB OMTG-Files/03_Examples/01_Android/01_HelloWorld-JNI/HelloWorld-JNI.apk
afdc6f54e1df  1.4MiB Samples/Android/01_HelloWorld-JNI/HelloWord-JNI.apk
d956dd86cb59  1.5MiB OMTG-Files/02_Crackmes/01_Android/Level_03/UnCrackable-Level3.apk
f083fe1172cb  1.5MiB OMTG-Files/02_Crackmes/01_Android/Level_03/UnCrackable-Level3.apk
e30352cf97c2  1.5MiB Crackmes/Android/Level_03/UnCrackable-Level3.apk
d94bb3a9caaf  1.7MiB Tools/base_images/mstg-cover.xcf
8b7f92390e0a  1.8MiB Generated/MSTG-eng.pdf
f784584fadda  1.8MiB Tools/base_images/mstg-cover.xcf
871d210b0217  1.8MiB Tools/base_images/mstg-cover.xcf
6224e007f469  1.8MiB Tools/base_images/mstg-cover-hugo.xcf
0b37fcc92bc0  1.8MiB OMTG-Files/03_Examples/01_Android/01_HelloWorld-JNI/HelloWorld-JNI.apk
57e649f0852b  2.0MiB Generated/MSTG.pdf
f9ad46bc7a80  3.5MiB OMTG-Files/Download/apkx.tgz
75d6959d9b82  3.5MiB OMTG-Files/Download/apkx-0.9.tgz
df213a1388b2  3.5MiB OMTG-Files/Download/apkx-0.9.tgz
c0ccd9ae4f00  3.5MiB OMTG-Files/01_Tools/01_Android/01_apkx/apkx-0.9.tgz
e86687f2b8bc  3.5MiB OMTG-Files/Download/apkx-0.9.tgz
eb732d32fb05  3.9MiB OMTG-Files/01_Tools/01_Android/01_apkx/apkx-libs.jar
de15a7a6083a  4.3MiB Tools/base_images/mstg-cover-hugo.psd
5eb7d1823088  5.4MiB Samples/Android/02_CertificatePinning/certificatePinningXamarin.apk
db20645ff816  7.1MiB OMTG-Files/Download/apkx.tgz
c3ee203c327f  7.3MiB Generated/MSTG-eng.epub
c6bb8ee75ba4  7.3MiB Generated/MSTG-eng.epub
7f720c95d02b  7.3MiB Generated/MSTG-eng.epub
d32e0b7d56ea  7.3MiB Generated/MSTG-eng.epub
e57065feb96b  7.3MiB Generated/MSTG-eng.epub
7a39223e2ef3  7.3MiB Generated/MSTG-eng.epub
23484416704d  7.3MiB Generated/MSTG-eng.mobi
d786c5ea5aa1  7.3MiB Generated/MSTG-eng.mobi
7e38f032d455  7.3MiB Generated/MSTG-eng.mobi
cbcb0128e036  7.3MiB Generated/MSTG-eng.mobi
3ac9b80b56df  7.3MiB Generated/MSTG-eng.mobi
64322fcde218  7.4MiB Generated/MSTG-eng.mobi
b0f5b6df39c0  7.5MiB Generated/MSTG-EN.mobi
930709c7d183  7.7MiB Generated/MSTG-EN.epub
c253c5d9436e  8.8MiB Tools/MSTG_tpl.docx
f252841672eb  8.8MiB Tools/reference.docx
1fca076cd0e6  9.1MiB Generated/MSTG-EN.mobi
54354c955820  9.1MiB Generated/MSTG-EN.mobi
e657b7e93b03  9.1MiB Generated/MSTG-EN.mobi
7598860a7fe7  9.1MiB Generated/MSTG-EN.mobi
75a49adc162c   10MiB Generated/MSTG.pdf
69af985f4d29   11MiB Generated/MSTG-EN.pdf
63fee440a4b7   11MiB Generated/MSTG-EN.pdf
466f884d8d71   11MiB Generated/MSTG-EN.pdf
9d8fb723748c   11MiB Generated/MSTG-EN.pdf
43bbaa68f2fa   11MiB Generated/MSTG-EN.pdf
f56f670cade0   11MiB Generated/MSTG-EN.pdf
96e1a3a3f87a   11MiB Generated/MSTG-EN.pdf
7c591b1450bd   11MiB Generated/MSTG-EN.pdf
bf4e39b9371d   11MiB Generated/MSTG-eng.pdf
c17f0db3de5e   11MiB Generated/MSTG-eng.pdf
3b64133ef809   11MiB Generated/MSTG-eng.pdf
42103a6e8f8d   11MiB Generated/MSTG-eng.pdf
5c54eabdf645   11MiB Generated/MSTG-eng.pdf
d136a1ebd9ea   11MiB Generated/MSTG-eng.pdf
e4a424c68876   12MiB Generated/MSTG-EN.epub
68063611e4f4   12MiB Generated/MSTG-EN.epub
3269f50097c8   12MiB Generated/MSTG-EN.epub
42b781a5d6c5   12MiB Generated/MSTG-EN.epub
4523bd646d47   12MiB Generated/MSTG-EN.pdf
dcdb9f19c98f   15MiB Generated/MSTG-EN.pdf
2ab48b5a56a0   15MiB Generated/MSTG-EN.pdf
9e6fb7c02e53   15MiB Generated/MSTG-EN.pdf
ea1b6bc13f15   15MiB Generated/MSTG-EN.pdf
7e41a142886c   16MiB Generated/MSTG-EN.docx
b1ef232f6394   16MiB Generated/MSTG-EN.docx
dd4f1916190a   16MiB Generated/MSTG-EN.docx
f81918418d56   16MiB Generated/MSTG-EN.docx
f7afbcc27574   19MiB Generated/MSTG-EN.docx
68cd3a62c0c6   19MiB Generated/MSTG-EN.docx
sushi2k commented 5 years ago

This seems to be the way to go to remove the majority of the large files https://rtyley.github.io/bfg-repo-cleaner/

https://stackoverflow.com/a/17890278

Let's discuss during the summit, haven't done this before :-)

commjoen commented 4 years ago

cleaning it up, will cost releases. shall we close this or do you want to clean up specific documents still for specific commits?

cpholguera commented 3 years ago

Let's give it another try!

sushi2k commented 3 years ago

List of files over 1MB in git:

git rev-list --objects --all \
| git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' \
| sed -n 's/^blob //p' \
| sort --numeric-sort --key=2 \
| cut -c 1-12,41- \
| $(command -v gnumfmt || echo numfmt) --field=2 --to=iec-i --suffix=B --padding=7 --round=nearest | grep "MiB"
eea7fd0b4d29  1.1MiB Generated/MSTG-eng.mobi
662dc0b9f336  1.2MiB Tools/base_images/mstg-cover.png
c2f81fda048a  1.2MiB Tools/base_images/mstg-cover.png
fb88a530efe0  1.2MiB Tools/base_images/mstg-cover.png
3c18aa8f1d3a  1.2MiB Tools/base_images/mstg-cover.png
17c5eb4b86e1  1.2MiB tools/base_images/mstg-cover.png
113607a570d1  1.3MiB Crackmes/Android/Level_03/UnCrackable-Level3.apk
fd56af789bfa  1.3MiB Crackmes/Android/Level_03/UnCrackable-Level3.apk
e1009063fbc9  1.3MiB Crackmes/Android/Level_03/UnCrackable-Level3.apk
d75d1ee5edb2  1.3MiB Document/cover.jpg
48e892c4a13f  1.4MiB Crackmes/Android/Level_03/UnCrackable-Level3.apk
294b7b6347c0  1.4MiB Crackmes/Android/Level_03/UnCrackable-Level3.apk
c9e11b6748de  1.4MiB OMTG-Files/03_Examples/01_Android/01_HelloWorld-JNI/HelloWorld-JNI.apk
afdc6f54e1df  1.4MiB Samples/Android/01_HelloWorld-JNI/HelloWord-JNI.apk
d956dd86cb59  1.5MiB OMTG-Files/02_Crackmes/01_Android/Level_03/UnCrackable-Level3.apk
f083fe1172cb  1.5MiB OMTG-Files/02_Crackmes/01_Android/Level_03/UnCrackable-Level3.apk
e30352cf97c2  1.5MiB Crackmes/Android/Level_03/UnCrackable-Level3.apk
d94bb3a9caaf  1.7MiB Tools/base_images/mstg-cover.xcf
8b7f92390e0a  1.8MiB Generated/MSTG-eng.pdf
f784584fadda  1.8MiB Tools/base_images/mstg-cover.xcf
871d210b0217  1.8MiB Tools/base_images/mstg-cover.xcf
6224e007f469  1.8MiB tools/base_images/mstg-cover-hugo.xcf
81891ecacd6e  1.8MiB tools/base_images/mstg-cover.xcf
0b37fcc92bc0  1.8MiB OMTG-Files/03_Examples/01_Android/01_HelloWorld-JNI/HelloWorld-JNI.apk
57e649f0852b  2.0MiB Generated/MSTG.pdf
8a2b8b03598f  2.1MiB Document/Images/cover.jpg
0af54bebe422  2.5MiB overflows_fix_images_microtype.txt
366b8b46382b  2.8MiB overflows_master.txt
3e9ac49e6e7e  3.5MiB Crackmes/Android/Level_04/r2pay-v0.9.apk
f9ad46bc7a80  3.5MiB OMTG-Files/Download/apkx.tgz
75d6959d9b82  3.5MiB OMTG-Files/Download/apkx-0.9.tgz
df213a1388b2  3.5MiB OMTG-Files/Download/apkx-0.9.tgz
c0ccd9ae4f00  3.5MiB OMTG-Files/01_Tools/01_Android/01_apkx/apkx-0.9.tgz
e86687f2b8bc  3.5MiB OMTG-Files/Download/apkx-0.9.tgz
eb732d32fb05  3.9MiB OMTG-Files/01_Tools/01_Android/01_apkx/apkx-libs.jar
dc4eb2f2107d  4.1MiB Crackmes/Android/Level_04/r2pay-v1.0.apk
de15a7a6083a  4.3MiB tools/base_images/mstg-cover-hugo.psd
5eb7d1823088  5.4MiB Samples/Android/02_CertificatePinning/certificatePinningXamarin.apk
db20645ff816  7.1MiB OMTG-Files/Download/apkx.tgz
c3ee203c327f  7.3MiB Generated/MSTG-eng.epub
c6bb8ee75ba4  7.3MiB Generated/MSTG-eng.epub
7f720c95d02b  7.3MiB Generated/MSTG-eng.epub
d32e0b7d56ea  7.3MiB Generated/MSTG-eng.epub
e57065feb96b  7.3MiB Generated/MSTG-eng.epub
7a39223e2ef3  7.3MiB Generated/MSTG-eng.epub
23484416704d  7.3MiB Generated/MSTG-eng.mobi
d786c5ea5aa1  7.3MiB Generated/MSTG-eng.mobi
7e38f032d455  7.3MiB Generated/MSTG-eng.mobi
cbcb0128e036  7.3MiB Generated/MSTG-eng.mobi
3ac9b80b56df  7.3MiB Generated/MSTG-eng.mobi
64322fcde218  7.4MiB Generated/MSTG-eng.mobi
b0f5b6df39c0  7.5MiB Generated/MSTG-EN.mobi
930709c7d183  7.7MiB Generated/MSTG-EN.epub
c253c5d9436e  8.8MiB Tools/MSTG_tpl.docx
f252841672eb  8.8MiB tools/old/reference.docx
1fca076cd0e6  9.1MiB Generated/MSTG-EN.mobi
54354c955820  9.1MiB Generated/MSTG-EN.mobi
e657b7e93b03  9.1MiB Generated/MSTG-EN.mobi
7598860a7fe7  9.1MiB Generated/MSTG-EN.mobi
75a49adc162c   10MiB Generated/MSTG.pdf
69af985f4d29   11MiB Generated/MSTG-EN.pdf
63fee440a4b7   11MiB Generated/MSTG-EN.pdf
466f884d8d71   11MiB Generated/MSTG-EN.pdf
9d8fb723748c   11MiB Generated/MSTG-EN.pdf
43bbaa68f2fa   11MiB Generated/MSTG-EN.pdf
f56f670cade0   11MiB Generated/MSTG-EN.pdf
96e1a3a3f87a   11MiB Generated/MSTG-EN.pdf
7c591b1450bd   11MiB Generated/MSTG-EN.pdf
bf4e39b9371d   11MiB Generated/MSTG-eng.pdf
c17f0db3de5e   11MiB Generated/MSTG-eng.pdf
3b64133ef809   11MiB Generated/MSTG-eng.pdf
42103a6e8f8d   11MiB Generated/MSTG-eng.pdf
5c54eabdf645   11MiB Generated/MSTG-eng.pdf
d136a1ebd9ea   11MiB Generated/MSTG-eng.pdf
e4a424c68876   12MiB Generated/MSTG-EN.epub
68063611e4f4   12MiB Generated/MSTG-EN.epub
3269f50097c8   12MiB Generated/MSTG-EN.epub
42b781a5d6c5   12MiB Generated/MSTG-EN.epub
4523bd646d47   12MiB Generated/MSTG-EN.pdf
dcdb9f19c98f   15MiB Generated/MSTG-EN.pdf
2ab48b5a56a0   15MiB Generated/MSTG-EN.pdf
9e6fb7c02e53   15MiB Generated/MSTG-EN.pdf
ea1b6bc13f15   15MiB Generated/MSTG-EN.pdf
5a4cfcbc193c   16MiB OWASP_MSTG-1.2-en.mobi
7e41a142886c   16MiB Generated/MSTG-EN.docx
b1ef232f6394   16MiB Generated/MSTG-EN.docx
dd4f1916190a   16MiB Generated/MSTG-EN.docx
f81918418d56   16MiB Generated/MSTG-EN.docx
f7afbcc27574   19MiB Generated/MSTG-EN.docx
68cd3a62c0c6   19MiB Generated/MSTG-EN.docx
546de07ee2e5   26MiB OWASP_MSTG-1.3.pdf
be667b921761   26MiB OWASP_MSTG-1.2-en.pdf
52d3ff75f2ab   26MiB OWASP_MSTG-1.2-en.pdf
fc7cffcf00e9   27MiB OWASP_MSTG-1.3_WIP_.docx
4442600cf70e   27MiB OWASP_MSTG-1.2-en_WIP_.docx
b380677cf601   28MiB OWASP_MSTG-1.3.epub
d8246e12d2ff   29MiB OWASP_MSTG-1.2-en.epub

In .git we have 417MB

$ owasp-mstg.git ❯ du -hc
417M    ./objects/pack
  0B    ./objects/info
417M    ./objects
4.0K    ./info
 60K    ./hooks
  0B    ./refs/heads
  0B    ./refs/tags
  0B    ./refs
417M    .
417M    total
sushi2k commented 3 years ago

First test with a fork of the MSTG:

owasp-mstg-fork-git-cleanup ❯ java -jar ~/Downloads/bfg-1.14.0.jar --strip-blobs-bigger-than 7M owasp-mstg.git

Using repo : /Users/sven/PentestTools/OWASP/owasp-mstg-fork-git-cleanup/owasp-mstg.git

Scanning packfile for large blobs: 32170
Scanning packfile for large blobs completed in 422 ms.
Found 59 blob ids for large blobs - biggest=29947485 smallest=7399457
Total size (unpacked)=783218484
Found 354 objects to protect
Found 22 commit-pointing refs : HEAD, refs/heads/1402-1174-oauth, refs/heads/DarioI-review/android-platform-overview, ...
Found 6 tag-pointing refs : refs/tags/1.0.1, refs/tags/1.0.2, refs/tags/1.1.0, ...

Protected commits
-----------------

These are your protected commits, and so their contents will NOT be altered:

 * commit 7f1220cf (protected by 'HEAD') - contains 1 dirty file :
    - tools/reference.docx (8.8 MB)

WARNING: The dirty content above may be removed from other commits, but as
the *protected* commits still use it, it will STILL exist in your repository.

Details of protected dirty content have been recorded here :

/Users/sven/PentestTools/OWASP/owasp-mstg-fork-git-cleanup/owasp-mstg.git.bfg-report/2021-07-23/06-32-29/protected-dirt/

If you *really* want this content gone, make a manual commit that removes it,
and then run the BFG on a fresh copy of your repo.

Cleaning
--------

Found 7693 commits
Cleaning commits:       100% (7693/7693)
Cleaning commits completed in 6,110 ms.

Updating 27 Refs
----------------

    Ref                                                  Before     After
    ------------------------------------------------------------------------
    refs/heads/1402-1174-oauth                         | 187dd3bb | d4554caa
    refs/heads/DarioI-review/android-platform-overview | 8d235a46 | 5c0a72c3
    refs/heads/add-webviews-cleanup                    | 4576fbe4 | 7f24a245
    refs/heads/consistency-for-storage-4-and-7         | a4139438 | 69512998
    refs/heads/controls-refactor                       | 9362662d | a875016e
    refs/heads/cpholguera-patch-1                      | 217ceff4 | 8ef0fa8f
    refs/heads/cpholguera-patch-2                      | dd11c624 | 75044de7
    refs/heads/cpholguera-patch-4                      | 03b16842 | 348e6eac
    refs/heads/fix_slack_link                          | 9c01a6f2 | 96f039fa
    refs/heads/master                                  | 7f1220cf | 23173ea8
    refs/heads/new_release                             | ab1e0ec5 | 2e10e57d
    refs/heads/optimize-link-checker                   | 882e8a82 | c15e9973
    refs/heads/origin/controls-refactor                | bca3f9df | 6c111574
    refs/heads/remove_hash_of_checklist                | c7e8178e | 2a16e964
    refs/heads/sushi2k-patch-1                         | 46bcf446 | e86e8412
    ...

Updating references:    100% (27/27)
...Ref update completed in 49 ms.

Commit Tree-Dirt History
------------------------

    Earliest                                              Latest
    |                                                          |
    ............DmmmmmmmmmmmmmmmmmDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD

    D = dirty commits (file tree fixed)
    m = modified commits (commit message or parents changed)
    . = clean commits (no changes to file tree)

                            Before     After
    -------------------------------------------
    First modified commit | d222e475 | 3becfd95
    Last dirty commit     | ab1e0ec5 | 2e10e57d

Deleted files
-------------

    Filename                      Git id
    -------------------------------------------------------------------------
    MSTG-EN.docx                | dd4f1916 (16.4 MB), 7e41a142 (16.4 MB), ...
    MSTG-EN.epub                | 42b781a5 (11.7 MB), 3269f500 (11.7 MB), ...
    MSTG-EN.mobi                | 7598860a (9.1 MB), e657b7e9 (9.1 MB), ...
    MSTG-EN.pdf                 | 2ab48b5a (14.8 MB), 63fee440 (10.7 MB), ...
    MSTG-eng.epub               | c6bb8ee7 (7.3 MB), e57065fe (7.3 MB), ...
    MSTG-eng.mobi               | 23484416 (7.3 MB), 7e38f032 (7.3 MB), ...
    MSTG-eng.pdf                | 3b64133e (11.3 MB), 5c54eabd (11.4 MB), ...
    MSTG.pdf                    | 75a49adc (10.5 MB)
    MSTG_tpl.docx               | c253c5d9 (8.8 MB)
    OWASP_MSTG-1.2-en.epub      | d8246e12 (28.6 MB)
    OWASP_MSTG-1.2-en.mobi      | 5a4cfcbc (16.2 MB)
    OWASP_MSTG-1.2-en.pdf       | be667b92 (26.3 MB), 52d3ff75 (26.3 MB)
    OWASP_MSTG-1.2-en_WIP_.docx | 4442600c (27.5 MB)
    OWASP_MSTG-1.3.epub         | b380677c (28.5 MB)
    OWASP_MSTG-1.3.pdf          | 546de07e (25.8 MB)
    ...

In total, 9765 object ids were changed. Full details are logged here:

    /Users/sven/PentestTools/OWASP/owasp-mstg-fork-git-cleanup/owasp-mstg.git.bfg-report/2021-07-23/06-32-29

BFG run is complete! When ready, run: git reflog expire --expire=now --all && git gc --prune=now --aggressive
owasp-mstg-fork-git-cleanup ❯ cd owasp-mstg.git                                                                                                           8s
owasp-mstg-fork-git-cleanup/owasp-mstg.git ❯git reflog expire --expire=now --all && git gc --prune=now --aggressive
Enumerating objects: 32172, done.
Counting objects: 100% (32172/32172), done.
Delta compression using up to 8 threads
Compressing objects: 100% (31930/31930), done.
Writing objects: 100% (32172/32172), done.
Selecting bitmap commits: 7600, done.
Building bitmaps: 100% (180/180), done.
Total 32172 (delta 23721), reused 2693 (delta 0), pack-reused 0
sushi2k commented 3 years ago

Instead of 417MB it's now 130MB (for .git only)

owasp-mstg-fork-git-cleanup/owasp-mstg.git ❯ du -hc                                                                                                      37s
129M    ./objects/pack
428K    ./objects/info
130M    ./objects
8.0K    ./info
 60K    ./hooks
  0B    ./refs/heads
  0B    ./refs/tags
  0B    ./refs
130M    .
130M    total