search

kiwix / java-libkiwix

Libkiwix binding for Java & Kotlin
https://central.sonatype.com/artifact/org.kiwix/libkiwix
GNU General Public License v3.0
3 stars 4 forks source link

Remove binary blob from the git history #46

Closed kelson42 closed 1 year ago

kelson42 commented 1 year ago

Someone has achivied to upload blobs in this repository which is now huge. Before releasing we have to clean the whole history to avoid blobs in it.

mgautierfr commented 1 year ago

Of which blobs are you speaking of ?

kelson42 commented 1 year ago

I just have seen the repo was over 100MB

kelson42 commented 1 year ago

I have run BFG on the repostory. Here is the log:

$ java -jar /home/kelson/Downloads/bfg-1.14.0.jar --strip-blobs-bigger-than 1M java-libkiwix.git

Using repo : /home/kelson/code/java-libkiwix.git

Scanning packfile for large blobs: 3559
Scanning packfile for large blobs completed in 52 ms.
Found 41 blob ids for large blobs - biggest=100580908 smallest=13060460
Total size (unpacked)=945099916
Found 103 objects to protect
Found 62 commit-pointing refs : HEAD, refs/heads/10.1.1, refs/heads/11.0.0, ...

Protected commits
-----------------

These are your protected commits, and so their contents will NOT be altered:

 * commit f9dc43e1 (protected by 'HEAD')

Cleaning
--------

Found 449 commits
Cleaning commits:       100% (449/449)
Cleaning commits completed in 315 ms.

Updating 9 Refs
---------------

    Ref                                Before     After   
    ------------------------------------------------------
    refs/heads/10.1.1                | 4a6a0846 | 280e4ca8
    refs/heads/11.0.0                | a0b5d2d5 | 10a4b7e4
    refs/heads/9.4.1                 | f5c49cf8 | 8e7b5248
    refs/heads/Issue#19              | 726459fd | b0e09600
    refs/heads/libkiwix_aar_generate | d5341836 | 568c0a15
    refs/pull/10/head                | d5341836 | 568c0a15
    refs/pull/12/head                | a0b5d2d5 | 10a4b7e4
    refs/pull/20/head                | 726459fd | b0e09600
    refs/tags/v9.4.1                 | f5c49cf8 | 8e7b5248

Updating references:    100% (9/9)
...Ref update completed in 42 ms.

Commit Tree-Dirt History
------------------------

    Earliest                                              Latest
    |                                                          |
    ...........................DDDmmmmmmm.......................

    D = dirty commits (file tree fixed)
    m = modified commits (commit message or parents changed)
    . = clean commits (no changes to file tree)

                            Before     After   
    -------------------------------------------
    First modified commit | b071e75c | 9874962c
    Last dirty commit     | 0ae595a7 | 01dd007d

Deleted files
-------------

    Filename      Git id                                     
    ---------------------------------------------------------
    kiwix.aar   | 790a42d2 (95.9 MB)                         
    libkiwix.so | c424378b (13.4 MB), f2506f5c (14.8 MB), ...
    libzim.so   | a6a99ec6 (21.5 MB), 9f901fa6 (19.4 MB), ...

In total, 151 object ids were changed. Full details are logged here:

    /home/kelson/code/java-libkiwix.git.bfg-report/2023-10-28/14-06-51

BFG run is complete! When ready, run: git reflog expire --expire=now --all && git gc --prune=now --aggressive

Here the difference of size:

$ du -sh java-libkiwix.git java-libkiwix.git.backup/
2.9M    java-libkiwix.git
305M    java-libkiwix.git.backup/

Finally I have pushed the new git log:

$ git push
Enumerating objects: 1532, done.
Counting objects: 100% (119/119), done.
Writing objects: 100% (1532/1532), 1.10 MiB | 10.30 MiB/s, done.
Total 1532 (delta 119), reused 119 (delta 119), pack-reused 1413
remote: Resolving deltas: 100% (597/597), completed with 45 local objects.
To github.com:kiwix/java-libkiwix.git
 + 4a6a084...280e4ca 10.1.1 -> 10.1.1 (forced update)
 + a0b5d2d...10a4b7e 11.0.0 -> 11.0.0 (forced update)
 + f5c49cf...8e7b524 9.4.1 -> 9.4.1 (forced update)
 + 726459f...b0e0960 Issue#19 -> Issue#19 (forced update)
 + d534183...568c0a1 libkiwix_aar_generate -> libkiwix_aar_generate (forced update)
 + f5c49cf...8e7b524 v9.4.1 -> v9.4.1 (forced update)
 ! [remote rejected] refs/pull/10/head -> refs/pull/10/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/12/head -> refs/pull/12/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/13/head -> refs/pull/13/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/14/head -> refs/pull/14/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/15/head -> refs/pull/15/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/17/head -> refs/pull/17/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/18/head -> refs/pull/18/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/20/head -> refs/pull/20/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/22/head -> refs/pull/22/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/23/head -> refs/pull/23/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/25/head -> refs/pull/25/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/25/merge -> refs/pull/25/merge (deny updating a hidden ref)
 ! [remote rejected] refs/pull/26/head -> refs/pull/26/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/27/head -> refs/pull/27/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/30/head -> refs/pull/30/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/32/head -> refs/pull/32/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/33/head -> refs/pull/33/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/34/head -> refs/pull/34/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/35/head -> refs/pull/35/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/36/head -> refs/pull/36/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/38/head -> refs/pull/38/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/4/head -> refs/pull/4/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/40/head -> refs/pull/40/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/44/head -> refs/pull/44/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/45/head -> refs/pull/45/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/48/head -> refs/pull/48/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/52/head -> refs/pull/52/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/52/merge -> refs/pull/52/merge (deny updating a hidden ref)
 ! [remote rejected] refs/pull/53/head -> refs/pull/53/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/55/head -> refs/pull/55/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/58/head -> refs/pull/58/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/59/head -> refs/pull/59/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/6/head -> refs/pull/6/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/60/head -> refs/pull/60/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/61/head -> refs/pull/61/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/63/head -> refs/pull/63/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/66/head -> refs/pull/66/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/67/head -> refs/pull/67/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/68/head -> refs/pull/68/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/68/merge -> refs/pull/68/merge (deny updating a hidden ref)
 ! [remote rejected] refs/pull/7/head -> refs/pull/7/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/8/head -> refs/pull/8/head (deny updating a hidden ref)
error: failed to push some refs to 'github.com:kiwix/java-libkiwix.git'

Any use will have to reclone its repository before any further dev. work.