search

dfm / tess-atlas

MIT License
8 stars 7 forks source link

Edit git history for notebooks for main #31

Closed avivajpeyi closed 3 years ago

avivajpeyi commented 3 years ago
avivajpeyi commented 3 years ago

This is what i tried:

git clone --mirror  git@github.com:dfm/tess-atlas.git
bfg --delete-files notebooks/0.1.*/*.ipynb --protect-blobs-from gh-pages tess-atlas.git
cd tess-atlas.git
git reflog expire --expire=now --all && git gc --prune=now --aggressive

Logs:

$ bfg --delete-files toi-*.ipynb --protect-blobs-from gh-pages tess-atlas.git

Using repo : /Users/avaj0001/Documents/tess-atlas.git

Found 294 objects to protect
Found 33 commit-pointing refs : HEAD, refs/heads/V0.2.0, refs/heads/add-badge-to-readme, ...

Protected commits
-----------------

These are your protected commits, and so their contents will NOT be altered:

 * commit 653498a8 (protected by 'gh-pages') - contains 102 dirty files : 
        - notebooks/0.1.0/toi-116.ipynb (1.7 MB)
        - notebooks/0.1.0/toi-123.ipynb (1.4 MB)
        - ...

WARNING: The dirty content above may be removed from other commits, but as
the *protected* commits still use it, it will STILL exist in your repository.

Details of protected dirty content have been recorded here :

/Users/avaj0001/Documents/tess-atlas.git.bfg-report/2020-09-24/20-37-35/protected-dirt/

If you *really* want this content gone, make a manual commit that removes it,
and then run the BFG on a fresh copy of your repo.

Cleaning
--------

Found 163 commits
Cleaning commits:       100% (163/163)
Cleaning commits completed in 287 ms.

Updating 30 Refs
----------------

        Ref                                                    Before     After   
        --------------------------------------------------------------------------
        refs/heads/V0.2.0                                    | ead2a59c | 147ca557
        refs/heads/add-badge-to-readme                       | 6be96a85 | 16526f20
        refs/heads/cleanup                                   | 940fba48 | cfea3200
        refs/heads/functionalise_run_toi                     | e3a44f75 | bbd2e8a4
        refs/heads/logo                                      | 90cc796b | 466ad44d
        refs/heads/main                                      | eeb4ea7f | 6af74414
        refs/heads/master                                    | c582852c | e4d643e0
        refs/heads/refactor_run_tois                         | 53cc169a | 9ed6c6c0
        refs/heads/replace_template_with_quick_tess_notebook | df661737 | 25429d86
        refs/heads/run_tois_v0.2.0_test                      | e7dc6e4b | 05115c5c
        refs/heads/template_fix                              | 52d47797 | 5bf78fc5
        refs/pull/11/head                                    | 5acb7d4c | 3f733512
        refs/pull/12/head                                    | 0d8e09db | 8ae11d59
        refs/pull/13/head                                    | af28c045 | 266ac441
        refs/pull/14/head                                    | 2a20c091 | 1d65f685
        ...

Updating references:    100% (30/30)
...Ref update completed in 49 ms.

Commit Tree-Dirt History
------------------------

        Earliest                                              Latest
        |                                                          |
        ...DDDDDDDDDDDDDDDmmDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDmDDD

        D = dirty commits (file tree fixed)
        m = modified commits (commit message or parents changed)
        . = clean commits (no changes to file tree)

                                Before     After   
        -------------------------------------------
        First modified commit | 7a2d2968 | 17559232
        Last dirty commit     | eeb4ea7f | 6af74414

Deleted files
-------------

        Filename         Git id                                                  
        -------------------------------------------------------------------------
        toi-103.ipynb  | e3482133 (1.0 MB), 1cf4bb8d (1.6 MB)                    
        toi-105.ipynb  | eaac7aee (739.3 KB)                                     
        toi-1075.ipynb | bd2de095 (1.8 MB)                                       
        toi-110.ipynb  | 8b8595ab (2.0 MB)                                       
        toi-116.ipynb  | d7547e83 (906.3 KB), 698481e5 (1.7 MB)                  
        toi-1178.ipynb | 10837d01 (2.3 MB)                                       
        toi-1179.ipynb | 0bcc4b12 (1.6 MB)                                       
        toi-118.ipynb  | 75b360ed (846.7 KB)                                     
        toi-1180.ipynb | e2d2a4dc (1.7 MB)                                       
        toi-119.ipynb  | dd2dff1d (2.4 MB)                                       
        toi-121.ipynb  | e8223df3 (1.1 MB)                                       
        toi-1210.ipynb | c827f3a7 (1.7 MB)                                       
        toi-1211.ipynb | 04eb06f8 (1.3 MB)                                       
        toi-1212.ipynb | dd8d5017 (2.4 MB)                                       
        toi-123.ipynb  | 6cc6bd2f (1.4 MB)                                       
        ...

In total, 324 object ids were changed. Full details are logged here:

        /Users/avaj0001/Documents/tess-atlas.git.bfg-report/2020-09-24/20-37-35

BFG run is complete! When ready, run: git reflog expire --expire=now --all && git gc --prune=now --aggressive

--
You can rewrite history in Git - don't let Trump do it for real!
Trump's administration has lied consistently, to make people give up on ever
being told the truth. Don't give up: https://www.rescue.org/topic/refugees-america
--

The following message states that it didn't work because of protected commits...: 

WARNING: The dirty content above may be removed from other commits, but as
the *protected* commits still use it, it will STILL exist in your repository.

Details of protected dirty content have been recorded here :

/Users/avaj0001/Documents/tess-atlas.git.bfg-report/2020-09-24/20-37-35/protected-dirt/

If you *really* want this content gone, make a manual commit that removes it,
and then run the BFG on a fresh copy of your repo.
avivajpeyi commented 3 years ago

From this link, there are some additional steps needed:

1. Get a "mirror" of the git repository and use bfg utility to remove all file over 100 MB

2. Look at the comma separated /home/user/repo.git.bfg-report/2017-06-08/15-07-17/protected-dirt/ file that bfg creates to find the full path to those large files that bfg did not remove.

3. get a "clone" of the repository with workspace and remove all the files using "git rm" command and commit the delete.

4. Run the bfg utility in this "clone" repository". Now bfg will delete the files.
avivajpeyi commented 3 years ago

The CSV created by bfg: /Users/avaj0001/Documents/tess-atlas.git.bfg-report/2020-09-24/20-37-35/protected-dirt/653498a8-gh-pages.csv

698481e54b05cba38ad9371dbcd9fc1421f11522,DELETE,regular-file,notebooks/0.1.0/toi-116.ipynb,1740272,
6cc6bd2fc37d9f6c8a9ae447ea845fa96892b6fc,DELETE,regular-file,notebooks/0.1.0/toi-123.ipynb,1490162,
d6eae515d0f6ab913eaf1aa60103f80dbf9e8233,DELETE,regular-file,notebooks/0.1.0/toi-143.ipynb,808077,
54e6463aeeff85948feea7a8ed0f21f039b78c2f,DELETE,regular-file,notebooks/0.1.0/toi-144.ipynb,864020,
b52b0ab8be964a827e6c92dda04988cbba8c5030,DELETE,regular-file,notebooks/0.1.0/toi-191.ipynb,966077,
7741ca469e1aae8f1d3b5373d7e2a1395f5ded1a,DELETE,regular-file,notebooks/0.1.0/toi-248.ipynb,848710,
dd58fda34574d3f47e97c33428e605e8f46c20f1,DELETE,regular-file,notebooks/0.1.0/toi-271.ipynb,1780417,
bd5bbb3f84ff74671d2983e90746fcf6d0ec4254,DELETE,regular-file,notebooks/0.1.0/toi-425.ipynb,1088939,
a2cce67f87aa21a52e9baf3a26d5c55ef3805706,DELETE,regular-file,notebooks/0.1.0/toi-440.ipynb,1137868,
a3ca83933bd6326d8c58f1de5347b46ca0f82bf8,DELETE,regular-file,notebooks/0.1.0/toi-467.ipynb,1109803,
f358a926acd40df3d7da8634824a6d39e879ae65,DELETE,regular-file,notebooks/0.1.0/toi-495.ipynb,1363394,
8b0a8a6c0319a9792a99eaea0cac336d1cc94415,DELETE,regular-file,notebooks/0.1.0/toi-511.ipynb,1192239,
b8d30a0a1122e9aee1a1b6cd1c3d54aaf4389d2e,DELETE,regular-file,notebooks/0.1.0/toi-524.ipynb,1161123,
0a43c25f1723cfa5707419a7c68978c276ee0b7a,DELETE,regular-file,notebooks/0.1.0/toi-538.ipynb,1811929,
e3482133bdaae34aa60620dfe2b46105fd8b5d4e,DELETE,regular-file,notebooks/0.1.1/toi-103.ipynb,1068157,
eaac7aee74a42575ad6754c79010fbf5a7bf0d8b,DELETE,regular-file,notebooks/0.1.1/toi-105.ipynb,757021,
d7547e83b299274e6d053a8c6acf2a5b0dfbcf61,DELETE,regular-file,notebooks/0.1.1/toi-116.ipynb,928053,
75b360ede35954bf68f3339e6c5eda72e7fe8189,DELETE,regular-file,notebooks/0.1.1/toi-118.ipynb,867029,
dd2dff1d0ab2388683d8d6a810e9af56225b8de5,DELETE,regular-file,notebooks/0.1.1/toi-119.ipynb,2539991,
e8223df39436e7a5dc447398a595d235b5e36bdb,DELETE,regular-file,notebooks/0.1.1/toi-121.ipynb,1172801,
c3caaa2bb9e6f48f9d239d83094be61eeaa7863d,DELETE,regular-file,notebooks/0.1.1/toi-128.ipynb,956639,
a224009c5452e6c63ddc801d55f7ffd39031baee,DELETE,regular-file,notebooks/0.1.1/toi-139.ipynb,1057975,
2732d9d1afb4744dca0e235e0d70bc9ea4121ad3,DELETE,regular-file,notebooks/0.1.1/toi-178.ipynb,3344153,
c012937823f88a77549e2bd6f09ef24e5deb03dc,DELETE,regular-file,notebooks/0.1.1/toi-181.ipynb,1136723,
68b918c5569fa5c02a3ee138aa255742bada2313,DELETE,regular-file,notebooks/0.1.1/toi-187.ipynb,2226285,
71cc6a40cc434b4154ab477237f6e6abbd6661ea,DELETE,regular-file,notebooks/0.1.1/toi-189.ipynb,1841933,
ccd3dd88d1b5bf26ecc81822868e0f73a0c15c2a,DELETE,regular-file,notebooks/0.1.1/toi-190.ipynb,1301078,
604f2a2034b1ddcda2c881ca736c57d74c3a0063,DELETE,regular-file,notebooks/0.1.1/toi-195.ipynb,1380150,
dab832a13ac1f08832b6e2952fadd994f7aa9a0b,DELETE,regular-file,notebooks/0.1.1/toi-197.ipynb,1057099,
b6ee6f706b075b1aa3a54653407aa9e222e6abaa,DELETE,regular-file,notebooks/0.1.1/toi-198.ipynb,1059463,
60df81c0d91b35c4ecc8857c53033b7ccc2109aa,DELETE,regular-file,notebooks/0.1.1/toi-202.ipynb,1182678,
ff31de1356b7ac7e508784c212c4500ee8e2b1fe,DELETE,regular-file,notebooks/0.1.1/toi-204.ipynb,1416588,
16c1fde754d3a478612635df5c6b07e90c1a4358,DELETE,regular-file,notebooks/0.1.1/toi-211.ipynb,2139102,
cf8c683f24e696a649597a2ad5b5627c2e4b34a6,DELETE,regular-file,notebooks/0.1.1/toi-229.ipynb,1199456,
eb4a46a29548773bdfee6a399b7886a5a29f645f,DELETE,regular-file,notebooks/0.1.1/toi-236.ipynb,1055310,
602a0a6b8883446cb98b670d8d8cf07eeefb5e74,DELETE,regular-file,notebooks/0.1.1/toi-260.ipynb,1239269,
5a953606f29e885a656aab0d9e541c73770bf9b7,DELETE,regular-file,notebooks/0.1.1/toi-261.ipynb,2364046,
c0f3f092e138ab4ad49d65cb1117f0376a252913,DELETE,regular-file,notebooks/0.1.1/toi-263.ipynb,1714471,
9f0045572877669b144dbe33499a21a966d203ce,DELETE,regular-file,notebooks/0.1.1/toi-268.ipynb,1424418,
b54dc2ae7ad95595ab354458624c7a4098e6aad2,DELETE,regular-file,notebooks/0.1.1/toi-274.ipynb,1180652,
6be95a7ce983d3dfae00381528798db81c16042e,DELETE,regular-file,notebooks/0.1.1/toi-275.ipynb,2026708,
1f1a3337de22733e4a295197c05af80a09682487,DELETE,regular-file,notebooks/0.1.1/toi-278.ipynb,1739483,
b7526587cf79f0aacefd8f7b85b1b546bcbc5126,DELETE,regular-file,notebooks/0.1.1/toi-281.ipynb,1252263,
2446babbdf91a8ea7ab179ac70be9d2b29ca2017,DELETE,regular-file,notebooks/0.1.1/toi-283.ipynb,1754290,
091393d3372e9680340052bd23d8397f95af9a4b,DELETE,regular-file,notebooks/0.1.1/toi-349.ipynb,901059,
ac150957e025cefc851c8345909c6f7d7488f89e,DELETE,regular-file,notebooks/0.1.1/toi-367.ipynb,2144521,
3e4005b2b0c9de0c6dab9cfddc217e5fbfca91eb,DELETE,regular-file,notebooks/0.1.1/toi-390.ipynb,1723725,
4301fa564bfe1b51ae9649947742dbfffd1fdd91,DELETE,regular-file,notebooks/0.1.1/toi-393.ipynb,1349735,
06a63a0c4a2556a8d27fd20dacea1bf0b3aba586,DELETE,regular-file,notebooks/0.1.1/toi-398.ipynb,2039755,
6fb8578dd88277f9b0469f87e364535cc1ad15aa,DELETE,regular-file,notebooks/0.1.1/toi-416.ipynb,840903,
0af3c0e2985c20006d1679a1c7d39bdbf9b997b5,DELETE,regular-file,notebooks/0.1.1/toi-419.ipynb,2177124,
cc9482d650cb52d6e10b329377d788cc091559ed,DELETE,regular-file,notebooks/0.1.1/toi-420.ipynb,2229418,
fc245d85f0a83a0cf4a2726f00abcd65a1b8a984,DELETE,regular-file,notebooks/0.1.1/toi-422.ipynb,2063556,
2a144f83b70ad5ed79579c3404f189091ff6787f,DELETE,regular-file,notebooks/0.1.1/toi-427.ipynb,1986663,
2621b192ba0e518bd5956b5f03cd9740aed07748,DELETE,regular-file,notebooks/0.1.1/toi-429.ipynb,1827324,
2c59ad4c6639424ad05e747d2cf9ae2e8cfe781c,DELETE,regular-file,notebooks/0.1.1/toi-433.ipynb,2217097,
71eee25d639b2e8c99d234101c3bd2c61912ad73,DELETE,regular-file,notebooks/0.1.1/toi-436.ipynb,1624668,
9c1ff419e962e61e0198b447a795ecc176f3576b,DELETE,regular-file,notebooks/0.1.1/toi-439.ipynb,1054619,
2980d9fc94bd7b4a6a63213ad57200555715c1d4,DELETE,regular-file,notebooks/0.1.1/toi-454.ipynb,1210431,
33fdae5f19c9a6d73d8a1e8911fad2a9d808e775,DELETE,regular-file,notebooks/0.1.1/toi-457.ipynb,1763267,
28437c4646ec400a73a77219e1ebebac1f96b267,DELETE,regular-file,notebooks/0.1.1/toi-461.ipynb,748571,
a267138ad2d8f28f80fefd2699470313990bd3cb,DELETE,regular-file,notebooks/0.1.1/toi-469.ipynb,1174070,
132ef02316da4b1cdaf6fc1ee7118071bc453a55,DELETE,regular-file,notebooks/0.1.1/toi-478.ipynb,931243,
8f4de09bf44c5444e618b7232596626967b0941b,DELETE,regular-file,notebooks/0.1.1/toi-486.ipynb,2168904,
4135db83d77f6882fbb8bc4da96bb314770e8158,DELETE,regular-file,notebooks/0.1.1/toi-493.ipynb,1374105,
1d7f02df50bd0fd307c2453ba8eef0227f247bb3,DELETE,regular-file,notebooks/0.1.1/toi-494.ipynb,1779595,
3190d695315436b3987ae73a4e5a4d9c12ec253d,DELETE,regular-file,notebooks/0.1.1/toi-502.ipynb,1416767,
96b95c30fb2860481368d30aef67f15ab7f6a7a8,DELETE,regular-file,notebooks/0.1.1/toi-510.ipynb,1421154,
a74df39330e7638d8b81dac2165edd3f7624081b,DELETE,regular-file,notebooks/0.1.1/toi-519.ipynb,1015584,
01e480b9f3da6ccec668f397fc0ff07b772c7397,DELETE,regular-file,notebooks/0.1.1/toi-520.ipynb,2102802,
0c1924e6c039325ce3ad10927c82aaa1e16fa34a,DELETE,regular-file,notebooks/0.1.1/toi-521.ipynb,1991652,
e09714d5d73dc297539d7dc9041851d2ad4201d7,DELETE,regular-file,notebooks/0.1.1/toi-530.ipynb,1965405,
634d9e8f2886858b52f8d03fa540e4bbe6321ba7,DELETE,regular-file,notebooks/0.1.1/toi-533.ipynb,2117595,
37ff69b885e8c585928b1af4ebe5692b753c97db,DELETE,regular-file,notebooks/0.1.1/toi-537.ipynb,1295287,
82027284fead0265061a3472caefd4ff2f30c489,DELETE,regular-file,notebooks/0.1.1/toi-549.ipynb,2114460,
8b8595ab8c61fee026cc9aa04594e373ed7adf8c,DELETE,regular-file,notebooks/0.2.0/toi-110.ipynb,2115963,
b33797da16f0dee37293865f72bbd90359b5e26a,DELETE,regular-file,notebooks/0.2.0/toi-1230.ipynb,1852768,
a67d4af0cce962c1beb2457b91e3545c43afa7e6,DELETE,regular-file,notebooks/0.2.0/toi-1244.ipynb,2347145,
f4a429e96d56de37217b412d1aaa07ebb9bd1d41,DELETE,regular-file,notebooks/0.2.0/toi-1323.ipynb,2091337,
c70496bc9530638c6d27b1fc7a5cc75a3d268cfe,DELETE,regular-file,notebooks/0.2.0/toi-1435.ipynb,1615073,
b4e30cc15efc9a4f11099b134ab2bf22f4ef8031,DELETE,regular-file,notebooks/0.2.0/toi-1794.ipynb,1809215,
28a00220203514a1f0c2302cf9e595f0a8a3420e,DELETE,regular-file,notebooks/0.2.0/toi-1803.ipynb,1532630,
bcd663ef9971bd72a612feaa2cd51a214f0df9c0,DELETE,regular-file,notebooks/0.2.0/toi-1841.ipynb,1610469,
828fa9769b8b368f99faffb13ae7bdf4d5a739a6,DELETE,regular-file,notebooks/0.2.0/toi-1861.ipynb,1684035,
24a6b5fbaa46a1ec04c4e95b269a020cc55d3d02,DELETE,regular-file,notebooks/0.2.0/toi-1871.ipynb,1588832,
1cdacf76d99a6229489b8e9773f1f5610e3e340e,DELETE,regular-file,notebooks/0.2.0/toi-230.ipynb,1926585,
808cc90a6e9dc17e136ba64aa342d11af6053c09,DELETE,regular-file,notebooks/0.2.0/toi-256.ipynb,1049919,
015e5b0622158827378d86e0db5c86d1fb4c045c,DELETE,regular-file,notebooks/0.2.0/toi-259.ipynb,1798153,
28bfe87822f954303c0b71db7658adbd5865d06c,DELETE,regular-file,notebooks/0.2.0/toi-279.ipynb,1391265,
0f49e3cb487d7549f17aa1fdf770036ae2f1b0d1,DELETE,regular-file,notebooks/0.2.0/toi-402.ipynb,1836128,
7b5afdc61c02cb7400e700b71cc6d15b500b1c66,DELETE,regular-file,notebooks/0.2.0/toi-418.ipynb,1987740,
9fc1b3cecf1db6fad5da2f98b7f5c0ef31faf125,DELETE,regular-file,notebooks/0.2.0/toi-421.ipynb,1379483,
85f7ba426aada604d647fcc9f56867e05740a662,DELETE,regular-file,notebooks/0.2.0/toi-428.ipynb,1267470,
c39059f01ddbd4cc7a97f43617d61d0c768a8f3a,DELETE,regular-file,notebooks/0.2.0/toi-525.ipynb,1186632,
4c6749392a7c7ddb02bcac8593b9f8cd39a2871c,DELETE,regular-file,notebooks/0.2.0/toi-700.ipynb,2107904,
40c8b02c1ef1412a9e3ef31093adadc83c839bf8,DELETE,regular-file,notebooks/0.2.0/toi-726.ipynb,1206008,
b57c939f775a85a1e1197fd9e1eeb1635d25f94b,DELETE,regular-file,notebooks/0.2.0/toi-770.ipynb,1799368,
9039e55b8925adb8d3fe006d22c299c5d988e961,DELETE,regular-file,notebooks/0.2.0/toi-778.ipynb,1216809,
44f73c123ea989b25271e675b940567551e11bd1,DELETE,regular-file,notebooks/0.2.0/toi-801.ipynb,1979528,
6b7457df139b32460dc890480981714dc562f9d1,DELETE,regular-file,notebooks/0.2.0/toi-833.ipynb,1660371,
b62538756d6f4d96be31b92640f453bed68d78c1,DELETE,regular-file,notebooks/0.2.0/toi-910.ipynb,1093012,
9ca60bb91139b9d9c9bca037dddc0bc8ea734659,DELETE,regular-file,notebooks/TEST/toi-1235.ipynb,2233639,
avivajpeyi commented 3 years ago

I tried doing it with just git (not using bfg) and it worked! 

# Create tracking branches of all branches
for remote in `git branch -r | grep -v /HEAD`; do git checkout --track $remote ; done

# Remove `notebooks/` from all commits, then remove the refs to the old commits
git filter-branch --index-filter 'git rm -rf --cached --ignore-unmatch notebooks/' --prune-empty --tag-name-filter cat -- --all
git for-each-ref --format="%(refname)" refs/original/ | xargs -n 1 git update-ref -d

# Ensure all old refs are fully removed
rm -rf .git/logs .git/refs/original

# Perform a garbage collection to remove commits with no refs
git gc --prune=all --aggressive

# Force push all branches to overwrite their history
git push origin --all --force
git push origin --tags --force

Hurray!