Run the git BFG repo cleaner - Githubissues

Exo-TiC / ExoTiC-ISM

This is a repository for the reduction pipeline detailed in Wakeford, et al., 2016, ApJ. The method implements marginalization across a series of models to represent stochastic models for observatory and instrument systematics. This is primarily for HST WFC3, however, may be extended to STIS in the future.

MIT License

8 stars 6 forks source link

Run the git BFG repo cleaner #49

Closed ivalaginja closed 4 years ago

ivalaginja commented 5 years ago

The initial commit for this repository contained way more files and data than now, so the git history is pretty heavy size wise. I would like to run the BFG repo cleaner on it to get rid of all those old files in the git history that we cleaned up.

PR #48 is a good reference for the file types that got deleted.

ivalaginja commented 4 years ago

Before running the BFG cleaner, this repository has a size of 4.61 GB on disk, which includes all of the git history and everything stored in it. If I sum the sizes of the files and directories within the repo (in the current version), it's a little less than 2.9 GB. 2.86 GB of those are the limb darkening folder, and all the rest is a little over 1 MB.

So it does actually make sense to run the cleaner, since we might reduce the size of the repo significantly.

ivalaginja commented 4 years ago

I use these websites as reference:

https://rtyley.github.io/bfg-repo-cleaner/ - this is the original website and this is where you can download the BFG
https://torchbox.com/blog/how-remove-unwanted-files-your-git-repository/ - just another pretty website explaining the same thing
https://github.com/firemodels/fds/wiki/The-BFG-Repo-Cleaner - and another good website about it

ivalaginja commented 4 years ago

Note how the BFG will protect your latest commit on the branches master and develop, so we are running a certain risk of losing stuff from other branches if we are not careful.

ivalaginja commented 4 years ago

I just realized that the push limit of the cleaned up repo is usually 2GB. Since we would still be above that even after the cleanup, we might not actually be able to do this. I will try and report back as I go along.

ivalaginja commented 4 years ago

Supposedly, this did something... it failed to push to some remote refs that I didn't even know were there. It seems to have compressed the repo to about 1.5 GB but that's not what I see when I make a fresh clone.

Here is the terminal output in case somebody wants to know what happened, but I am closing this issue now:

Screen Shot 2020-02-03 at 20 08 20

Also, this totally deleted the branch for PR #86 and closed that PR without merging, oops... I will make a new branch and a new PR for this.

ivalaginja commented 4 years ago

I realized that one motivation to clean out the repo is also to speed up the CI, since that checks the repo out every single time it runs. Might be worth to look into this at a later point.