[BUG]: Presence of large files in commit history

[kellijohnson-NOAA]

Is there an existing issue for this?

• [X] I have searched the existing issues

Current Behavior

There are potentially large files in the commit history that can be removed.

Expected Behavior

No response

Steps To Reproduce

No response

Environment

No response

Anything else?

No response

[seananderson]

Can easily fix with BFG https://rtyley.github.io/bfg-repo-cleaner/ but it (or any history rewriting) requires anyone with push access to reclone or they’ll just push the files back.

[kellijohnson-NOAA]

I talked to @chantelwetzel-noaa and she was fine with recloning. I am assuming you and @ericward-noaa will be okay with either forcing a reset or recloning? Please let me know if you aren't and we can find some other path forward.

[ericward-noaa]

Yep totally agree - there shouldn't be any large files here. So am fine with recloning

One of the examples of files that's likely an issue is in sandbox/dover_example/sdmTMB_save.RData -- that file is ~ 44MB . Can we drop some pieces of that object or split it if it's important to keep all of it?

[seananderson]

The only really big file was that sandbox/dover_example/sdmTMB_save.RData file. I removed it from history with BFG and updated the gitignore (you'll need to force add .RData files now). Everyone needs to grab a fresh clone.

[kellijohnson-NOAA]

I was hoping to get rid of all .RData and .png files thinking that they are reproducible with the code that is present? I should have said in the original issue that I am planning on continuing development of this repository to shift code from scripts to functions to create a package for running sdmTMB on nwfsc data like I did for VASTwestcoast. I could also do this by forking and removing these files and leaving this repository as is. Thoughts @ericward-noaa and @seananderson? Would you like to see this repo archived as is or is it okay to basically manipulate it into its next stage?

[ericward-noaa]

No need to archive this -- I think you should modify as you see fit.

This is tangential, but if helpful, I've put together this repo demonstrating automation of index standardization with sdmTMB: https://github.com/ecosystem-state/wcbts-auto

[kellijohnson-NOAA]

Removed all [rR][dD]ata and png files with 3cda550. The repository is now super quick to clone after doing some garbage collection thanks to @seananderson's tips and bfg.

[kellijohnson-NOAA]

Big files are still in the repository see Pull request #10 and comment from @seananderson . I will fix this once and for all after merging in the pull request and prior to bringing the repository to pfmc-assessments.

[kellijohnson-NOAA]

I removed the files, so new clones will be needed, but because we have closed pull requests I am not actually sure if the history is entirely deleted. I was alerted with

 + 2f4851b...1032827 main -> main (forced update)
 ! [remote rejected] refs/pull/10/head -> refs/pull/10/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/12/head -> refs/pull/12/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/14/head -> refs/pull/14/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/16/head -> refs/pull/16/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/2/head -> refs/pull/2/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/3/head -> refs/pull/3/head (deny updating a hidden ref)
error: failed to push some refs to 'https://github.com/kellijohnson-NOAA/indexwc.git'

which google told me was because I have Pull Requests in the repository. I am not super concerned about it right now, the repo already downloads 10x faster.