[CHECKLIST] Investigate alternative storage solutions

[dutterbutter]

Overview

The fixture are getting crazy large in size, every time we do a blob change, it result in like 99% file change from the old ones, so git history ends up storing both files, even though we will use the most recent one.

Downloading all these files is just more waste of time and bandwidth, the repo size will go over half a gigabyte if we continue that way very soon (right now it is around 300mb+).

Potential Solution

• Upload the actual files to ipfs (we already using ipfs for the dApp) so we just upload the files and change the hash or the ID of the most recent update to the new one in a script, and this script would be in a git repo.
• When using the fixture we just run the script and it will download the files for us. also we could have a script to automate the upload process too.
• Other solutions to consider is the use of a public s3 bucket

Tasks

• [ ] Finding a solution to storage
• [ ] Removing all submodules
• [ ] Integrating the downloading of files for tests / deployments, etc.
• [ ] Update GitHub actions across all repos using fixture submodules

[nepoche]

Github History Strategy

The problem:

when we run git clone <webb-repo> on essentially any repo that interacts with fixtures, it takes forever to download and git stores a TON of disk space. How should we manage the fixtures and their relation to our different repositories?

Observations:

• Even after removing the submodule entirely and pushing to the remote, a subsequent git clone is still incredibly bloated.

  • Tried to look into the bloat, and utilized a script for:

  [List all Git repository objects by size.](https://gist.github.com/magnetikonline/dd5837d597722c9c2d5dfa16d8efe5b9)

- Wow! Looks like a lot of git blobs for things related to earlier handling of fixtures in the repo.

• Fresh git clone of protocol-solidity before any submodules init. This shows the issue is with the git history of the project: ~356MB

Things I’ve tried

I attempted these things on a fork of the protocol-solidity repository.

BFG Repo-Cleaner

• Using the bfg-repo-cleaner tool to clean the history of blob sizes >1MB:

  [BFG Repo-Cleaner](https://rtyley.github.io/bfg-repo-cleaner/)

  • It looks like the history is in-tact with respect to commit messages.
• After cleaning and configuring bare submodules, I deleted my local repo and re-cloned protocol-solidity.
  • After cloning: ~8.7MB
    • After cloning and git submodule update --init:
      • protocol-solidity-fixtures: 324MB
      • protocol-solidity (root): 482MB
    • Likely due to the raw size of protocol-solidity-fixtures
      • ~150MB from ptau folder.
      • ~100MB for vanchor 16.
      • ~75MB for the rest of the fixtures.

Git-filter-repo

https://github.com/newren/git-filter-repo

• An alternative to BFG repo-cleaner. I installed it and used their useful analysis tool:
  • git filter-repo --analyze

What do we all need to do?

1. When the git history gets cleaned and updated, we need to ditch our old clones (and associated work on this clone). Then, re-clone the cleaned repository. That way, old git history is not pushed to the remotes.