LLNL / SSAPy

A Python package allowing for fast and precise orbital modeling.
https://software.llnl.gov/SSAPy/
MIT License
36 stars 8 forks source link

Store data files using Git LFS #2

Closed aperloff closed 4 months ago

aperloff commented 5 months ago

Background

SSAPy relies on some large binary files. The problem with binary files an git is that they can cause the repository to grow very large over time and thus causing it to take excessive amounts of time to clone the repo. The reasons for this are beyond the scope of this message.

The problem

We couldn't initially use Git LFS to store these files because the LLNL GitHub group didn't have enough GitHub LFS storage. We therefore created a second repository (SSAPy-Data) for storing these large files, agreeing that these files weren't to be changed much after their initial commit. Nevertheless, some of the files were >100MB, which is above the single file limit for GitHub. Thus, the files needed to be split into pieces and recombined. This lead to complicated checkout and data file update procedures.

The solution

The LLNL GitHUb group recently purchased enough storage and bandwidth to allow for further use of Git LFS. This PR removes the requirement to have an external module (i.e. the SSAPy-Data repository) and adds back in the large files using Git LFS. All of the binary files are added back in using Git LFS, even if they aren't that big, as there is no grantee that they will be small if updated by the external organizations which created them.