Bandwidth Caps for Git LFS

swincas / cookies-n-code

A repo for code review sessions at CAS

http://astronomy.swin.edu.au/

MIT License

30 stars 34 forks source link

Bandwidth Caps for Git LFS #28

Open jacobseiler opened 6 years ago

jacobseiler commented 6 years ago

I recently started tracking a large file using Git LFS. It's only a single file that is ~128Mb in size.

The repository is connected to my tests so each time I want to build my code on Travis it has to access the large file. After working for a few hours, I got an email stating


Git LFS has been disabled on your personal account jacobseiler because you’ve exceeded your data plan by at least 150%. Please purchase additional data packs to cover your bandwidth and storage usage:

https://github.com/account/billing/data/upgrade

Current usage as of 20 Aug 2018 02:38AM UTC:

Bandwidth: 1.5 GB / 1 GB (150%)
Storage: 0.13 GB / 1 GB (13%)

This seems to say that I can only git clone the repo a certain amount of times before I go over the bandwidth limit? I'm absolutely baffled by this...

Does anybody have any experiencing using Git LFS and care to offer some help?

manodeep commented 6 years ago

This offers some hope

jacobseiler commented 6 years ago

Thanks for that. I believe this fix is only valid if the directory you're building with Travis is the one under Git LFS. That is, if I'm building my rsage repo, it will work if the rsage repo itself is under Git LFS.

However my setup is as follows:

The original rsage repo that is being built by Travis is not under Git LFS
After the rsage repo has built its executable, it runs a suite of tests.
The first step in this testing suite is to clone a secondary repo which contains all of my test data. This repo is the one that is under Git LFS

I could move the test data into the rsage repo, put it under Git LFS and implement the fixed you linked. However, I'm hesitant to do this because it then means that the user will have to a) pull the test data when they clone the rsage code; b) Perform an extra when cloning and increase the overhead/complexity.

I think this is what will end up happening but thought I'd throw it out there first...

manodeep commented 5 years ago

@jacobseiler You could also also automate that extra step under a Makefile target. For instance, in my fork of sage, make tests pulls down the necessary data. Is that option possible? (And cache that new directory on travis)

jacobseiler commented 5 years ago

In my .travis.yml I have added the following lines

cache:
  directories:
    - tests/rsage_testdata/

This will then cache all of the test data and ensure that it won't be pulled for every Travis build.

I've moved away from Git LFS because the bandwidth is repository specific. So if you made a fork of rsage and ran the make tests command, it would pull down the test data and count the bandwidth against my own account. This could get me into trouble down the line if I ever ask people to collaborate on the repo/it gets popular.

Feel free to close the issue.