Video files bloating repo (move to LFS)

[gianlucatruda]

The Video/ and data/ directories total around 100MB, which is bloating this repo considerably. I'm already implementing Git Large File Storage (LFS) for the sample data I've collected (coming soon). I think it would be a good option to migrate the existing video data over to LFS. I'm happy to take care of that and submit a PR soon.

[LittlePea13]

I have all the video files from the 2020 paper, I am getting them in the overleaf doc format, but I am not familiar with LFS. If you can let me know how to transfer them to LFS once u are done I will put them there.

[gianlucatruda]

@LittlePea13 great! I wrote a whole explainer to go along with the PR I just submitted. It links to some good references too.

I've never used LFS before, but it seems that it just intercepts commits for certain files (that you specify) and replaces them with a pointer. The LFS files get stored outside the repo, but the pointers are stored in the repo. Once it's set up and told to track certain files, you can essentially stick to a normal git workflow.

[gianlucatruda]

UPDATE: You can verify that a file is stored on LFS like so:

Unfortunately, PRs from a fork of the repo don't seem to let the LFS work properly, so the data I added to sample_data/ in this PR don't seem to be in LFS. There are ways to try fix this, but they seem to require re-writing commit history. If you do that on a branch, you're unable to submit a PR to master. You just see this:

I've been fighting with this for a few hours now and haven't got it right yet.

Fortunately, @LittlePea13's branch seems to have everything from the Nemcova dataset in LFS successfully, so that PR should go ahead fine. It seems that the issue was the PR from a fork. Either that or I have no idea what I'm doing (which is a possibility I'm prone to considering 😅).

[gianlucatruda]

Update, we seem to have run out of LFS data or bandwidth. This is hampering pulls and pushes.

[gianlucatruda]

See workarounds implemented in issue #17