sixteencolors / sixteencolors-archive

Artpack archive, organized by year
142 stars 28 forks source link

Use LFS for all new files added/changed #38

Closed tracker1 closed 1 year ago

tracker1 commented 4 years ago

This will use LFS for tracking new Zip files added/changed in the repository.

closes #27

fcambus commented 4 years ago

Thanks for relaunching the discussion on this topic. I'm not very knowledgable about Git LFS at the moment, so I was hoping someone would jump in, but so far it hasn't happened :-)

When I first saw your issue, I noticed that there was still a 1GB limit, but I can't find any references about the limit anymore, did GitHub lift it?

matthewpkemp commented 4 years ago

Thanks for relaunching the discussion on this topic. I'm not very knowledgable about Git LFS at the moment, so I was hoping someone would jump in, but so far it hasn't happened :-)

When I first saw your issue, I noticed that there was still a 1GB limit, but I can't find any references about the limit anymore, did GitHub lift it?

References are here in respect to repo limits: https://help.github.com/en/github/managing-large-files/versioning-large-files

And here re lfs: https://help.github.com/en/github/managing-large-files/about-git-large-file-storage

tracker1 commented 4 years ago

It probably doesn't really matter since these files very rarely change in practice... Also, git lfs requires end-users to install/enable it git lfs install after checkout at times.

Also, not sure if the 1GB limit also applies to LFS, and it wouldn't move the existing files without rewriting the history, I think there's a script/command for that.

lordscarlet commented 1 year ago

I forgot this discussion ever happened, and I did the same. 😆

https://github.com/sixteencolors/sixteencolors-archive/pull/44

If all it means is that I have to pay $5/mo if it goes over, I'm not too worried about that. although, now I can't find where I saw that $5 number.

The fact that people may have to install LFS worries me slightly more. Does anyone know how big of a deal that is?

sairuk commented 1 year ago

as long as you have git >=1.8.3.1 you can install the git-lfs package (linux) and run git lfs install in the repo working tree to initialize it, the install may only have to be run once? i haven't played with lfs in a while now and there is no content on lfs yet in this commit to test further with.

decent tutorial on lfs here: https://github.com/git-lfs/git-lfs/wiki/Tutorial

doesn't look like the intent is to migrate the existing archive to lfs, probably worth considering having one process for pulling bins because after this is done you'll get some bins with pull but new stuff wont come down locally without a git lfs fetch iirc

just a note that git lfs migrate will need to be used and pushed to migrate existing bins to lfs. if you run git lfs ls-files on this branch atm there is nothing on lfs

after migrate you can see the lfs content

$ git lfs ls-files
495112ee0b - 2018/impure71.zip
da4584e41a - 2018/mist0218.zip
c6e0ef0e7f - 2018/mistergirls-2-perling-aint-easy.zip
a5fc4da32d - 2018/ninjapenguin2018.zip
3bbf9039f3 - 2019/blocktronics-67rpm.zip
9b55aad043 - 2019/blocktronics-blocky-horror.zip
31e1fdfda2 - 2019/blocktronics-dsotb.zip

if the entire archive is moving to lfs, .gitattributes should be updated to include rar files capture ./2011/cph.artpack.24.back.in.business.2011.rar

i used git lfs migrate import --no-rewrite -m "test migrate to lfs" */*.zip to test

lordscarlet commented 1 year ago

@sairuk what if you do the same for https://github.com/sixteencolors/sixteencolors-archive/pull/44 ?

sairuk commented 1 year ago

@lordscarlet i have added notes to #44 with the result of trying to do the same, in short lfs is already over quota trying to checkout that branch, which was always going to be a risk with a full migration.