search

sixteencolors / sixteencolors-archive

Artpack archive, organized by year
142 stars 28 forks source link

Add missing packs from 16colo.rs and enable LFS #44

Open lordscarlet opened 1 year ago

lordscarlet commented 1 year ago

I apologize for not doing these in separate commits, but:

  1. Convert files to use git lfs
  2. Add approximately 1,200 missing packs from https://16colo.rs
sairuk commented 1 year ago

This diff can't be rendered in the github ui (taking too long, unicorns etc) but I assume this migrates the existing library to lfs?

checking out this branch locally atm results in an lfs quota error which is already problematic unfortunately.

$ git checkout enable-lfs
Updating files: 100% (5661/5661), done.
Downloading 1990/acdu1190.zip (91 KB)
Error downloading object: 1990/acdu1190.zip (85f7235): Smudge error: Error downloading 1990/acdu1190.zip (85f7235f99b4b6b2a72cad8c160579acde00624a768e57df95360536dbbafd7c): batch response: This repository is over its data quota. Account responsible for LFS bandwidth should purchase more data packs to restore access.

the resulting state is a broken local working copy of master

$ git checkout enable-lfs
error: The following untracked working tree files would be overwritten by checkout:
    .gitattributes
    1994/ac!-05l.lha
    1994/coopcol2.lha
    1994/ep-bcsfb.lha
    1995/1oo!-ger.lha
    1995/al-cohol.lha
    1995/avl-clip.lha
    1996/1oo-answ.lha
    1996/1oo_gnor.lha
    1996/con-fasc.lha
    1996/m's-odds.lha
    1997/azki4all.lha
    1997/dgs-new!.lha
    1997/dkb_egs.lha
    1997/dkb_zack.lha
    1998/release.nfo
    2004/acid-100.nfo
    2016/BLNDR2016.ZIP
    2016/TRACKERFIX_2016.ZIP
Please move or remove them before you switch branches.
Aborting

can repair this state with git clean -fdx (force/+directoriess/+ignored) but yeh unfortunately the enable-lfs branch still cannot be checked out from this repo at the moment

i ran into similar quota issues and looked at ipfs to backend lfs, you can add alternate storage solutions to backend lfs but these all add additional complexity. My notes from back when i tested the ipfs backend are here: https://github.com/sairuk/ipfs-lfs-test just to demonstrate an option

lordscarlet commented 1 year ago

This seems like it's back to the territory of "just leave it as is"

On Tue, Nov 8, 2022, 1:31 PM sairuk @.***> wrote:

This diff can't be rendered in the github ui (taking too long, unicorns etc) but I assume this migrates the existing library to lfs?

checking out this branch locally atm results in an lfs quota error which is already problematic unfortunately.

$ git checkout enable-lfs Updating files: 100% (5661/5661), done. Downloading 1990/acdu1190.zip (91 KB) Error downloading object: 1990/acdu1190.zip (85f7235): Smudge error: Error downloading 1990/acdu1190.zip (85f7235f99b4b6b2a72cad8c160579acde00624a768e57df95360536dbbafd7c): batch response: This repository is over its data quota. Account responsible for LFS bandwidth should purchase more data packs to restore access.

the resulting state is a broken local working copy of master

$ git checkout enable-lfs error: The following untracked working tree files would be overwritten by checkout: .gitattributes 1994/ac!-05l.lha 1994/coopcol2.lha 1994/ep-bcsfb.lha 1995/1oo!-ger.lha 1995/al-cohol.lha 1995/avl-clip.lha 1996/1oo-answ.lha 1996/1oo_gnor.lha 1996/con-fasc.lha 1996/m's-odds.lha 1997/azki4all.lha 1997/dgs-new!.lha 1997/dkb_egs.lha 1997/dkb_zack.lha 1998/release.nfo 2004/acid-100.nfo 2016/BLNDR2016.ZIP 2016/TRACKERFIX_2016.ZIP Please move or remove them before you switch branches. Aborting

can repair this state with git clean -fdx (force/+directoriess/+ignored) but yeh unfortunately the enable-lfs branch still cannot be checked out from this repo at the moment

i ran into similar quota issues and looked at ipfs to backend lfs, you can add alternate storage solutions to backend lfs but these all add additional complexity. My notes from back when i tested the ipfs backend are here: https://github.com/sairuk/ipfs-lfs-test just to demonstrate an option

— Reply to this email directly, view it on GitHub https://github.com/sixteencolors/sixteencolors-archive/pull/44#issuecomment-1307660265, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAEMH3M2JLSLJ3LAJUOFTTWHKMBPANCNFSM6AAAAAARVTS3UU . You are receiving this because you authored the thread.Message ID: @.***>

sairuk commented 1 year ago

yeh seems so, what was the motivation to go to lfs? lfs max unpaid size is only 1GB. Is the repo topping out or are the packs getting too large to host here in a normal git repo?

Can this repo get an exception from github to enable this historical (+moddern) collection to be hosted free of charge? I've read elsewhere of a couple of other projects (historical/emulation) approached github (maybe before the ms acquisition) and have some kind of deal in place.

tracker1 commented 1 year ago

@lordscarlet @sairuk ... didn't see this when I created my individual PRs, was getting an error trying to do it all in a single request... I didn't do anything with LFS on my separate PRs.

lordscarlet commented 1 year ago

Someone mentioned they were having trouble downloading the archive, and had written some sort of script to get it to work. So I thought I would try LFS. It definitely doesn't need it, but I thought it might help some people.

On Tue, Nov 8, 2022 at 10:11 PM sairuk @.***> wrote:

yeh seems so, what was the motivation to go to lfs? lfs max unpaid size is only 1GB. Is the repo topping out or are the packs getting too large to host here in a normal git repo?

Can this repo get an exception from github to enable this historical (+moddern) collection to be hosted free of charge? I've read elsewhere of a couple of other projects (historical/emulation) approached github (maybe before the ms acquisition) and have some kind of deal in place.

— Reply to this email directly, view it on GitHub https://github.com/sixteencolors/sixteencolors-archive/pull/44#issuecomment-1308147741, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAEMH27RZ6C7ZOEXRKEWYLWHMI73ANCNFSM6AAAAAARVTS3UU . You are receiving this because you authored the thread.Message ID: @.***>

lordscarlet commented 1 year ago

I think I need to do a new PR without LFS

On Wed, Nov 16, 2022 at 4:50 PM Michael J. Ryan @.***> wrote:

@lordscarlet https://github.com/lordscarlet @sairuk https://github.com/sairuk ... didn't see this when I created my individual PRs, was getting an error trying to do it all in a single request... I didn't do anything with LFS on my separate PRs.

— Reply to this email directly, view it on GitHub https://github.com/sixteencolors/sixteencolors-archive/pull/44#issuecomment-1317716407, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAEMHZTDTC7M7OLTJT4LLDWIVJLDANCNFSM6AAAAAARVTS3UU . You are receiving this because you were mentioned.Message ID: @.***>

fcambus commented 1 year ago

I also tried to convert the archive to use LFS at some point including rewriting all the commit history, and I now regret not having taken notes about the process... But basically it didn't gain us much at all regarding storage space because the packs do not change. So it's not worth it, but we should probably document the details so we don't end up trying to revisit this decision every couple of years or so.

fcambus commented 1 year ago

@lordscarlet Yes, it would be nice if you could do a new PR without LFS, and possibly separate the commits per year or decades like @tracker1 did?

Do we want to add the MIST packs though? I have carefully been avoiding adding them while I was still updating the archive on a regular basis: this is an ANSI/ASCII Art Archive after all :-)

tracker1 commented 1 year ago

Sorry, that reminds me, need to update (still) to get actual zip files, where it looked like it snagged a couple html results in them. Though my separation was mostly that I couldn't push a larger commit to github.