libretro / libretro-thumbnails

Boxarts, titlescreen, and in-game screenshots for the no-intro DATs used by RetroArch
352 stars 350 forks source link

Cannot download SNES thumbnails via Retroarch, also can't download master via PC #307

Open tapperp opened 7 years ago

tapperp commented 7 years ago

Using Android TV, latest Retroarch hangs when trying to download the SNES thumbnails.

Also, when trying to download the master thumbnails file from this repository on PC, the download will not complete. I tried twice -- the 1st time it stopped around 4.5GB, and the 2nd time around 3GB.

Thanks for any fixes.

RobotLimeLtd commented 7 years ago

Also does not download from GPD XD Android Device. Download progress goes 0%, 1%, 2%, then resets back to 0% in an endless loop (until you quit RetroArch).

I'm not sure how RetroArch grabs the files, is it using the Web API and hitting the 1,000 files per directory limit, therefore unable to create the download zip? There's around 2,400+ files in each of the 3 directories.

I've tried downloading just this SNES directory manually using https://minhaskamal.github.io/DownGit/#/home and although it succeeded, each of the 3 folders only had 1000 files in them.

RobotLimeLtd commented 7 years ago

Entire repository is 31GB. If I run a shallow clone, that cuts it down to a 12GB download. The SNES thumbnails represent 1.2GB of that - perhaps there's a case to split the SNES directory up into separate parts?

Juzamous commented 7 years ago

Have the exact same issue with Lakka ... all other system are downloading fine. Not working from PC or Lakka Directly :(

c0d3h4x0r commented 7 years ago

I can confirm the exact same issue on my GPX XD device. This is hugely annoying. Some other consoles' thumbnails also took several retries to finally download successfully.

c0d3h4x0r commented 7 years ago

Suggestion: update libretro to use a partial trie of folders (case-insensitive, since Windows and OSX treat different casings as the same folder, and substituting underscore for space) as a lookup mechanism to prevent any one folder from containing too many files. As long as the libretro code is written in such a way that it walks the folder hierarchy as deeply as it can when looking for a given file, the trie depth need not be constant/fixed, allowing the flexibility to restructure portions of the repository as needed to continue adhering to the max-files-per-folder limit as the ROM set for a given platform (such as MAME) grows over time.

Examples:

libretro-thumbnails/Nintendo - Super Nintendo Entertainment System/NamedBoxarts/3//n/3 Ninjas Kick Back (USA).png libretro-thumbnails/Nintendo - Super Nintendo Entertainment System/Named_Boxarts/7/t/h/7th Saga, The (USA).png libretro-thumbnails/Nintendo - Super Nintendo Entertainment System/Named_Boxarts/a/c/m/ACME Animation Factory (USA).png libretro-thumbnails/Nintendo - Super Nintendo Entertainment System/Named_Boxarts/a/c/t/ActRaiser (USA).png

wafflesinmybaubles commented 7 years ago

@RobotLimeLtd

Also does not download from GPD XD Android Device. Download progress goes 0%, 1%, 2%, then resets back to 0% in an endless loop (until you quit RetroArch).

It's actually still downloading. It's just counting in intervals of 2 in your case for some unknown reason. So it will continue to count in 2's until it reaches 100%. So wait for it to keep resetting to 0% 50 times.

mangax2 commented 6 years ago

There were some needlessly huge box art images being used for thumbnails (some were well over 1000px height or width). So, I resized all of them to be no wider than 256px and no taller than 512px -- then converted them to 256 color as well. The quality is still great and, most importantly, it resulted in a considerable size reduction. The MAME collection alone went from 7.18gb to 2.9gb. I just uploaded them to DropBox in case anyone else might be interested ... https://www.dropbox.com/sh/rigty96qbeps0mq/AAAlgb1ludrSqoFXuJ_KrFfBa?dl=0

artik commented 6 years ago

Fantastic MangaX2! Thanks those packs

lineacero commented 6 years ago

Fantastic MangaX2! Thanks those packs