libretro / RetroArch

Cross-platform, sophisticated frontend for the libretro API. Licensed GPLv3.
http://www.libretro.com
GNU General Public License v3.0
10.06k stars 1.81k forks source link

[Suggestion] Reduce amount of extracted files #12141

Open andiandi13 opened 3 years ago

andiandi13 commented 3 years ago

There's thousand of files when we extract retroarch, which can be a pain when copying the folder elsewhere or even during the extraction.

It'd be great if files could be packed in .zip files according to categories.

Example of big folders that could be zipped :

Shaders.zip (around 4000 files) Cheats per hardware (Nintendo - Nintendo DS.zip, containing all .cht inside) Assets folder, with each driver, like xmb.zip. Overlays, thumbnails... Anyway, any folder with a huge amount of files.

Files would be extracted inside retroarch each time it's needed. I can't estimate the feasibilty of such thing, but having so much much files can be really annoying.

Retroarch 1.9.0 Windows x64

keithbowes commented 3 years ago

These things are already available as ZIP files (though, perhaps not in the exact structure that you want). It would surely just be a matter of using the existing code that extracts ROMS to extract those things, but I've never been successful at using zlib to do anything, so the person to do it isn't me.

I do have a Windows version of RetroArch for testing purposes (I run it in WINE). The .7z file extracts all the contents to the same folder (so that would be easy to transport; just move the entire folder). That seems different for the installer, which extracts the binary to %ProgramFiles% and the support files to %AppData%.

BTW, there are currently three groups of shaders files: Cg, GLSL, and GLSLang. I have no idea why the Cg shaders are still around (even the Cg website advises to use GLSL or (on Windows) HLSL). GLSLang seems to be libretro's attempt to create a shader language that can be used with multiple graphics APIs (OpenGL 3.2+, Metal, DirectX, etc), seemingly only keeping GLSL around for mobile devices (where OpenGLES can only use GLSL shaders). Anyway, all three seem to be bundled with the Windows version of RetroArch, but I don't think they all should be in the same ZIP file.

Here on Linux, Cg shaders are useless outside of the legacy X windowing system; in both DRM and Wayland, they don't work.

Anyway, most of these things are optional. In fact, the reason I started using rgui as the menu driver was it was the only one that didn't complain about missing assets.

andiandi13 commented 3 years ago

These things are already available as ZIP files (though, perhaps not in the exact structure that you want). It would surely just be a matter of using the existing code that extracts ROMS to extract those things, but I've never been successful at using zlib to do anything, so the person to do it isn't me.

I do have a Windows version of RetroArch for testing purposes (I run it in WINE). The .7z file extracts all the contents to the same folder (so that would be easy to transport; just move the entire folder). That seems different for the installer, which extracts the binary to %ProgramFiles% and the support files to %AppData%.

BTW, there are currently three groups of shaders files: Cg, GLSL, and GLSLang. I have no idea why the Cg shaders are still around (even the Cg website advises to use GLSL or (on Windows) HLSL). GLSLang seems to be libretro's attempt to create a shader language that can be used with multiple graphics APIs (OpenGL 3.2+, Metal, DirectX, etc), seemingly only keeping GLSL around for mobile devices (where OpenGLES can only use GLSL shaders). Anyway, all three seem to be bundled with the Windows version of RetroArch, but I don't think they all should be in the same ZIP file.

Here on Linux, Cg shaders are useless outside of the legacy X windowing system; in both DRM and Wayland, they don't work.

Anyway, most of these things are optional. In fact, the reason I started using rgui as the menu driver was it was the only one that didn't complain about missing assets.

Thank you for your answer. The folders are available as .zip files (like when we update them in RA Online Updater) but unfortunately, they can't be put as this in the file structure of retroarch :/

As you said, RA has so many useless things included in the main archive ! Example, I don't want xmb menu driver, and it has 4000 files !! Other things are obsolete has you said as well...

I think Retroarch should be minimalist regarding amount of files, and you should download more files if you want. Just like cheats, which currently has 23.000 files, imagine if it was included by default !

inactive123 commented 3 years ago

We will have to think of ways we can tackle this. Having to write a ton of files on an SSD for every update can definitely contribute to a lot of wear and tear, so that is an immediate concern.

andiandi13 commented 3 years ago

We will have to think of ways we can tackle this. Having to write a ton of files on an SSD for every update can definitely contribute to a lot of wear and tear, so that is an immediate concern.

I thought so. In the meantime, it would be great to remove all useless files from the standard retroarch pack. For instance XMB has as said before, that has way too much files. This files or features could be downloaded separately maybe, or included but as a zip to extract before using.

andiandi13 commented 3 years ago

@twinaphex I checked all files of a fresh Retroarch extraction (1.9.1 windows) and here is a list of secondary or useless files that could either be removed or included as a zip file to extract :

If you add everything up you have almost 9000 files !

Retroarch 1.9.1 raw = 10500 files Retroarch 1.9.1 without all these files = 1500 files

Literally ten times less files.

I thought about a "Retroarch basic" or "lite" version that you can add removing all these files (and more) or rethink the way assets are downloaded, not providing and assets.zip with so much files, but an assets.zip with the minimum required fiels for each menu driver, and then a themes.zip with all other assets and skins of each menu driver.

Some options could also be removed or hidden by default to make it more user friendly (Achievements, Images, Music, Latency, Logging, and all these things that the common user don't use at all)

andiandi13 commented 3 years ago

Update : I wanted to add that there are many useless folder containing system files that just confuse the casual user, namely :

bearer iconengines imageformats platforms styles

All these files (DLL on windows) should be in Retroarch root folder (and btw it'd be more convenient if all system dll files in root folder were in only one folder, like "libraries")

hizzlekizzle commented 3 years ago

I think those all come from Qt and can't really be moved or omitted, unfortunately. The easiest thing, I think, if you don't want to see all the clutter is just make a shortcut/launcher one directory level up.

andiandi13 commented 3 years ago

I think those all come from Qt and can't really be moved or omitted, unfortunately. The easiest thing, I think, if you don't want to see all the clutter is just make a shortcut/launcher one directory level up.

Really ? I don't really understand why qgenericbearer.dll has to be in "bearer" folder and wouldn't be recognized in "libraries" folder for example.

andiandi13 commented 3 years ago

@twinaphex Any hope to see RA become lighter anytime soon ? It has many updates these weeks but it come with the same crazy amount of files.

Yrouel commented 2 years ago

Updating RA on my 3DS is painful, via FTP is almost impossible and having to physically move the microSD is super annoying oh and RA is so insanely "fragmented" that Universal Updater just doesn't work and those devs are struggling to find some workaround.

I really hope this issue gets seriously picked up, it's time for RA to go on a stringent file diet...

neakmenter commented 2 years ago

I approve this idea for sure! Every time I update or move my RA folder - my word it takes ages! AND if you're using a medium with a large allocation chunk size (such as the default large ExFat SD Card format) the RA folder can take a massive amount of wasted space from small files. Also, how about the ability to put whole systems library as a singe (uncompressed) zip (with the actual 1 zip 1 rom zips inside) so that it they may be transferred as one large file?

Yrouel commented 2 years ago

The xmb folder can be ditched completely for the 3DS release. rgui is plenty as default and removing xmb significantly reduces the number of files to copy like 4519 less, it's bonkers

LibretroAdmin commented 2 years ago

Update : I wanted to add that there are many useless folder containing system files that just confuse the casual user, namely :

bearer iconengines imageformats platforms styles

All these files (DLL on windows) should be in Retroarch root folder (and btw it'd be more convenient if all system dll files in root folder were in only one folder, like "libraries")

This has been addressed now today, should be implemented in nightlies and should be ready for the next stable.

LibretroAdmin commented 2 years ago
  • Wii U folder from assets/pkg (100 files) : why is it even here on Windows version ?

Pushed this change through now for the non-WiiU platforms as well, should start being stripped out of nightlies (and the next stable) from now on.

LibretroAdmin commented 2 years ago

switch foler from assets/pkg : same (just 11 files)

Removed this too for non-Switch platforms

LibretroAdmin commented 2 years ago

autoconfig (770+ files ! Why not just make a big autoconfig.ini with all devices ?)

While your main point still stands, we did change things so that the folders/files for other platforms are removed now before the package is built (i.e. Linux version no longer has autoconfig/dinput and autoconfig/xinput, and likewise for Windows with autoconfig/udev and autoconfig/x, just to cite one example).

LibretroAdmin commented 2 years ago

Some more optimizations have taken place on the infrastructure level:

Yrouel commented 2 years ago

Some more optimizations have taken place on the infrastructure level:

No love for the 3DS? Copying thousands of files to a microsd is really frustrating and even more so through ftp. I mentioned in a previous comment that the xmb folder can be completely removed from the release and made optional

LibretroAdmin commented 2 years ago

And then if the user selects XMB, all assets would be missing and they'd complain about that.

Would anyone miss it if XMB were removed from the 3Ds build? It probably doesn't have the RAM headroom anyway and people are probably better off with RGUI (or Ozone/MaterialUI, although they would eat some RAM too)

Yrouel commented 2 years ago

And then if the user selects XMB, all assets would be missing and they'd complain about that.

Can it be downloaded at that point? Or just remove it from the menu if it won't work properly anyway.

It's 4500+ files alone it would make quite the dent

andiandi13 commented 2 years ago

I think XMB issue should be addressed somehow. It's extracted by default IIRC right ? 4000+ files is very much just for a theme, would it be possible to include it as an archive and then extract it as soon as XMB is select as a menu driver ?

LibretroAdmin commented 2 years ago

The issue needs more thought and more work than you guys might realize. So just be appreciative of the current reductions for now and maybe some solutions will emerge in the next upcoming weeks

andiandi13 commented 2 years ago

For sure its already a great improvement

neakmenter commented 2 years ago

This is great work and already will speed up copying the folder over a network and reduce SSD wear! Well done Libretro Team!

andiandi13 commented 2 years ago

@LibretroAdmin Oh, I just remembered something about XMB. Actually there are additional themes installed by default, but if you only include the "monochrome" theme it'd be only 700+ files and you can get rid of 4500+ files 2022-09-07 15_10_45-RetroArch

I don't think many users would complain about having to download extra XMB themes.

LibretroAdmin commented 1 year ago

OK we don't have a real code solution yet for the extra icon packs but since Android already only bundles the 'monochrome' icon set as is, I've decided to take the plunge and only have the 'monochrome' icons for 3DS/Vita as well in the install package. So this should cut down on the thousands of files you guys are referring to.

Code solution can hopefully come later where it will become more modular to install custom sets without it all being so hardcoded.

EDIT: OK, this has been done. Will apply it for WiiU and Switch as well.

Yrouel commented 1 year ago

EDIT: OK, this has been done. Will apply it for WiiU and Switch as well.

I still think xmb could be axed entirely (I see you mention a potential code option for the future) but this is a pretty good compromise. From 5235 items to 714, quite the dent!

andiandi13 commented 1 year ago

@LibretroAdmin Are these changes planned for next update for Windows and Linux ? Because nothing changed in v1.11.0 or nightly - but it looks like it has for other system.

Jaffacakelover commented 1 year ago

I've decided to take the plunge and only have the 'monochrome' icons for 3DS/Vita as well in the install package. So this should cut down on the thousands of files you guys are referring to.

Code solution can hopefully come later where it will become more modular to install custom sets without it all being so hardcoded.

EDIT: OK, this has been done. Will apply it for WiiU and Switch as well.

Unfortunately this may have mildly broken 3DS Retroarch's GLUI menu driver (#14544), although I can't prove that it used [the now removed] Ozone assets without further testing. I was going to suggest the 'more modular' download option as part of this: Rather than downloading all themes at once, a submenu could open with a list of the themes (and perhaps a short description and preview thumbnail) so users can download asset packs individually.

Jaffacakelover commented 1 year ago

The prospect of using an atlas / spritesheet for PNG assets, to save on I/O calls, was proposed in https://github.com/libretro/retroarch-assets/pull/435 . I did a quick test, using Leshy SpriteSheet Tool and Pinga compression for this example.

Ozone **ozone/png/icons** 642 files (PNG) **746 KB** (764,140 bytes) **ozone png spritesheet** 2 files (PNG + TXT) PNG = 477 KB (488,677 bytes) TXT = 25.2 KB (25,905 bytes) = **502 KB** (514,582 bytes) **ozone png spritesheet (compressed)** 2 files (PNG + TXT) PNG = 267 KB (274,195 bytes) TXT = 25.2 KB (25,905 bytes) = **293 KB** (300,100 bytes)
Monochrome **xmb/monochome/png** 710 files (PNG) **2.29 MB** (2,411,172 bytes) **monochome png spritesheet** 2 files (PNG + TXT) PNG = 4.33 MB (4,548,392 bytes) TXT = 30.0 KB (30,775 bytes) = **4.36 MB** (4,579,167 bytes) **monochome png spritesheet (compressed)** 2 files (PNG + TXT) PNG = 1.96 MB (2,061,721 bytes) TXT = 30.0 KB (30,775 bytes) = **1.99 MB** (2,092,496 bytes)

As Ozone and Monochrome are grayscale icon sets, they will see greater savings compared to full-colour icon sets such as FlatUI/FlatUX.

RenanSD007 commented 1 year ago

It would be great to remove Shaders on all platforms that support them, on Android for example there are currently +3500 shaders that I and many don't even use this and for those who want to use there is already an option to download, Removing these files not only saves space, but also shortens the installation and extraction time of the files.

Jaffacakelover commented 1 year ago

One quick idea - The assets for individual cores could be moved into the core packages themselves - if you've not installed an N64 core, why would you need the N64 GUI assets? They could then be extracted into the assets folder when the core is downloaded.

These are the downsides I see though: • You'd have to add an Assets folder to every core, rather than in 1 central location. • Any updates to assets would have to be uploaded to those separate core projects. • For the N64 there's 3 cores, so you'd have to add the same assets to 3 packages (and make sure they're ignored / overwritten if a user installs more than 1 N64 core). • This approach wouldn't work with the 'spritesheet' idea either. • The core doesn't need anything unpacking / extracting right now, it's just downloaded and stored.

Some of that could be solved with scripting maybe, but it doesn't sound worth the trouble.

RenanSD007 commented 1 year ago

Any updates on this? Whenever I install a new version of RetroArch it extracts the shaders that I had previously deleted

If not remove from base.apk at least give an option to disable shader extraction, as I said before there is already an option to download the shaders whenever you want

hizzlekizzle commented 1 year ago

You can use the F-Droid version, which doesn't come with anything except for the bare minimum assets needed for the GLUI menu.

RenanSD007 commented 1 year ago

You can use the F-Droid version, which doesn't come with anything except for the bare minimum assets needed for the GLUI menu.

I always want to keep it up to date and I saw that F-droid only has an old update, and also I use XMB menu instead of GLUI, for me only the shaders are a problem because having to keep deleting about 4000 files with each update is annoying, the rest of the files I don't see a problem with being in the base.apk

hizzlekizzle commented 1 year ago

fdroid has the latest stable: image

You can still use XMB, you'll just need to download the assets. You can choose not to download the shaders.

the rest of the files I don't see a problem with being in the base.apk

That's the thing, though: you want the XMB assets but you don't want shaders. Other people want shaders but don't want the XMB assets. This is clearly a never-win situation for us.

RenanSD007 commented 1 year ago

fdroid has the latest stable:

It would be nice to at least update monthly to keep up with the nightly version :)

That's the thing, though: you want the XMB assets but you don't want shaders. Other people want shaders but don't want the XMB assets. This is clearly a never-win situation for us.

I see, thanks for the answer, I will try to use the F-droid version and install the XMB menu