search

libretro / flycast

Flycast is a multiplatform Sega Dreamcast emulator. NOTE: No longer actively developed, use upstream repo for libretro from now on - https://github.com/flyinghead/flycast
http://reicast.com
GNU General Public License v2.0
155 stars 77 forks source link

Texture Dumping creates duplicates with different names #641

Open GranvilleR opened 5 years ago

GranvilleR commented 5 years ago

While dumping and loading textures, i've noticed that often multiple copies of the same texture will be dumped under different file names. I first noticed this when fiddling with Rayman 2, but it also affects Sonic Adventure and seemingly other games as well. I looked through an archive of a bunch of textures for Powerstone 2 that had been batch AI upscaled and noticed there were several identical duplicate copies of textures in the pack (though nowhere near as many as some other games). I assume this problem occurs in all games to some extent.

This duplication in both Rayman 2 and Sonic Adventure occurs when a new area or level is loaded. In Rayman 2 for example, well over 8 copies of the same basic grass texture were dumped under the following different file names within just the first two levels-

https://i.imgur.com/XRJDZlZ.png

1d2aaf53 1d29f48b 1d284dfb 1d295b4b 1d29419b 1d29759b 1d298073 6b88cfa7 9521e1ab a386572f

If I progress further in the game, many more copies would also be created with new names. These are just a handful of examples, every single environmental texture in the game is affected by this issue. Note that these duplicate file names are not randomly generated every time the areas are loaded at least, the dumped file names remain consistent even if you replay the same areas and levels.

In Sonic Adventure, I noticed that this duplication issue extends even to the common object textures reused throughout the game, even the pattern used to simulate a reflective environment map on rings. This again happens every time the game loads a different area, even different sections of the same Station Square hub for instance (and areas of levels that are separated by loading screens).

It IS technically possible to create custom texture packs for these games, but you have to include every single duplicate in the load folder for it to work. This would require a ton of unnecessary time and work and be especially wasteful on storage space. I created a 4x AI upscaled Rayman 2 pack for the N64 version (using the Dreamcast textures as a basis), and the folder is about 200MB. A hypothetical Dreamcast conversion of this pack would conservatively be at least in the 250-300MB range due to the DC version having a greater quantity of textures overall. But that size is easily going to skyrocket to dozens of times larger if you have to duplicate many of the same highres textures over and over again. It could easily reach into the double digits of gigabytes. given how many levels and areas often reuse assets.

I've discussed this with some of the developers on the Libretro Discord server a few months back and as I recall, they verified this issue on their end as well. I just wanted to create an issue here in case it's ever forgotten about. This issue also occurs on the emulator that Flycast was branched off from.

Koloses commented 4 years ago

I'm having the same issue with Stunt GP. While the textures are 1.6 GB my pack is now over 6GB due to the fact I have to include duplicate texture loads to replace. I see that PPSSPP has a solution where you have a .ini file for textures that you can use to map image to given address.

Also the other interesting thing is that sometimes MD5 sum doesn't match for duplicate images(it does for most copies) but the actual image content is the same so I had to filter textures using some powershell scripting and imagemagick to make sure I'm not replicating the work that is already done.

flyinghead commented 4 years ago

The way to compute the hash of textures has changed since this issue was opened. There should now be less duplicates than before, but there will always be some.

The hash of the texture depends both on the image content and some other parameters (native format, palette, etc.) In some cases these parameters will differ but the resulting image will be the same. But in some other games, that difference must be taken into account.

Also note that textures using a palette are dumped as RGBA and, if the game changes the palette, you will end up with multiple dumps of this texture, one per palette.