[Feature/Enhancement] CHD or Alternative Compression Format

[0x90shell]

Apologies if this is addressed elsewhere. I couldn't find any issue to reference.

Are there any plans for a compressed file format on Xenia's roadmap?

Currently, Xenia can read ISOs or their extracted contents. It would be helpful if a compressed format, ideally archival quality, could be used instead. CHD has become a standard for the majority of disc-based console emulators at this point, and it may be a good option for Xenia. CXBX-Reloaded is currently considering using CHD or XISO for a compressed format. CHD files would reduce storage requirements while allowing the contents to be read without a notable performance hit. There is an existing CHD library to ease some of the implementation headaches, as well.

[Razzile]

Xenia can load games in the Games on Demand (GoD) format, which while I'm not sure how compressed they are, they are certainly much better than raw ISO images. As a bonus as GoD is an xbox 360 format already they will be compatible with 360s too

[RinMaru]

CXBX-Reloaded is currently considering using CHD or XISO for a compressed format.

CXBX-R is not considering CHD anymore since XISO has garbage data that doesn't compress well. its considering its own format mostly

[0x90shell]

@RinMaru Yes, I noticed that. It appears they may need to take an approach like Dolphin did, which is more overhead for devs. I was hoping CXBX could leverage existing library.

@Razzile After a quick search, it doesn't appear GoD format is compressed at all. Quick search showed it can actually be larger than XEX(extracted) format. I hadn't considered GoD before, but I'll need to test when I get a chance.

[DocMAX]

Please add multidisk support with CHD

[BobbyMcJefferson]

Yeah, we definitely need this. I just converted my entire PS2 library to CHD and the reduction in filesize is ridiculous.

[tastyratz]

CHD brings many benefits. It's come up a number of times in other requests or reports for xbox emulators too such as xemu https://github.com/mborgerson/xemu/issues/129 .

A custom format sounds like it COULD be ideal, but, it creates a lot more heavy lifting for dev. When there are other priorities, it might end up being a big ask destined to get shelved.

Eventually, maybe a joint project with xbox/xbox360 emu devs (since they both have garbage padded xiso's) could work. RVZ for Dolphin was developed around compressing iso's with junk padding in them. Maybe that can be adopted in some way by other emulators that are not Nintendo and modified for xiso specifics?

I think adding CHD support via existing public libraries is probably a lower bar easier win though. It lets you compress with lzma just as well as any other 7z collection but with the added bonus of on the fly instant access streaming. Unzip times add up to a lot of loading. On the fly access also makes streaming across the network possible from a nas.

CHD files can be converted back safely without risk should someday a custom format be developed that works better. It might be an easy win.

[Triang3l]

How compatible is the format with seeking to arbitrary files and within files?

[ghost]

I kind of thought this issue would to sort of die out on its own, but since it's still alive I feel the need to address two elephants in the room:

1. CHD as a compressed format for "ISO"s.

   • In general, dumping Xbox 360 (and Xbox) disc images is a very difficult procedure, and something like 98-99% of all people wielding these ISOs are simply holders of pirated software, this due to the fact that these discs can only be read by a select few optical drive models, and you cannot dump disc images from for instance a modified console.
   • The disc image format troublesome on its own, as all Xbox and Xbox 360 discs are, as has been previously mentioned, padded with garbage data to somewhere around 8GB. Installing these games to hard drive, or simply copying the files from disc over FTP using a modified console is a much wiser choice than dealing with these disc images, as this eliminates the garbage padding completely.
   • There are some very rare exceptions where the game will only run properly from a disc image, but I can't remember what they are off the top of my head.

2. The massive life-changing space saving.

   • To keep things somewhat realistic, I decided to 7z up the ~135GB of data that I copied from the hard drive of my Xbox 360 console, 43 games in total.
   • The compression ratio for this was ~82%. Meaning that only ~18% of space was saved compared to the uncompressed GoD (Games on Demand) installs, or somewhere around 25 gigabytes. And this is with a relative highly efficient compression method.
   • This is a very insignificant amount in the grand scheme of storage space. Do you have more than 43 games sitting around that you absolutely can't spare the space for? Maybe you should delete them. :) Just kidding, but you know what I mean.
   • Like mentioned, the GoD installs already deal with the issue of garbage padding on the discs by not including it. You may be just as well off simply enabling some form of drive compression (NTFS folder compression?) for your games.

There are of course some games that will compress better than others, but I still cannot imagine it would be worth the time investment and subsequent performance to repack the games just to have them in a slightly less acessible format that you then have to convert back into a different intermediate format if you ever want to extract and/or edit some of the files or such. Maybe if there was a standardized library for STFS access, and CHD would support access through this somehow, but even then it seems rather doubtful.

There's also the matter of which console generation this is. It's not the PlayStation 2 where suddenly we had these gigantic new optical discs called DVDs and we didn't need to compress the assets for our games to fit them on there. It's more like the Xbox 360 where we are still on these optical discs called DVDs, and we have to fight pretty hard to fit all of our data on there.

[0x90shell]

@beeanyew Those are good points. For me the conversion resulted in 20-25% size reduction in Xbox 360 ISO files. I only have 5 games to try, and I ran them through CHDMAN, and it averaged 22% reduction by converting to the CHD format. Most people would say a 1/5th decrease qualifies as a significant reduction in any given field, but you bring up some good points of X360's DVD usage pushing the format to size limits. I couldn't say what average reduction would end up being for the entire library.

@Triang3l As I understand it, CHD supports the same read functionality available to the console's disc readers being emulated.

[gibbed]

As I understand it, CHD supports the same read functionality available to the console's disc readers being emulated.

Then CHD would not be the best choice for this if we wanted to come up with a highly compressible storage format for Xbox 360 titles. The Xbox 360 does not provide access to disc contents in this manner.

[koitsu]

I couldn't say what average reduction would end up being for the entire library.

Who cares? Why do you need the entire Xbox 360 library? Please get a grip.

[mirh]

If you want to save space, check these instructions. Xenia already incidentally supports something for that purpose.

But if you want a specific format for discs, then CHD seems kinda pointless given what bloats the majority of most dvds. And it wouldn't seem much fair to promote non-archive-quality formats either (it is alleged that even redump standards have some shortcoming in this regard, but idk maybe they have improved on this as of lately).

Assuming much of what applies to the OG xbox formats isn't superseded for the 360, the problem at the end of the day is that we don't know the seeds for the padding.

[Rex000]

We really need this. Other emulators already have this feature for a while now. Just a few games have occupied a humungous amount of space in my hard drive. Even games that are only a few hundred MB's on PS3 have 8GB iso's on Xbox 360.

[Triang3l]

8GB iso's on Xbox 360.

That's an issue specific to raw images of discs, however, SVOD/STFS and loose files take up as much space as the game actually uses.

[Rex000]

8GB iso's on Xbox 360.

That's an issue specific to raw images of discs, however, SVOD/STFS and loose files take up as much space as the game actually uses.

I was only exaggerating. While there actually are games that exceed 8GB's, a normal Xbox 360 game still has a 7.3GB disc/iso even if the actual game is a couple hundred MB's in size.

[Masamune3210]

So....extract the game from the disc image? I'm pretty sure xenia supports running a xex from a folder, does it not?

[tastyratz]

Game disc's had padding as a deterrent, that's not the same at all. Extracting the contents or repacking in another iso is still uncompressed storage without the padding.

CHD support would allow for lzma storage on the fly. Great for people who might be storage poor, saving images on a NAS, and CPU rich.

[Masamune3210]

The padding is not a deterrent, it served a purpose in that it forced the actual data on the disc out to the parts of the disc that could be read faster

[BobbyMcJefferson]

We really need this. Other emulators already have this feature for a while now. Just a few games have occupied a humungous amount of space in my hard drive. Even games that are only a few hundred MB's on PS3 have 8GB iso's on Xbox 360.

There is the ZArchive format and apparently people got it working in Xenia: https://github.com/xenia-canary/xenia-canary/pull/165

I tried compressing a game folder with ZArchive but Xenia couldn't load the file. In fact Xenia doesn't even know the filetype.

[RinMaru]

We really need this. Other emulators already have this feature for a while now. Just a few games have occupied a humungous amount of space in my hard drive. Even games that are only a few hundred MB's on PS3 have 8GB iso's on Xbox 360.

There is the ZArchive format and apparently people got it working in Xenia: xenia-canary#165

I tried compressing a game folder with ZArchive but Xenia couldn't load the file. In fact Xenia doesn't even know the filetype.

it hasent been merged yet thats why. not sure what the holdup is but i guess it needs to be reviewed.

[0x90shell]

Looks like they just merged it. 🔥

[DocMAX]

So in summary whats the best format to use currently?

[antoniodesousa]

So in summary whats the best format to use currently?

ZArchive but you'll have to use the Canary branch. It's not merged to master.

[Darknior]

We really need this. Other emulators already have this feature for a while now. Just a few games have occupied a humungous amount of space in my hard drive. Even games that are only a few hundred MB's on PS3 have 8GB iso's on Xbox 360.

There is the ZArchive format and apparently people got it working in Xenia: xenia-canary#165

I tried compressing a game folder with ZArchive but Xenia couldn't load the file. In fact Xenia doesn't even know the filetype.

Thanks for the information, for me this way is better than compress a CD game. Compress the game folder is better, and maybe will help us like on CEMU to add games the update files. To have a single file by game ...

[Papermanzero]

I have breaking news concerning CHD. CHD added the zstd/zarchive compression with mame version 0.262. That means all the benefits from Zarchive are now also available for CHD. Means the highest possible lossless compression (which is also used for cemu) together with dif file support and archive formats are now all avalable in chd.

so for isos as well as game folders (dlc, games) the best way to archive files is CHD.

[Darknior]

so for isos as well as game folders (dlc, games) the best way to archive files is CHD.

You are wrong, CHD is only good to compress an ISO !! If you want to compress a directory you must use same method than on Cemu for WiiU with the WUA file format. We already have it on Xenia, to compress directories and that take less place than ISO, it is perfect. We only miss a solution to include Updates and DLC in the game archive like on Cemu :(

This format is ZArchive and give .ZAR games :)

[Papermanzero]

Of course you can create archives. either you are using a raw image or you a using an iso in chdman. Here an iso example: Create a folder, put all the data inside and then make an iso out of it. https://github.com/iquirino/IsoCreatorTool Use this iso with createdvd in chdman an you have your chd.

The benefit is you can use the chd as image. So even if the game tries to write data to the chd (which is the case for some x360 and ps3 games) a dif chd can be created. With the according chdlib it is easy to implement chd. https://github.com/rtissera/libchdr

If we go back in time it would be like god2iso 😅

Edit: concerning dlc and updates: Even for that you can create an “update iso” what MS did during xbox original and x360 times.

Edit2: dont get me wrong. I am not saying zarchive is bad. Just that chd would be a great alternative. Like pcsx2 did with iso, cso and chd.

[Darknior]

Edit2: dont get me wrong. I am not saying zarchive is bad. Just that chd would be a great alternative. Like pcsx2 did with iso, cso and chd.

Yes i understand for sure. Why not an alternative. But i really stand and wish for an ZAR support with update and dlc included in an unique file.

[mooock]

Is it allowed somewhere to post bug reports on zar implementation? I experience very bad performance when running games from a network drive. running the same files from the extracted files everything is fine. also if i move the zar file to a SSD it also works fine. I know some now will say its because of network performance. but i run a 10GbE with SSD cache. Iso also runs fine from network. I think something to these archives needs to have some caching or maybe it has some early implementation that don't focus on performance yet. It tried all sort of cachen function in the config. but i could not fix the issue.

does any of you guys experience this?

the problem can easily be reproduced on Project Gotham 3. on the first track just pressing next.....