Compressed disc image support

[rtissera]

Hello,

I am working on libchdr and batocera.linux amongst others. First, congrats for the great progress done on xemu, it's really impressive.

I would like to reconsider this issue : https://github.com/mborgerson/xemu/issues/129 HDD games converted back to ISO (which is basically, similar to the original ISO stripped from random junk data if I am not wrong), could be compressed very efficiently with CHD.

I can help with CHD implementation in xemu if needed. I am open to discussion on some other kind of solution like a custom format (like compressed iso block compression based on zstd for instance).

Regards, keep up the good work !

[liberodark]

Indeed CHD is the most widespread format and surely also the best format to compress iso. Especially when we are already making xiso (so iso from an HDD version) to save space. It would be really great to be able to make CHD which could still compress the game losslessly.

Proof :

• 007 - From Russia With Love 6.21 Go (iso Standard)
• 007 - From Russia With Love 2.0 Go (xiso)
• 007 - From Russia With Love 2.0 Go (hdd)
• 007 - From Russia With Love 1.7 Go (chd)

This game is one of the worst examples. Because in some games we earn more than 1 Go

[Darknior]

WOAW yes if possible CHD like on all other console will help a lot to save hard drive space :) The MAME chd library can be a good start point :)

[Veuks]

Good initiative

[zognic]

Great idea!!

[Redemp]

That would be a great idea 💡! Saves slot of space!

[KevinLeenders]

One format to rule them all!

[mborgerson]

@rtissera I agree that there is opportunity to reduce disc image size through introduction of support for some format which enables a block compression scheme. Although I do not anticipate extreme compression ratios, I do expect a 10-20% reduction in disc image size is possible (varies by game, it is not uncommon for games to already compress their data); that is, if the user wishes to exclude other parts of the disc image such as the randomized data stream and video partition, including these parts would not compress well and diminish the overall compression ratio considerably.

I'm not convinced that CHD is the ideal format yet, but that's only because I'm not familiar with it. @Ernegien and I have been considering introducing a custom disc image format to support all features of the Xbox disc security system for accurate emulation, but this work has stalled temporarily due to higher priority tasks taking precedence. Such a format would ideally also include optional opportunistic compression.

You are absolutely welcome to prototype your proposed solution, and if it is a good fit + brings substantial benefit I will be motivated to include it. Thanks for volunteering to work on it, and keep me updated on your progress.

P.S. As an aside, when rallying support for a proposal in the future, I ask that you please encourage your followers to simply use a 'thumbs up' emoji reaction as opposed to a comment of support in order to keep the conversation substantive.

[rtissera]

@mborgerson sounds like the batocera crowd got a little too excited about the idea, sorry about that ! I started playing around in a branch on my fork. It's not ready yet for PR or whatever, but I put up some ground work already.

https://github.com/rtissera/xemu/commits/chd

Let's see how we can make things move forward.

[GXTX]

Theoretically the "garbage" data in important, it's pressed onto the disc for specific reasons whether it be due to alignment or using it for nonce. It's an a very unfortunate thing that the redump standards strip this from the image for what ever reason, and this should change as these discs will only die more often as time passes - we should be striving for perfect backups.

Another thing to consider is the fact that drive space is becoming more and more cheap as time rolls on, an entire known collection of say North American games would only occupy ~6TB with the entire known world collection including duplicates being ~18TB. Now let's be honest with ourselves here and be extremely truthful - people who own entire collections aren't prevalent in the scene, and even if they were a 10TB drive is sub-$200 USD.

This is a non-issue that only serves to complicate things, and seems to come from some "emulation scene wide" push to push CHD on emulators.

[rtissera]

I may be wrong but I think the redump team is ripping 1:1 including the video partition (which needs to be stripped to have a working image with xemu as stated on the github instructions). Adding CHD support does not mean you have to use ISO converted from extracted "meaningful" data. You could still convert a redump XBOX disc and - of course CHD compression will be less efficient because of the random "garbage" data - but would still help save some space.

Le lun. 8 mars 2021 à 20:20, wutno notifications@github.com a écrit :

Theoretically the "garbage" data in important, it's pressed onto the disc for specific reasons whether it be due to alignment or using it for nonce. It's an a very unfortunate thing that the redump standards strip this from the image for what ever reason.

Another thing to consider is the fact that drive space is becoming more and more cheap as time rolls on, an entire known collection of say North American games would only occupy ~8TB with the entire known world collection including duplicates being ~18TB. Now let's be honest with ourselves here and be extremely truthful - people who own entire collections aren't prevalent in the scene, and even if they were a 10TB drive is sub-$200 USD.

This is a non-issue that only serves to complicate things, and seems to come from some "emulation wide" push to push CHD on emulators.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mborgerson/xemu/issues/238#issuecomment-793007509, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAW4WBMDO3G7GV6KSE5YMHLTCUPODANCNFSM4YXYXLKQ .

[GXTX]

I may be wrong but I think the redump team is ripping 1:1 including the video partition

The security sectors and others are stripped from the base iso generated from DICUI as well as the in between data. These are imperfect dumps and it's been brought up with redump members before.

Adding CHD support does not mean you have to use ISO converted from extracted "meaningful" data.

You're right - it wouldn't force anything - however... I can only think of the negative side of the scene in regards to their data acquisition methods. I believe as a whole you'd end up more users not caring about a better backup in favor of something smaller. The proof of this is even present in this issue thread with you and the 2nd post talking about "HDD" - this is a scene term that has very negative connotation - people who own their games and extract them from a properly dumped disc I don't see using this term.

[mborgerson]

Another thing to consider is the fact that drive space is becoming more and more cheap as time rolls on

@GXTX: Plummeting storage costs lowers contention for space and urgency to compress data, but think about the other files someone might have on their system: do you expect most people to store movies, music, and images all in uncompressed formats too?

With a relatively modest collection of games on my laptop used for testing, I saved ~36GiB when compressing for storage, reducing overall size of the collection by ~23%. In addition to storage, I'm sure those who keep their backups on remote systems would appreciate the savings and reduction in transfer time.

If a method to compress ISOs with fast real time decompression rates is available, I would personally use it. For those who do not, they should be able to opt out. If someone wants to spend their time working on this feature, all the better.

This is a non-issue that only serves to complicate things, and seems to come from some "emulation scene wide" push to push CHD on emulators.

Again, I haven't researched CHD so I'm not sure it will be a good fit for xemu yet. That said, there are issues with existing XISOs that a new format should help address:

• Capturing all available details of a disc for emulation or reproduction
• Optionally compressing disc data for storage savings

I may be wrong but I think the redump team is ripping 1:1 including the video partition (which needs to be stripped to have a working image with xemu as stated on the github instructions).

@rtissera: Today this is true, but only because xemu lacks the ability to switch to the game partition. Eventually the plan is for xemu to more accurately emulate Xbox security features, eliminate need for a patched kernel, and supporting switching over. Of course, if a user still decided to keep their XISOs without all the security features and use a patched kernel, this would still work.

I can only think of the negative side of the scene in regards to their data acquisition methods. I believe as a whole you'd end up more users not caring about a better backup in favor of something smaller.

@GXTX: I support those who want true 1:1 backups of their discs, I would like perfect copies myself. But lets consider that many xemu users just want to play their games and they would happily sacrifice a 'better backup' for something which, from their perspective, is functionally equivalent and takes up a fraction of disk space and still lets them play their game.

[Shoegzer]

@GXTX Fully agree with your suggestion. The mame team has strived for perfect rom, cd, hd etc. dumps for decades and history has shown it's paid off there too. It's never a good idea to archive and use iso images with data stipped out, even if the data is deemed nonessential. From the standpoint of preservation and historical accuracy it's dangerous even to support such formats given that it increases the chance of ongoing circulation of "bad" images in the wild.

We got lucky with dreamcast gdi dumps but I think that was primarily due to the obvious inferiority of alternatives - for example many early "scene" dumps downgraded images to cdi with downsampled mp3 vs. wav audio tracks, stripped cutscenes and other horrible things. Thankfully the scene has been fairly effective at healing itself and 1:1 gdi dumps are generally more available now, as people tend to only archive and use those. It took a long time, but at least the situation didn't deteriorate.

With this in mind I'd certainly be in favor of xemu supporting only full dumps and ideally deprecating support for xiso, but since the latter may never happen, at least a full dump option would be fine. It's great that's being considered on such a fantastic and well-coded project as xemu. There is currently a need to keep both xiso and full images until the latter are supported.

You're right that hard drive space is cheap these days and only going to get cheaper in the future. From what I've seen, most people generally don't care so much about space today, especially if they can have perfect dumps. That sentiment is only going to grow in time.

The chd format is widely used not only with mame but many other emulation projects. I'm not sure if it would support xbox security features but I think it's worth a look and/or outreach to the mame team.

[mborgerson]

Further discussion is welcome on Discord chat; locking the issue for now.