search

rtissera / libchdr

Standalone library for reading MAME's CHDv1-v5 formats.
BSD 3-Clause "New" or "Revised" License
99 stars 42 forks source link

Support chd parent/clone loading #17

Closed Sanaki closed 3 years ago

Sanaki commented 4 years ago

As was discussed in https://github.com/libretro/beetle-psx-libretro/issues/587, support for this would be beneficial for various retroarch cores, especially for PS1. Presumably the frontend/core would handle parent locating, so that part at least shouldn't be necessary to implement in this library.

rtissera commented 4 years ago

Do you have some example parent/clones CHD to share or some link / instructions on how to generate such CHDs from BIN/CUE files ? The API is so far indeed expecting a single file, that's the biggest issue I can think of. I skipped that feature when writing libchdr because it did not seem like very important because the library gained traction.

Sanaki commented 4 years ago

Example: Parent is Legend of Mana (redump). This is the standard edition of the most downloaded PS1 hack on romhacking, the combat redux for Legend of Mana. I used this as an example to ensure the uploaded chd doesn't contain copyrighted content. It's zipped purely to allow uploading on github, and it contains only 28 modified hunks, with the rest sourced from the parent. Legend of Mana (USA) (Combat Redux) (clone).zip Hack source: https://www.romhacking.net/hacks/2085/

Creation method: chdman createcd -i "Legend of Mana hacked.cue" -op "Legend of Mana.chd" -o "Legend of Mana hacked.chd"

Extraction method: chdman extractcd -i "Legend of Mana hacked.chd" -ip "Legend of Mana.chd" -o "Legend of Mana hacked.cue"

Personally, my biggest interest is in handling hacks and translations without excess space and without removing the clean dump. For retroachievements, this would allow distributing the bonus set hacks as clone chds (around 6KB each) and simplify that quite a bit. For general usage, this would allow designating the first disc of any given game as a parent to save potentially a significant amount of space from repeat data across discs.

Bonus FF7 disc 1 chd, in case you're curious: \~Bonus\~ Final Fantasy VII (USA) (Disc 1 of 4).zip

TheRealYasri commented 4 years ago

This maybe be stupid, and not sure who in the development stream would need to do it, but would't the best route for the end user be if a game is multi disk to make it into a single chd. I can not think of a use case were you would want 1 disk of a set of say 3 to not be kept together.

It would also let you not have to store the same data mirrored on multiple disks, you just store it once in a single chd and transparently stitch it back together for anything using it outside the library.

Support some type of universal checksums stored in the chd, so external programs can read it and see what disks are contented inside. This checksum would need to be the same even if independent people coveted the same original files. So I would think it would be best to checksum the original file and use that.

Added support for patches to be added to the main chd (with there own checksum same as adding extra disks) or as a external file in the same folder. Could be something like this. gamename.chd gamename.001.chd gamename.002.chd

The naming order of the patch chd files could not matter by having the chd patches store its info like name, version, what checksum of disks its compatible with inside for a emulator to read and figure out which it wants to use and the order.

When a emulator loads the main chd file it would add the game as a single entry in the ui and if there are patches included(inside or in dir) they could add them as a menu options to pick which to turn on or off(with name and description info from the patch chd) and order to apply them before launching.

Then when launching the game, pass the selected game/patches names and their order to the library to do the heavy lifting and pass back the result. So stitching together of the files is independent of the emulator.

Maybe you can look at using physfs. Add support for chd to it and have it create the file system to hand back to a emulator to use. It could just see a file tree with just folders for each disc, with the patches applied to each disk already. You could do the same for hdd based system. just put each disk in its own folder with any patches already applied over the top. https://www.icculus.org/physfs/

For things like dosbox have the main chd be read only, then when the emulator changes files create a patch chd like above to house the filesystem changes. So to the emulator all it sees is a file system it can do whatever it wants to.

/cheers

Sanaki commented 4 years ago

This library implements CHD, it doesn't add to CHD's functionality. For that, you'll need to take it up with MAME over at https://github.com/mamedev/mame/.

That aside, I personally wouldn't use merged chds even if they were supported. That would create issues with retroachievements as well as disc datting. Considering the purpose for which chd exists, I'm not sure they'd be interested in implementing it either.

TheRealYasri commented 4 years ago

This library implements CHD, it doesn't add to CHD's functionality. For that, you'll need to take it up with MAME over at https://github.com/mamedev/mame/.

Was thinking that was the case, but brought it up here because this is the library that non mame projects use as far as I understand. And thought the dev maintaining this would be more likely to understand the were I am coming from. And if he/she agreed with me, they would have more community standing to ask the upstream devs to add something that may not directly improve chd for how they use.

Sanaki commented 4 years ago

Given that it's not directly relevant to this particular issue, it would be better in its own issue regardless.

TheRealYasri commented 4 years ago

That aside, I personally wouldn't use merged chds even if they were supported. That would create issues with retroachievements as well as disc datting. Considering the purpose for which chd exists, I'm not sure they'd be interested in implementing it either.

With my idea the checksum from the original iso/bin/etc would be stored in the main chd, so when a emulator is scanning a folder it would only check the main chd grab the checksums in it and compare it against the dat you would normally use. And for retroachievements inside the emulator it could only allow it for whitelisted disks or patch checksums. Or when it send info to retroachive it could send the checksums along side for the disks/patches being used.

Given that it's not directly relevant to this particular issue, it would be better in its own issue regardless.

I think its on topic for "Support chd parent/clone loading". Just that it is a complex topic that effects many different things if you what a coherent system. And I was trying to be thorough for any edge cases I could think of.

TheRealYasri commented 4 years ago

I personally wouldn't use merged chds

With my idea you also would not need to merge a mulidisk set into one chd if you did not want to. The first disk would the main chd and any other disks could be a "patch chd" in the the same folder using the above naming rules. It would be up to the user if they want to bundle it as one chd or as several. I would want the library to stitch everything together in the end, so a given emulator would see the same filetree no matter how the real files on real harddrive are broken up.

Was just adding a option if a game is 4 disks and the person had ten different mods, they could package them up as a single file if they want for convenience. While still being able to reverse the process if they need to later.

Personally I think this would work better then the current system mame uses for all its arcade system.

If you want merged put all the file in parent.chd

if you want non-merged set parent.chd child.chd

if you want split parent.chd parent.001.chd

The emulator would read the main chd and get the metadata which could have what types it is and emulator could decide what else it needs to look for and how to display everything to the end user based on that and user settings.

Sanaki commented 4 years ago

I think its on topic for "Support chd parent/clone loading".

I disagree. This issue is still about implementing current functionality of chd that is missing from libchdr. You are talking about changes to the chd format that are currently unimplemented -and- out of scope for this repository. If you want to talk about your idea, you can reference this issue, but at least do so from an issue directly about that topic.

TheRealYasri commented 4 years ago

I disagree. This issue is still about implementing current functionality of chd that is missing from libchdr. You are talking about changes to the chd format that are currently unimplemented -and- out of scope for this repository. If you want to talk about your idea, you can reference this issue, but at least do so from an issue directly about that topic.

Guess will just have to agree to disagree then.

Just posted my idea for parent/clone loading and how it could be made to have simpler interface for emulators to feed data in and out of, so It does not need to worry about how the dirty work is done. And a more end user friendly file naming scheme that you would want to encourage all downstream projects to adopt.

Yes the project for chd upstream would need patches but @rtissera would have better standing to ask for that then me and I would think he would be more likely to understand how chd effects nonmame projects then the main dev who as they should be are more considered about how chd effects mame, sense its not really a separate project.

So while yes my ideas could not be done without up-streaming patches, I think they could be worth considering if he wants to add support for what your asking, because the end result I think could be better for everyone.

rtissera commented 4 years ago

Please keep on track with the issue itself, thanks ;)

alucryd commented 4 years ago

Looks like libretro is working on this: https://github.com/alucryd/beetle-psx-libretro/commits/r5/chd-clone-support

rtissera commented 4 years ago

I will merge back happily if properly done.

alucryd commented 4 years ago

I will probably send them a PR today to build against system libs, I'll see if they'd be willing to make a PR themselves at the same time.

Sanaki commented 4 years ago

That branch was created in seeing if libchdr could already do it. It doesn't appear any changes were made past this: https://github.com/libretro/beetle-psx-libretro/issues/587#issuecomment-580530781

rz5 commented 4 years ago

It's like @Sanaki said - seems like libchdr doesn't have the parent/clone logic in place. I was not willing to do the work of studying MAME's chdman.cpp and implement it.

What was done in beetle-psx-libretro's chd-clone-support branch was minimal:

That's all I did, I don't intend on resuming work on the feature. If you want a PR with the 3 small changes, I'll happily send them.