Subchannel data reading and CD copy protection Libcrypt.

[ADormant]

Many PSX emulators/CD plugins including Mednafen have problems with game protection in some games and require real discs will this be a problem with this emulator too? Will this emulator be able to decrypt Libcrypt-protected ISOs?

http://www.razyboard.com/system/morethread-subchannel-woes-pete_bernert-41714-382429-0.html http://www.razyboard.com/system/morethread-ff9-black-screen-pete_bernert-41709-1031440-0.html http://www.razyboard.com/system/morethread-sub-channel-reading-problem-pete_bernert-41714-176709-0.html http://www.razyboard.com/system/morethread-cdr-pluging-dosenund039t-read-drives-pete_bernert-41714-1398147-0.html http://forums.pcsx2.net/Thread-Why-isn-t-there-Copy-Protection http://psx-scene.com/forums/f20/ps2-disc-security-explanations-50245/ http://psx-scene.com/forums/f13/ps2-copy-protection-15762/ http://everything2.com/title/Playstation+copy+protection http://psx-scene.com/forums/f10/subchannel-data-102099/ http://www.tomsguide.com/forum/89-4-what-copy-protection http://consolecopyworld.com/psx/psx_protected_games.shtml http://consolecopyworld.com/psx/psx_libcrypt_tutorial.shtml http://snesorama.us/board/showthread.php?40865-How-to-Play-(libcrypt)-a.k.a.-protected-ps1-games.-Playing-multiple-disc-backups. https://forum.slysoft.com/threads/ps1-psone-backups.25000/ http://forum.fobby.net/index.php?t=msg&goto=2914& http://forum.fobby.net/index.php?t=msg&goto=3759&

Another nice option is automatic unpacking of ECM compressed ISOs OSX and Linux versions of PCSXR have it so probably it can be copied from it.

[simias]

If I understand No$ correctly the libcrypt key storage is relatively simple. On normal CDs the Q subchannel holds the "minute:second:frame" timestamp for the current sector. On CD players that's used to display the current position within the track apparently.

On libcrypt protected games some sectors have bad Q subchannel data with a wrong CRC:

[...] the modified sectors have wrong CRCs (which means that the PSX cdrom controller will ignore them, and the GetlocP command will keep returning position data from the previous sector).

So the subchannel doesn't exactly hold the key, instead the game can use GetLocP to know whether a given sector's subchannel data is good or corrupted and that gives one bit of information. Using 16 different sectors the game can reconstitute the 16bit key:

A modified sector means a "1" bit, an unmodified means a "0" bit. The 16bit keys of the existing games are always having eight "0" bits, and eight "1" bits

So the actual contents of the Q subchannel data doesn't matter too much, you just need to single out the invalid sectors (using an sbi file or similar) and return the previous sector's info in GetLocP.

It sounds relatively straightforward, the only potential issue I can imagine is that GetLocP seems to be a bit ahead of the currently read sector in certain cases in my tests. Currently in my code I always set it one sector ahead but I'm pretty sure that's not accurate, I think the controller can buffer several sectors (3 maybe?). Depending on how libcrypt reads the sector it might be necessary to fine tune this.

And of course there's the problem of the million different formats for PSX dumps, for now I think I want to focus on bin/cue + sbi since it seems to be the de-facto standard for "good" dumps.

Personally I think the frontend should handle the various CD image formats and CD-drive APIs across different platforms but that's not how it's done in libretro right now and discussing about it in #retroarch it seems not everybody agrees with me. So we'll have to reinvent the wheel once again it seems... Or maybe find a lib that handles it for us, maybe libcdio could be a good starting point.

[ADormant]

You can try Libarchive as well I think OSX and Linux versions of PCSXR(though probably some old version) use it. https://github.com/libarchive/libarchive

[simias]

Twinaphex mentioned that while adding a CD interface to the libretro API might be a bit too much it could be possible to put the CDROM code in libretro-common, which could then be used by other emulators.

That sounds like a good compromise, however the code has yet to be written... We'll see if other people are motivated to work on that. For the meantime I think I'm going to write a crappy cuesheet parser in rustation and maybe we'll factor it away when the time comes.

libarchive could be nice in order to support compressed formats but I think RetroArch already supports decompressing things like 7zip archives so it might not be necessary to bother with that.

[i30817]

I believe libarchive supports 7z/zip with a ecm'ed image inside, which is the best compression possible for psx i think.

[simias]

Isn't the most popular format 7zipped bin/cue? That's what's used in the big redump archives.

[ADormant]

I found also this https://github.com/centaurean/density I see a large number of ECM compressed isos online.

[i30817]

ECM supposedly works by removing junk data off the psx cds that can recreated. The process is reversible, and that's what the library does probably. Supposedly and in practice this is better for compression because 7zip doesn't take advantage of the pattern that ecm uses.

So it's a bit like compression on top of a compression... Speed is a problem ofc.

BTW, it's not a given that you need the cd to be untouched at the end of compression and decompression to be competitive. People will use a lossy format if it's convenient and doesn't matter, for instance the inbuilt compression in dolphin is gz, BUT dolphin removes 'junk' data of the filesystem prior to compression and never puts it back after decompression.

For decompression speed squash lz4 from the linux kernel seems very competitive (i'm planning to use it on a few dozen gigabytes compressed mountable archive overlay (with a rw layer) filesystem for my DOSbox games myself). It has a rust library for it, dunno how good https://github.com/bozaro/lz4-rs

[simias]

Yeah ECM removes all the error detection and correction data AFAIK. Maybe the sector header as well. For data sectors (Mode 2 Form 1) that's around 13% of the data and it compresses quite poorly with general purpose algorithms. And of course this data can be regenerated losslessly when needed.

I think for PSX you could save a lot of space for certain games by compressing the redbook audio tracks. It could be done losslessly with FLAC or some other lossless audio codec. It should typically compress the raw audio by around 50%, a general purpose compression algorithm like 7z will do worse on average.

Of course if you're willing to go lossy then you can reduce that even further but I'm not sure if it's worth it nowadays. It makes it more difficult to single out bad dumps. The entire PSX library across all regions is something like 2TB in size 7zipped (adding the size of the NA, EU and J redump archives). That fits easily on a single 50euro hard drive. By using a better image format we could probably shave off a couple hundred GB without losing anything but I'm fine with 7zipped bin/cue for the time being.

Eventually if we end up putting the code in an external project like libretro-common it might be worth adding support for all kinds of "exotic" image formats but at this point I don't want to spend weeks writing and debugging CD image parsers...

Of course if you or somebody else want to do it you're more than welcome to try!

[i30817]

I do that redbook audio lossy compression for dosbox games (it has builtin support as well as limited support for mounting dirs a 'cd' as longs as it doesn't have RB audio ... or you need to change cds) ... it's the only reason that game collection isn't in the hundred of gb instead of dozens.

It all depends on the size of your collection.

BTW, for psx games with redbook audio i've seen the 'ape' format used, which is a lossless compressor. I don't know if any emulator has native support, maybe. Regardless, nothing that flac doesn't do better likely.

I do agree the library way will end up being best probably. Cdemu, pcsxr and probably others use libarchive.