DIC incorrectly dumps the 7th guest game disc

[MrPepka]

Version 20230606 Describe the bug One of the dumpers (RobDangerous) tries to dump the disc with the game 7th guest (the disc is already in the redump database), but despite 2 attempts, the hashes do not match redump because the penultimate sector is reversed and DIC does not descramble it. These are the correct hashes for this disc http://redump.org/disc/73514/ Screenshots Paste command-line text.

Disc title The 7th Guest (Disc 1) (Das Spiel)

Disc ringcode

8130116V104 50419710 01 > | IFPI L003 | IFPI 01B7, IFPI 01F3 | MADE IN GERMANY BY PMDC   I, MADE IN GERMANY BY PMDC   D -- | -- | -- | --

URL Tell me the link to get the disc. Amazon: eBay: discogs:

Log file 7thGuest-2nd-try_logs.zip

[MrPepka]

It's the same in the latest version (20230909) 7thGuest-3rd-try_logs.zip

[saramibreak]

the hashes do not match redump

superg changed the data at May 13 2022, 09:26 (http://redump.org/disc/73514/changes/)

the penultimate sector is reversed and DIC does not descramble it

The last sector is incomplete as a sector (It's reversed and MSF is missing).

LBA[308325, 0x4b465]: Track[01]: Reversed sector. (Not be scrambled)
========== LBA[308325, 0x4b465]: Main Channel ==========
       +0 +1 +2 +3 +4 +5 +6 +7  +8 +9 +A +B +C +D +E +F
0000 : 00 FF FF FF FF FF FF FF  FF FF FF 00 00 00 00 02   ................
0010 : 00 00 20 00 00 00 20 00  00 00 00 00 00 00 00 00   .. ... .........
0020 : 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................

I (and F1ReB4LL, redump former admin) think that the scrambled image (.scm) and descrambled image (.img) must be matched 100%. It is important in terms of preservation.

[MrPepka]

I didn't notice that this dump had the correct hashes before. I'll talk to the superg. In that case, I'm closing the ticket

[superg]

the hashes do not match redump

superg changed the data at May 13 2022, 09:26 (http://redump.org/disc/73514/changes/)

the penultimate sector is reversed and DIC does not descramble it

The last sector is incomplete as a sector (It's reversed and MSF is missing).

LBA[308325, 0x4b465]: Track[01]: Reversed sector. (Not be scrambled)
========== LBA[308325, 0x4b465]: Main Channel ==========
       +0 +1 +2 +3 +4 +5 +6 +7  +8 +9 +A +B +C +D +E +F
0000 : 00 FF FF FF FF FF FF FF  FF FF FF 00 00 00 00 02   ................
0010 : 00 00 20 00 00 00 20 00  00 00 00 00 00 00 00 00   .. ... .........
0020 : 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................

I (and F1ReB4LL, redump former admin) think that the scrambled image (.scm) and descrambled image (.img) must be matched 100%. It is important in terms of preservation.

Yes I 100% agree that scrambled and descrambled image should match, and that's exactly what I am doing in redumper.

In this case, however, DIC unscrambles most of the sectors but not all of them and this actually breaks this guarantee. 7th Quest discs from multiple regions were giving us constant headache because at some point you changed DIC descrambling code and it already stopped matching even before redumper dumps. user7 probably remembers more as he was the one getting those regional discs dumped. I spent some time investigating this and making it compliant in redumper. After all, see that particular sector for yourself (empty), it has to be descrambled regardless of MSF being valid or not.

Just for your reference, here's my descrambling: https://github.com/superg/redumper/blob/main/cd/scrambler.ixx#L27

And in general, as we started to discuss descrambling. There are only two major cases where you might not want to descramble:

1. gaps between data / audio tracks for some systems where it shifts to audio mid gap (PC Engine if I remember correctly)
2. shitty what I call "offset shifting" masterings of various systems, mostly VCD, Philips CDi and PC. The example of that you probably saw in the discs where couple last data track sectors fail to descramble in DIC looking garbage. What happens here is that between each track, data sync offset changes due to some mastering issues. During this shift you get a few "transitional" sectors with almost garbage. That includes shift in leadin and leadout too. In redumper I analyze each sync frame and print detail statistics to detect such discs and places where shift happens. Example of (2): http://redump.org/disc/74810/ - original dump http://redump.org/disc/97024/ - fixed dump As agreed with F1ReB4LL, we currently do two dumps for such discs, one is original and another one (optional) is a fixed variant where offset shift change is compensated. In a nutshell, fixed dump is what drive would return if using BE opcode and read data track as data. I strongly suggest you spend some time on this and bring it to the common base.

[MrPepka]

I am reopening the ticket for further discussion of this issue

[saramibreak]

After all, see that particular sector for yourself (empty), it has to be descrambled regardless of MSF being valid or not. Just for your reference, here's my descrambling: https://github.com/superg/redumper/blob/main/cd/scrambler.ixx#L27

If "force_descramble" is set to "true" in redumper, it seems 308325 also descrambles (As a result, it's a scrambled sector due to the reversed sector). If it's true, the current database of "The 7th Guest (Germany)" is "fixed dump" and the previous database is original dump.

[superg]

After all, see that particular sector for yourself (empty), it has to be descrambled regardless of MSF being valid or not. Just for your reference, here's my descrambling: https://github.com/superg/redumper/blob/main/cd/scrambler.ixx#L27

If "force_descramble" is set to "true" in redumper, it seems 308325 also descrambles (As a result, it's a scrambled sector due to the reversed sector). If it's true, the current database of "The 7th Guest (Germany)" is "fixed dump" and the previous database is original dump.

Sarami, if you refer to force_descramble from: https://github.com/superg/redumper/blob/main/split.ixx#L227 It has nothing to do with this 7th Guest issue, as the flag is inactive for this disc, please don't be misled by it.

Unfortunately I don't have 7th Guest (Germany) here but I've just dumped 7th Guest (USA) with both DIC and redumper and there is the same problem - DIC doesn't descramble LBA 307725: (left is DIC, right is redumper)

[superg]

With the help of the others, I've also verified "7th Guest (Germany)" rom, sector 308325 is descrambled in the redump.org DB, everything is correct and as we want it to be in the database.

I think there is some confusion here, maybe it's the terms that we use? When I say "scrambled", I mean the left panel of the previous pic, when I say "descrambled", I mean the panel on the right.

We want those zeroes in the DB like in the right panel. We don't want to have rescrambled content in the DB.

[saramibreak]

We want those zeroes in the DB like in the right panel.

OK, but is it "fixed dump" you say? When you reverts "bin" to "scm", scm does not match comparing with the sector returned by the drive.

The 7th Guest happened to have all zero user areas. Some discs, however, are partially zero and partially scrambled data (or garbage data). What do you do in such a case?

We don't want to have rescrambled content in the DB.

At least you should leave a comment in the database that the last sector is an error.

By the way, if 7th Guest (USA) is dumped by the general ripping software like isobuster (in other words, the software that 0xbe and all flag is used), how is the sector dumped? Is it returned as an error? or Is it dumped like redumper?

[superg]

OK, but is it "fixed dump" you say?

No it's not fixed dump. Fixed dump is currently used when you correct offset shift thing that I've explained earlier.

When you reverts "bin" to "scm", scm does not match comparing with the sector returned by the drive.

Yes you're correct. But this bin->scm conversion thing that we often bring up is a non-functional concept. For instance here: https://github.com/saramibreak/DiscImageCreator/blob/master/DiscImageCreator/output.cpp#L1736 If I understand everything correctly, you make a lot of assumptions based on a sector content like MSF, padding, sector mode, etc. It's already non reversible for some branching here e.g. you will not get the same scm by going scm->bin->scm. Did you test bin->scm for all the corner cases?

The 7th Guest happened to have all zero user areas. Some discs, however, are partially zero and partially scrambled data (or garbage data). What do you do in such a case?

I think the reasonable option is to try to get it clean, descramble only what is scrambled. Philips CDi has a lot of discs with huge areas of unscrambled portions, rescrambling them for preservation just doesn't make any sense to me.

At least you should leave a comment in the database that the last sector is an error.

Yeah this can be done but there are a lot of discs like this. Comments are already overloaded with all sorts of info.

By the way, if 7th Guest (USA) is dumped by the general ripping software like isobuster (in other words, the software that 0xbe and all flag is used), how is the sector dumped? Is it returned as an error? or Is it dumped like redumper?

IsoBuster detects an error in these sectors and presents you a dialog of what to choose there. I didn't even find an option to keep the original content as if you choose "Replace with Erroneous sector" it doesn't preserve scrambled state but just fills the sector with 0xAA value. All the other options present are just variants of filling your sector with zeroes. IsoBuster is totally broken with this.

Fundamentally, there are two options:

1. analyze by sector content and make a decision whether to unscramble the sector or not
2. always unscramble if sector has sync header

We're at (1) in redump and IMO this is reasonable. On the other hand, (2) will give us full bin->scm conversion but a lot of "unclean" dumps in the database.

[saramibreak]

I think the reasonable option is to try to get it clean, descramble only what is scrambled. Philips CDi has a lot of discs with huge areas of unscrambled portions, rescrambling them for preservation just doesn't make any sense to me.

I read again the previous discussion (http://forum.redump.org/topic/48206/offset-shifting-discs/), but there was no mention of this issue. After all, I agree the reversed sector is not scrambled if you understand it's error sector.

Yeah this can be done but there are a lot of discs like this. Comments are already overloaded with all sorts of info.

Then you should use the "Errors count".

1. analyze by sector content and make a decision whether to unscramble the sector or not We're at (1) in redump and IMO this is reasonable.

You have previously stated (http://forum.redump.org/post/107333/#p107333). That's a contradiction. Or change your mind?

I think we should not rely on sector content (or rely less) when deciding whether we should unscramble sector or not. It's impossible to come up with good decision algo if sector is partially damaged as pretty much any byte can be damaged and this shift issue here clearly demonstrate that.

2. always unscramble if sector has sync header

We already mentioned it (http://forum.redump.org/post/107337/#p107337). Additionally, we should check msf and sector mode (redumper are already doing it?).

[superg]

I read again the previous discussion (http://forum.redump.org/topic/48206/offset-shifting-discs/), but there was no mention of this issue. After all, I agree the reversed sector is not scrambled if you understand it's error sector.

Offset shifting is not directly related to this. Once you compensate for offset shift, sync is at the right place and most of the sectors descramble just fine. The transitional sectors are the exception but mostly they should be descrambled, that's actually the situation where force_descramble is becoming true in my code if we generate fixed dump (option has to be used, this is not by default)

Yeah this can be done but there are a lot of discs like this. Comments are already overloaded with all sorts of info.

Then you should use the "Errors count".

Data track errors count should strictly show the output of edccchk.exe, this is how it has been historically and any deviation from that meaning brings a lot of confusion to our dumpers. We had countless discussions on this and in order to keep things consistant, it should remain as it is (edccchk).

1. analyze by sector content and make a decision whether to unscramble the sector or not We're at (1) in redump and IMO this is reasonable.

You have previously stated (http://forum.redump.org/post/107333/#p107333). That's a contradiction. Or change your mind?

I don't really see where I contradict this in the link you provided. It's not the point of the discussion.

I think we should not rely on sector content (or rely less) when deciding whether we should unscramble sector or not. It's impossible to come up with good decision algo if sector is partially damaged as pretty much any byte can be damaged and this shift issue here clearly demonstrate that.

2. always unscramble if sector has sync header

We already mentioned it (http://forum.redump.org/post/107337/#p107337). Additionally, we should check msf and sector mode (redumper are already doing it?).

Yes, we already discussed it a year ago or so. I'm not saying we should do (2), I just list available options. Redumper currently analyzes MSF and sector mode, but not exactly the way you do it in DIC.

By the way, I have unit tests for my descrambling so it never breaks, you might want to check that out. https://github.com/superg/redumper/blob/main/tests/tests.cc#L189 and set of test sectors that I made: https://github.com/superg/redumper/tree/main/tests/unscramble

Right now, the important thing is that you don't rescramble that 7th Quest sector back.

[saramibreak]

@MrPepka Try it plz. DiscImageCreator_test.zip

[MrPepka]

Today I just received the logs from this new version, I confirm that the problem has been solved