superg / redumper

Low level CD dumper utility
GNU General Public License v3.0
179 stars 16 forks source link

CD-Rs with Errors in the Last 2 Sectors #13

Open 7Seventy7 opened 1 year ago

7Seventy7 commented 1 year ago

Many CD-Rs (betas, press asset discs etc) consistently have Errors in the Last 2 Sectors.

To be specific, these errors are NOT caused by scratches and when dumping with discimagecreator without the /c2 option, you consistently get the same hashes.

The current method to dat such discs in redump with dic is to dump twice and compare hashes. Here's an article about it on the Redump wiki: http://wiki.redump.org/index.php?title=Dumping_CD-Rs_with_Errors_in_the_Last_2_Sectors

Could you do something like this? If CD-R & Errors in Last 2 Sectors, Then (do whatever should be done).

ehw commented 1 year ago

Wanted to leave a comment on this and give my thoughts since I deal with this too, since we work with a lot of media on recordable discs.

I think the issue is that regardless whether or not the errors exist consistently, they're still C2 errors which indicate a physical media problem. There will be scenarios where a recorder (or maybe a fault with the media itself) will unintentionally leave mastering mistakes on the physical media itself. From my experience, not every drive model will report the same C2 error, some can read a bit more of the damaged sector than others even if none can totally read the sector fully. So if one drive can read one of the last two sectors differently, the question comes down to - which dump do you choose if you have multiple dumps created from different drives?

As far as redump.org submissions go, I think it will ultimately always be a manual decision that must be made no matter the solution. But, here are some things to consider:

1.) Do the last two sectors exist within the discs TOC? If they don't, then they don't affect any data that's part of any area meant to be accessed under normal use, theoretically. Therefore it doesn't really matter if errors exist or not. 2.) Do the last two sectors affect a data track or an audio track? Data tracks, depending if ECC/EDC is utilized, can be verified and even corrected. For audio tracks, this is trickier as C2 status bits are your only indicator of an issue that's exposed to the user (software). 3.) Can the last two sectors be refined with multiple drives? This is something I'm going to experiment with soon. As I said before, since every drive can read/interpret these kind of errors slightly differently, it might be possible to create a refined dump that best reflects what the original media actually has for those two sectors by combining the results of multiple dumps with various drives. 4.) If the sectors belong to a data track with EDC/ECC data, does the checksum reported in the header suggest a sector with a particular data pattern that can be assumed (like a sector that is all 00'd)? If the C2 status bits returned by the drive confirm that no bits affect the region that contains the checksums, then you could use this to at least determine what the data might possibly be in certain scenarios. However, this might go into "restoration" territory, and not preservation, as such data can always be manipulated or not indicative of the actual media itself.

Just something to think about as maybe this a problem that can be solved.

Tatsh commented 1 year ago

The guide on redump just says to verify this error is consistent but otherwise ignore it.

I've had many CD-Rs with the second to last sector giving c2 error but they still pass the analysis phase without passing --force-split.

7Seventy7 commented 1 year ago

The guide on redump just says to verify this error is consistent but otherwise ignore it.

I've had many CD-Rs with the second to last sector giving c2 error but they still pass the analysis phase without passing --force-split.

Ha I wrote the guide so I'm aware :)

The problem is - while one drive may give consistent hashes for discs with errors in the last two sectors, a different drive model will give a different set of consistent hashes.

These last 2 sectors should be handled like this: if cd-r and errors in last 2 sectors = zero out sectors and generate a bin.

The problem is many people are not using dic or redumper because most of their uber rare betas are not leaving them with a .bin file (just a scram / scm file). I know of a at least a thousand PS1/PS2 betas which are currently suffering an IsoBuster fate for this reason xD

Whovian9369 commented 11 months ago

Hey, just ran into this myself. http://redump.org/disc/88150/ is the current expected dump info

Some snippets from DIC logs:

LBA[029847, 0x07497]: Track[01]: Invalid mode. 
LBA[029848, 0x07498]: Track[01]: Invalid mode. Invalid reserved byte. Skip descrambling
LBA[029847, 0x07497]: P[00], Q[4101010637720006397202c5]{ Data,      Copy NG,                  Track[01], Idx[01], RMSF[06:37:72], AMSF[06:39:72]}, RtoW[0, 0, 0, 0]
LBA[029848, 0x07498]: P[00], Q[41010106377300063973b8b5]{ Data,      Copy NG,                  Track[01], Idx[01], RMSF[06:37:73], AMSF[06:39:73]}, RtoW[0, 0, 0, 0]

Full Redumper Log:

=== 2023-08-03 16:47:13 ========================================================
redumper v2023.07.19 build_195 [Jul 19 2023, 04:48:49]

command line: C:\Users\Whovian\Downloads\Games\Tools\MPF_net48-2.6.1-1184\Programs\Redumper\redumper.exe cd --drive=D --speed=24 --retries=20 "--image-path=C:\Users\Whovian\Downloads\Games\Tools\MPF_net48-2.6.1-1184\ISO\Gamecube - CD-R - MUSYX_2003_01_09 [Redumper]" --image-name=MUSYX_2003_01_09

drive path: D
drive: PLEXTOR - CD-R PX-W4824A (revision level: 1.07, vendor specific: 03/24/06 14:00)
drive configuration: GENERIC (read offset: +98, C2 shift: 294, pre-gap start: -75, read method: D8, sector order: DATA_C2_SUB)

image path: C:\Users\Whovian\Downloads\Games\Tools\MPF_net48-2.6.1-1184\ISO\Gamecube - CD-R - MUSYX_2003_01_09 [Redumper]
image name: MUSYX_2003_01_09

*** DUMP

disc TOC:
  track 1 {  data }
    index 01 { LBA:      0, MSF: 00:02:00 }
  track A {  data }
    index 01 { LBA:  29849, MSF: 06:39:74 }

[LBA:  29849] subcode desync (shift: -1)                    
[LBA:  29857] subcode desync (shift: +0)                     

media errors: 
  SCSI: 0
  C2: 2
  Q: 18

*** PROTECTION (time check: 74s)

protection: N/A

*** REFINE

[LBA:  29849] subcode desync (shift: -1)                    

media errors: 
  SCSI: 0
  C2: 1
  Q: 9

*** SPLIT (time check: 61s)

correcting Q... done

final TOC:
  track 1 {  data }
    index 00 { LBA: [  -150 ..     -1], length:    150, MSF: 00:00:00-00:01:74 }
    index 01 { LBA: [     0 ..  29848], length:  29849, MSF: 00:02:00-06:39:73 }
  track A {  data }
    index 01 { LBA: [ 29849 ..  36596], length:   6748, MSF: 06:39:74-08:09:71 }

warning: TOC / QTOC mismatch, track index 00 (track: 1, LBA: -150 <=> 29778)
warning: TOC / QTOC mismatch, track index 01 (track: 1, LBA: 0 <=> 29853)
warning: TOC / QTOC mismatch, track length (track: 1, LBA: 29849 <=> 29857)
warning: TOC / QTOC mismatch, track index 00 (track: A, LBA: 29849 <=> 29857)
warning: TOC / QTOC mismatch, track index 01 (track: A, LBA: 29849 <=> 29857)

final QTOC:
  session 1
    track 1 {  data }
      index 00 { LBA: [  -150 ..     -1], length:    150, MSF: 00:00:00-00:01:74 }
      index 01 { LBA: [     0 ..  29848], length:  29849, MSF: 00:02:00-06:39:73 }
    track A {  data }
      index 01 { LBA: [ 29849 ..  29849], length:      1, MSF: 06:39:74-06:39:74 }
  session 2
    track 1 {  data }
      index 00 { LBA: [ 29775 ..  29849], length:     75, MSF: 06:39:00-06:39:74 }
      index 01 { LBA: [ 29850 ..  29850], length:      1, MSF: 06:40:00-06:40:00 }
    track A {  data }
      index 01 { LBA: [ 29851 ..  29852], length:      2, MSF: 06:40:01-06:40:02 }
  session 3
    track 1 {  data }
      index 00 { LBA: [ 29778 ..  29852], length:     75, MSF: 06:39:03-06:40:02 }
      index 01 { LBA: [ 29853 ..  29856], length:      4, MSF: 06:40:03-06:40:06 }
    track A {  data }
      index 01 { LBA: [ 29857 ..  36596], length:   6740, MSF: 06:40:07-08:09:71 }

analyzing... done (time: 10s)

data disc detected, track offset statistics: 
  LBA: [   -75 ..  29848], count:  29924, sample offset:    -44130
  LBA: [ 29850 ..  29856], count:      7, sample offset: +17551764
  LBA: [ 29857 ..  36597], count:   6741, sample offset: +17555886

non-zero  TOC sample range: [   -88200 .. +17551212]
non-zero data sample range: [   -44130 .. +17555886]
warning: non-zero data range is not continuous, skipping Universal Hash calculation

warning: offset shift detected, to apply correction please use an option
disc write offset: -30

warning: incomplete pre-gap (session: 1, unavailable: 75/150)
warning: lead-in contains non-zero data (session: 1, sectors: 75/76)
warning: lead-out contains non-zero data (session: 1, sectors: 8/6748)
warning: lead-in starts with unavailable sector (session: 1)
checking tracks
errors detected, track: 1, sectors: {SKIP: 0, C2: 1}, samples: {SKIP: 0, C2: 339}
errors detected, track: A, sectors: {SKIP: 0, C2: 1}, samples: {SKIP: 0, C2: 93}
error: data errors detected, unable to continue
DopefishJustin commented 3 days ago

By collecting doujin games from Japan I have built up a whole stack of these discs now. The official word from Redump staff is to run redumper with --force-split --leave-unchanged and get two matching dumps. But probably 90% of the time, consecutive dumps do not match. Looking at the logs I can see the drive cycling through the same two or three different reads for each sector, so it's not like the results are totally random:

dump 1.log dump 2.log

[LBA: 114821] C2 error (bits: 1632, data crc: 96F36717, C2 crc: 39FBB6CC, retry: 75)
[LBA: 114821] C2 error (bits: 1632, data crc: 96F36717, C2 crc: 39FBB6CC, retry: 76)
[LBA: 114821] C2 error (bits: 1632, data crc: 96F36717, C2 crc: 39FBB6CC, retry: 77)
[LBA: 114821] C2 error (bits: 1608, data crc: 10A1A05B, C2 crc: 65C5EF5F, retry: 78)
[LBA: 114821] C2 error (bits: 1632, data crc: 96F36717, C2 crc: 39FBB6CC, retry: 79)
[LBA: 114821] C2 error (bits: 1632, data crc: 96F36717, C2 crc: 39FBB6CC, retry: 80)
[LBA: 114821] C2 error (bits: 1632, data crc: 96F36717, C2 crc: 39FBB6CC, retry: 81)
[LBA: 114821] C2 error (bits: 1632, data crc: 96F36717, C2 crc: 39FBB6CC, retry: 82)
[LBA: 114821] C2 error (bits: 1632, data crc: 96F36717, C2 crc: 39FBB6CC, retry: 83)
[LBA: 114821] C2 error (bits: 1632, data crc: 96F36717, C2 crc: 39FBB6CC, retry: 84)
[LBA: 114821] C2 error (bits: 1632, data crc: 96F36717, C2 crc: 39FBB6CC, retry: 85)
[LBA: 114821] C2 error (bits: 1632, data crc: 96F36717, C2 crc: 39FBB6CC, retry: 86)
[LBA: 114821] C2 error (bits: 1632, data crc: 96F36717, C2 crc: 39FBB6CC, retry: 87)
[LBA: 114821] C2 error (bits: 1632, data crc: 96F36717, C2 crc: 39FBB6CC, retry: 88)
[LBA: 114821] C2 error (bits: 1632, data crc: 96F36717, C2 crc: 39FBB6CC, retry: 89)
[LBA: 114821] C2 error (bits: 1632, data crc: 96F36717, C2 crc: 39FBB6CC, retry: 90)
[LBA: 114821] C2 error (bits: 1632, data crc: 96F36717, C2 crc: 39FBB6CC, retry: 91)
[LBA: 114821] C2 error (bits: 1632, data crc: 96F36717, C2 crc: 39FBB6CC, retry: 92)
[LBA: 114821] C2 error (bits: 1608, data crc: 10A1A05B, C2 crc: 65C5EF5F, retry: 93)
[LBA: 114821] C2 error (bits: 1632, data crc: 96F36717, C2 crc: 39FBB6CC, retry: 94)
[LBA: 114821] C2 error (bits: 1632, data crc: 96F36717, C2 crc: 39FBB6CC, retry: 95)
[LBA: 114821] C2 error (bits: 1608, data crc: CD34B2C3, C2 crc: 65C5EF5F, retry: 96)
[LBA: 114821] C2 error (bits: 1632, data crc: 96F36717, C2 crc: 39FBB6CC, retry: 97)
[LBA: 114821] C2 error (bits: 1632, data crc: 96F36717, C2 crc: 39FBB6CC, retry: 98)
[LBA: 114821] C2 error (bits: 1608, data crc: 10A1A05B, C2 crc: 65C5EF5F, retry: 99)
[LBA: 114821] C2 error (bits: 1632, data crc: 96F36717, C2 crc: 39FBB6CC, retry: 100)
[LBA: 114821] correction failure          

There has to be some way of making this process more deterministic to avoid wasting time dumping entire discs over and over trying to get a match through dumb luck. For example by always choosing the read with the lowest number of error bits, or the read which is returned the majority of the time.

Personally I would be in favour of dropping --leave-unchanged and just using 0x55 since the data is meaningless anyway but that would be a decision for Redump staff.

Ideally, redumper should detect and handle the situation automatically without needing extra parameters or human intervention (similarly to detecting SafeDisc etc.)

superg commented 3 days ago

As it was discussed numerous times before, this data is unreliable and you're correct, same values are luck based. These should be filled with default value, being it 0x55 or 0x00. It will be handled gracefully by redumper, just need time to implement.