Closed ehw closed 1 year ago
saramibreak
commented
1 year ago Original thread of redump.org http://forum.redump.org/topic/3808/raw-dvd-dumping-discussion/ http://forum.redump.org/topic/8108/friidump-053-source-code-update-linux-build/
I researched Friidump and tried to implement the raw dumping in the past. https://github.com/saramibreak/DiscImageCreator/blob/master/DiscImageCreator/execScsiCmdforDVD.cpp#L713 (This code probably do not work as we expected.)
ehw
commented
1 year ago I do remember seeing this in the code when we were initially doing our own research a while back but we don't understand why it doesn't work either.
Space: 71517175808 bytes
=> There is enough disk space for dumpingRawdump command [0]:0x3c [1]:0x02 [2]:0000 LBA[000000, 0000000]: [F:ReadDVDRaw][L:874] Opcode: 0xa8 ScsiStatus: 0x02 = CHECK_CONDITION SenseData Key-Asc-Ascq: 03-11-05 = MEDIUM_ERROR - L-EC UNCORRECTABLE ERROR LBA[000016, 0x00010]: [F:ReadDVDRaw][L:874] Opcode: 0xa8 ScsiStatus: 0x02 = CHECK_CONDITION SenseData Key-Asc-Ascq: 03-11-05 = MEDIUM_ERROR - L-EC UNCORRECTABLE ERROR LBA[000032, 0x00020]: [F:ReadDVDRaw][L:874] Opcode: 0xa8 ScsiStatus: 0x02 = CHECK_CONDITION SenseData Key-Asc-Ascq: 03-11-05 = MEDIUM_ERROR - L-EC UNCORRECTABLE ERROR LBA[000048, 0x00030]: [F:ReadDVDRaw][L:874] Opcode: 0xa8 ScsiStatus: 0x02 = CHECK_CONDITION SenseData Key-Asc-Ascq: 03-11-05 = MEDIUM_ERROR - L-EC UNCORRECTABLE ERROR LBA[000064, 0x00040]: [F:ReadDVDRaw][L:874] Opcode: 0xa8 ScsiStatus: 0x02 = CHECK_CONDITION SenseData Key-Asc-Ascq: 03-11-05 = MEDIUM_ERROR - L-EC UNCORRECTABLE ERROR LBA[000080, 0x00050]: [F:ReadDVDRaw][L:874] Opcode: 0xa8 ScsiStatus: 0x02 = CHECK_CONDITION SenseData Key-Asc-Ascq: 03-11-05 = MEDIUM_ERROR - L-EC UNCORRECTABLE ERROR LBA[000096, 0x00060]: [F:ReadDVDRaw][L:874] Opcode: 0xa8 ScsiStatus: 0x02 = CHECK_CONDITION SenseData Key-Asc-Ascq: 03-11-05 = MEDIUM_ERROR - L-EC UNCORRECTABLE ERROR LBA[000112, 0x00070]: [F:ReadDVDRaw][L:874] Opcode: 0xa8 ScsiStatus: 0x02 = CHECK_CONDITION SenseData Key-Asc-Ascq: 03-11-05 = MEDIUM_ERROR - L-EC UNCORRECTABLE ERROR LBA[000128, 0x00080]: [F:ReadDVDRaw][L:874] Opcode: 0xa8 ScsiStatus: 0x02 = CHECK_CONDITION SenseData Key-Asc-Ascq: 03-11-05 = MEDIUM_ERROR - L-EC UNCORRECTABLE ERROR
It does use opcode 3C it seems but it doesn't work...
The .raw is completely blank, but the DMI and PFI were dumped from the disc correctly.
Friidump works fine, however.
H:\Tools\rawdump\friidump>friidump.exe friidump.exe -d I: -T 1 -a -s FriiDump 0.5.3 - Copyright (C) 2007 Arep This software comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it under certain conditions; see COPYING for details.
Official support forum: http://wii.console-tribe.com
Forum for this UNOFFICIAL VERSION: http://forum.redump.org
WARNING: Extra parameters ignored Initializing DVD drive... OK
Drive model........: PLEXTOR/DVDR PX-760A/1.07 Supported..........: Yes Command............: 0 Method.............: 2 Requested sectors..: 16 Expected sectors...: 16
Press Ctrl+C at any time to terminate
Retrieving disc seeds, this might take a while... OK
Disc type..........: Wii (forced) Disc size..........: 2294912 Game ID............: SR Region.............: USA/NTSC Maker..............: 8P - Sega Japan Version............: 1.00 Game title.........: SONIC AND THE SECRET RINGS Contains update....: No
Writing to file "SONIC AND THE SECRET RINGS.iso" in ISO format
28% |---------*-----------------------| 37.81 MB/h, ETA: 26/04/2023 13:01:41
saramibreak
commented
1 year ago I tried the same option (-T 1 -a -s) It's extremely slow. The disc continues to spin up and down.
E:\MyApp>friidump.exe -d g: -T 1 -a -s
FriiDump 0.5.3 - Copyright (C) 2007 Arep
This software comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions; see COPYING for details.
Official support forum: http://wii.console-tribe.com
Forum for this UNOFFICIAL VERSION: http://forum.redump.org
Initializing DVD drive... OK
Drive information:
----------------------------------------------------------------------
Drive model........: PLEXTOR/DVDR PX-755A/1.08
Supported..........: Yes
Command............: 0
Method.............: 2
Requested sectors..: 16
Expected sectors...: 16
Press Ctrl+C at any time to terminate
Retrieving disc seeds, this might take a while... OK
Disc information:
----------------------------------------------------------------------
Disc type..........: Wii (forced)
Disc size..........: 2294912
Game ID............: ML
Region.............: Japan/NTSC
Maker..............: H4 - SNK Playmore
Version............: 1.00
Game title.........: METAL SLUG COMPLETE
Contains update....: Yes
Writing to file "METAL SLUG COMPLETE.iso" in ISO format
0% |*--------------------------------| 31.10 MB/h, ETA: 29/04/2023 19:58:259Btw, I confirmed that raw dumping works by DVD using PLEXTOR/DVDR PX-755A/1.08.
ehw
commented
1 year ago Yep, that's technically normal for Friidump and Plextors, its very slow. The drive constantly revs up and down, even when the program isn't running, however this only seems to occur when you're closer to the inner part of the disc on a Wii game. The further you go, the quieter the drive becomes. I'm at 36% right now on a Wii game and it's dead silent, drive is slightly cool to the touch too. But if you can stomach the slow speed and let it alone, it will dump your disc and match Redump. :)
I don't really know if there's a way to speed it up. There hasn't been a lot of research on this method and Plextors it seems. The method (not just the command) Friidump uses for Plextors dumps 16 sectors at a time. There's another method used for other drives where it can dump 5 groups of 16 sector blocks from the buffer (method 3?) that might work but wasn't implemented for Plextors. Depending on the size of the buffer on the drive, there might be a smaller/larger 16 sector block size limit that you could increase to in order to speed things up. But you'd have to look into it as I have no idea what might be possible...
Regardless, I think implementing this in DIC would still be useful. As I said, only Plextors can read RVT-R and NR discs with this method. It might not be preferable, but giving the alternatives we have, this is a legitimate alternative if one does not have thousands of dollars to spend.
Let me know if you need any testing.
(BTW, can Xbox discs work with READ BUFFER? I wonder about that, but not sure if this method can handle the Security Sectors...)
saramibreak
commented
1 year ago As I said, only Plextors can read RVT-R and NR discs with this method.
Hitachi-LG GDR drive can't read them? Redump wiki says that they can be read by the swap disc method. http://wiki.redump.org/index.php?title=Nintendo_GameCube_and_Wii_Dumping_Guide_for_PC
BTW, can Xbox discs work with READ BUFFER?
According to the docs/NEWS of Friidump, drives that can read the Xbox discs support it.
4. Toshiba Samsung SH-D162A, SH-D162B, SH-D162C, SH-D162D
Returns 2384 data bytes per sector like Lite-On does. Appears to support
streamed reading but performance with tested model (SH-D162D) was somewhat low
and unstable even with ordinary DVDs. Looks promising, if only good-working
method could be determined. Latest drives added, definitely need more testing
at this point.this method can handle the Security Sectors...
Does it means SS.bin or Security Sectors themselves? SS.bin is a kind of PFI. READ12 can't read PFI, then, I think READ BUFFER also can't generate SS.bin. Security Sectors themselves can be read partially using READ12, but they are not preservation target for redump.org.
ehw
commented
1 year ago GDR drives can with swapping but its not stable/reliable from experience. For that to work, your only option is to use rawdump but since it wasn't designed for dev discs, you have to time the swap just right so you can trick the drive into reading the disc. A lot of people have tried this method and its just not worth it. The GDR drives wont recognize the RVT-Rs or NR discs at all, which is strange because retail works fine. This is because dev discs contain an invalid DMI/PFI that most drives get confused with (except for RVT-R writers). You can't dump RVT-Rs or NR discs in retail Wii consoles either, as the drive firmware itself prevents them from being recognized. We tried over 100 drives as part of our GD-R project, nothing can be done. This method with Plextors, despite being really slow, is the only inexpensive and accessible way.
I did notice that Friidumps source code does mention quite a bit of drive models for certain methods. We did try a few at least for RVT-Rs but couldn't get anywhere. Didn't try retail, but if you have support for some of these or if not we could test DIC with retail. :)
That's interesting about Xbox discs. Is SS.bin required for redump? It'd be interesting if the drives capable of raw dumping can read an entire Xbox disc, that's what I was curious the most of since you can only read the DVD Video partition on the disc traditionally...
(btw, I have source code to rawdump if you ever want to look at it. It's decompiled, but its almost completely labeled and can be compiled. :))
saramibreak
commented
1 year ago DiscImageCreator_test.zip Error msg still exists but it seems that sectors are cached and can get these by READ_DATA_BUFF Reading speed is the same as Friidump.
I have source code to rawdump if you ever want to look at it. It's decompiled, but its almost completely labeled and can be compiled
I can't read assembly language. Can you convert it to c/c++ language?
ehw
commented
1 year ago Error msg still exists but it seems that sectors are cached and can get these by READ_DATA_BUFF
Reading speed is the same as Friidump.
Awesome! I'll take a look when I get home. I have another Wii dump that just finished with Friidump and it matched the entry. :)
I have source code to rawdump if you ever want to look at it. It's decompiled, but its almost completely labeled and can be compiled
I can't read assembly language. Can you convert it to c/c++ language?
It's actually not assembly, its in C#. I'll upload it when I get home. Note, I only decompiled rawdump.exe completely but I decompiled the more important functions from the .dll that issue commands to the drive. :)
saramibreak
commented
1 year ago It's actually not assembly, its in C#. I'll upload it when I get home. Note, I only decompiled rawdump.exe completely but I decompiled the more important functions from the .dll that issue commands to the drive. :)
I tried ILSpy, but it's not decompiled as I expected.
ehw
commented
1 year ago It's actually not assembly, its in C#. I'll upload it when I get home. Note, I only decompiled rawdump.exe completely but I decompiled the more important functions from the .dll that issue commands to the drive. :)
I tried ILSpy, but it's not decompiled as I expected.
Here you go. The .c files are individual functions found in the .dll file. They're externally called by the .exe but the .exe does some things with the drives and data too. The main exe program is completely decompiled and can be recompiled and works, we just decompiled the more important functions in the .dll.
One interesting thing I saw was that in the READ12 command call the buffer size is 16 on rawdump but 10 in Friidump. Also, the decompiler uses ints for things that are likely hex, so you may need to convert some of the numbers to make the code readable.
ehw
commented
1 year ago Hey sarami, I tested your test version. Haven't tested NR/RVT-Rs yet but Wii/GC retail works fine after an hour or so each. :)
I looked at the .raw file and there's data in there now. I even tested /re to resume the dump and it works as well.
The Medium_Error messages are annoying but it's okay because that is technically what the drive is reporting, and I believe it will go away with time but unsure. The only thing I'm worried about is that hopefully the presence of these errors doesn't deter dumps made with this method from being accepted, since the data itself is fine, the drive is just acting crazy when its trying to read the disc. Not sure if there might be something to help remedy that. Does DIC utilize the ECC/EDC data it gets from the raw reading method at least for checking the dump? Does the READ BUFFER command return both EDC/ECC or just EDC?
I guess the penultimate question is whether or not the Plextor method can be made faster or not. Friidump does have other methods that are nonstreaming that might be able to dump more blocks of 16 sectors at a time, but I'm not sure if this can be implemented or not...
It does work though so feel free to merge unless you can think of some improvements. Let me know and I can test too. :)
EDIT: On a completely unrelated note, I do kinda like Friidump's status bar with transfer rate information. Is there a way DIC might have something similar?
saramibreak
commented
1 year ago The only thing I'm worried about is that hopefully the presence of these errors doesn't deter dumps made with this method from being accepted, since the data itself is fine, the drive is just acting crazy when its trying to read the disc.
Do not output the err msg. DiscImageCreator_test.zip
Does DIC utilize the ECC/EDC data it gets from the raw reading method at least for checking the dump?
It's only supported by CD (EccEdc.exe). I've not confirmed that ECC/EDC of DVD etc is the same as CD.
I guess the penultimate question
I tried it but sectors that is beyond 16 sectors are not cached.
On a completely unrelated note
I'll not implement it.
ehw
commented
1 year ago Thanks! It works fine now. Data is being written to the .raw file still and /re works too, and nothing seems to be written to the error log files either. The only thing is that volDesc.txt and mainInfo.txt are blank so far, where I think normally they get generated before dumping starts.
Is this method only returning 2064 bytes per sector? Or is it returning 2064 + ECC? I'm not familiar at all with the structure you don't normally see...
ECC/EDC would be interesting to see if it could be utilized for the EccEdc.txt log for DVDs, but I can't find much documentation on it...
That's too bad about not being able to read more than 16 sectors at a time. You can't find a way to tell the drive to dump more sequential blocks of 16 sectors? I think this would need streamed support to work though, which I suppose is out. :(
I think what I'll do before we close this out is see a GameCube dump all the way to the end and see what DIC does. Hopefully the image descrambles to something that matches. Then we know we're good. :)
saramibreak
commented
1 year ago Is this method only returning 2064 bytes per sector? Or is it returning 2064 + ECC? I'm not familiar at all with the structure you don't normally see...
According to the docs/NEWS of Friidump, Plextor and Hitachi-LG return 2064 bytes (header +main), while Lite-On and TSST return 2384 bytes (header +main + ECC).
1. Hitachi-LG GDR8161B, GDR8162B, GDR8163B, GDR8164B, GDR8082N
Those drives can read GC/Wii media without swapping. Expected performance is
1600..1900 MB/h for *4B, *3B and 2100..2600 MB/h for *2B, *1B. Custom memory
dump command is used, which returns 2064 bytes of data. It was reproted that
they can not read other (e.g. PC) discs this way though, this needs
confirmation.
2. Lite-On LH-18A1H, DVDRW LH-18A1P, DVDRW LH-20A1H, DVDRW LH-20A1P
Reading performance for PC DVDs can go up to 5000 MB/h, which means program's
core as well as new methods are capable to output data at least at this rate.
Reading performance for GC was about 1600..1700 MB/h so likely this slowdown is
caused by drive logic itself. Though I only had one GC game to test with, so
possibly better results can be achieved depending on media. Best results were
obtained, when using method 5 with parameter 16,27 (--method5=16,27). This
combination isn't set as default because it can cause noticable delays
depending on medium quality and to make methods more general for use with other
devices. Lite-On won't read GC/Wii DVDs at all without swapping. Lite-On
returns 2384 bytes of data (2064 + ECC) by means of vendor specific READ BUFFER
command. Tested with models LH-18A1H, LH-18A1P and LH-20A1H.
3. Plextor
Plextor would return 2064 bytes of already unscrambled data with READ BUFFER
command. It works good with ordinary DVDs but due the lack of streamed reading
support is practically useless for GC/Wii dumping because of very low
performance. Works nevertheless and could be used for some experiments and
testing. Results from PX-760A.
4. Toshiba Samsung SH-D162A, SH-D162B, SH-D162C, SH-D162D
Returns 2384 data bytes per sector like Lite-On does. Appears to support
streamed reading but performance with tested model (SH-D162D) was somewhat low
and unstable even with ordinary DVDs. Looks promising, if only good-working
method could be determined. Latest drives added, definitely need more testing
at this point.ECC/EDC would be interesting to see if it could be utilized for the EccEdc.txt log for DVDs, but I can't find much documentation on it...
Nintendo discs are scrambled. When are unscrambled, EDC is checked by unscrambler.
ehw
commented
1 year ago Works for me although I might not be able to test it for normal DVDs.
I'll try and test some RVTs/NRs when GameCube is done which should be another day or so. If you have any more changes that need testing just post them here. :)
ehw
commented
1 year ago Hey @saramibreak, got a bug.
I was able to dump all the raw sectors with the Plextor but when it came to checking the sums I got this error:
Space: 93515280384 bytes
=> There is enough disk space for dumpingRawdump command [0]:0x3c [1]:0x02 [2]:0000 Start resuming -> Last sector num: de0b0 Checking sector num: de0a0
[F:CalcAndGetHash][L:465] GetLastError: 2, The system cannot find the file specified.
EndTime: 2023-04-28T19:10:11-0400
I have a ,raw but no "Sonic Gems Collection.bin".
saramibreak
commented
1 year ago valid extension was omitted. -> set .bin
ehw
commented
1 year ago valid extension was omitted. -> set .bin
- fixed: set .iso when dvd or xbox or bd is used
Still doesn't seem to work for me...dump is finished but wont get past CalcAndGetHash.
Space: 92064595968 bytes
=> There is enough disk space for dumpingRawdump command [0]:0x3c [1]:0x02 [2]:0000 Start resuming -> Last sector num: de0b0 Checking sector num: de0a0
[F:CalcAndGetHash][L:465] GetLastError: 2, The system cannot find the file specified.
EndTime: 2023-04-29T09:42:10-0400
Here are my files if you want to take a look:
https://www.mediafire.com/file/s4p2w55ras1wazf/test.7z/file
I also wanted to try an Xbox disc. I got the same error as well, and DIC doesn't seem to want to go further than the DVD Video section on the disc.
Space: 92033859584 bytes
=> There is enough disk space for dumpingRawdump command [0]:0x3c [1]:0x02 [2]:0000 Creating raw(LBA) 6832/ 6992 Layer is changed: 20 -> 01 Creating raw(LBA) 6992/ 6992 [F:CalcAndGetHash][L:465] GetLastError: 2, The system cannot find the file specified.
EndTime: 2023-04-29T10:58:15-0400
saramibreak
commented
1 year ago ehw
commented
1 year ago
- fixed: non-0xe7 drives also can execute unscrambler.exe
GameCube works now. :)
Space: 93460123648 bytes
=> There is enough disk space for dumpingRawdump command [0]:0x3c [1]:0x02 [2]:0000 Start resuming -> Last sector num: de0b0 Checking sector num: de0a0
GOD/WOD unscrambler 0.4.1 (xt5@ingenieria-inversa.cl)
This program is distributed under GPL license, see the LICENSE file for more info.
caching seed 0100 caching seed 0c04 caching seed 013c caching seed 060a caching seed 021e caching seed 0314 caching seed 040f caching seed 01a8 caching seed 0f81 caching seed 4049 caching seed 13c2 caching seed 60ac caching seed 21e4 caching seed 3146 caching seed 40f8 caching seed 1a83 caching seed 00f8 image successfully unscrambled. time elapsed: 9.00 seconds. ret = 0 Hashing: Sonic Gems Collection.iso EndTime: 2023-04-29T13:26:50-0400
Xbox still doesn't want to descramble though, even if it only dumped the DVD Video partition. This might also affect normal DVDs too.
Space: 93461745664 bytes
=> There is enough disk space for dumpingRawdump command [0]:0x3c [1]:0x02 [2]:0000 Creating raw(LBA) 6832/ 6992 Layer is changed: 20 -> 01 Creating raw(LBA) 6992/ 6992 [F:CalcAndGetHash][L:465] GetLastError: 2, The system cannot find the file specified.
EndTime: 2023-04-29T13:21:25-0400
I still don't know if READ BUFF dumping is possible for Xbox discs to view the Xbox Partitions though...
saramibreak
commented
1 year ago valid extension was omitted. -> set .iso
ehw
commented
1 year ago valid extension was omitted. -> set .iso
- fixed: set .raw when /raw is used
GC still works, Xbox or normal DVDs still don't. :(
Space: 92001153024 bytes
=> There is enough disk space for dumpingRawdump command [0]:0x3c [1]:0x02 [2]:0000 Creating raw(LBA) 6832/ 6992 Layer is changed: 20 -> 01 Creating raw(LBA) 6992/ 6992 [F:CalcAndGetHash][L:465] GetLastError: 2, The system cannot find the file specified.
EndTime: 2023-04-29T14:10:15-0400
(Side note: Dumping normal DVDs is definitely a lot faster with READ BUFF than GC/Wii. Does this opcode only work for DVDs? Can it be used for CDs as well? Might be useful for GD discs if so? Also, can you still specify LBA range with /raw if you only wanted to dump a range of sectors?)
ehw
commented
1 year ago I'm looking at the .raw output for the Xbox disc, I noticed that the output isn't actually scrambled. So I guess there's nothing to descramble then, just deinterweave the EDC part?
saramibreak
commented
1 year ago ehw
commented
1 year ago
- added: generate .iso from .raw
GC and Xbox .raw conversion works now. :)
Not sure if there's anything that can be done to completely read an Xbox disc with READ BUFF, but other than that I don't seem to have any other issues at the moment unless you have some more things to test. Feel free to merge when ready.
Describe the solution you'd like Currently, DiscImageCreator does allow for Wii/GC disc dumping for certain drives that utilize a certain debug command. There are only a handful of drives that support the command, but there are other drives out there that seem to be able to read GC/Wii discs.
With Friidump, there is support for dumping Wii/GC discs using the (Plextor specific?) READ BUFF command. It's a non-streaming command that can slowly read the unscrambled sectors from GC/Wii discs. Believe it or not, it works!
But there's a caveat. At least with Friidumps implementation of the command, its extremely slow. It takes a little over 1 1/2 days to dump a GC disc, and 8 days or so for a single layer Wii disc. Because the drive keeps revving up, not to full speed thankfully, the drives can get a little hot. I've dumped both a Wii and GC disc, and it was okay though.
There is an additional advantage with supporting this, however. The supported drives so far only work with retail GC and Wii discs, not developer NR or RVT-R discs. Aside from swapping, there is no drive that we have tested (even the ones confirmed working with retail) that actually work. There are some that might, but it requires more investigation. However, both NRs and RVT-Rs work with this method for Plextors as well! This makes this method the only real off the shelf method for dumping developer discs without disc swapping, proprietary software, using expensive writers and readers, etc.
I propose DIC adds support for this. Even if it's really slow, it does have some use cases especially as a last resort.
Describe alternatives you've considered It'd be nice if there was some other Plextor command that could read these faster, or to see if the READ BUFF command can be improved. I think currently Friidump limits READ BUFF to 16 sectors at a time.
Additional context There is nice documentation here: The command used (3C is READ BUFFER) https://github.com/bradenmcd/friidump/blob/d9461a9401d64bc1c8ffb53f108bc1771736f8cd/libfriidump/vanilla_2064.c
Friidump forces "method 2" for plextors, the code for which is here. There are only two lines that occur when the method is 2. One seems to flush the cache, the other seems to read the sector. https://github.com/bradenmcd/friidump/blob/d9461a9401d64bc1c8ffb53f108bc1771736f8cd/libfriidump/disc.c#L244
Source code repository to the latest version of Friidump. https://github.com/bradenmcd/friidump