denilsonsa
commented
1 month ago To verify if the issue was related to multi-threading, I tried this creating a CHD file three times with the default number of processes and three times more with a single thread:
for a in 1 2 3 ; do chdman createdvd -i AnyLargePS2DVD.iso -o multi-$a.chd ; done
for a in 1 2 3 ; do chdman createdvd -i AnyLargePS2DVD.iso -np 1 -o single-$a.chd ; doneBoth the source and the destination were on the same folder, a CIFS-mounted shared folder from my NAS. The result wasn't good:
$ md5sum multi-* single-*
ea3bc4e3ad10ca387e7ed734501d476a multi-1.chd
5ab5db891572d1301aa41b3b2624684e multi-2.chd
ea3bc4e3ad10ca387e7ed734501d476a multi-3.chd
c610c5723decf413b295b132d51e5ae3 single-1.chd
b2b54cdcd115663df0dbd27ac9afd3c0 single-2.chd
945f851408e0c87df6ad9fe816cd87f9 single-3.chdThen I decided the same approach, but with the source .ISO file on the CIFS mount and the destination .CHD file on /tmp RAM drive. The result was promising:
$ md5sum /tmp/multi-* /tmp/single-*
ea3bc4e3ad10ca387e7ed734501d476a /tmp/multi-1.chd
ea3bc4e3ad10ca387e7ed734501d476a /tmp/multi-2.chd
ea3bc4e3ad10ca387e7ed734501d476a /tmp/multi-3.chd
ea3bc4e3ad10ca387e7ed734501d476a /tmp/single-1.chd
ea3bc4e3ad10ca387e7ed734501d476a /tmp/single-2.chd
ea3bc4e3ad10ca387e7ed734501d476a /tmp/single-3.chdI also tried two other combinations of CIFS and RAM as either sources or destinations. Here's the results:
| CIFS destination | RAM drive destination | |
|---|---|---|
| CIFS source | inconsistent | perfect |
| RAM drive source | inconsistent | perfect |
This makes me believe the process of writing the CHD file has some assumption that isn't working here. Maybe it tries to write and read and seek too quickly, or without flushing the changes. I don't know, I haven't looked at the code.
Given the recent findings, I'm updating this issue title from "chdman can't verify CHD files created by createdvd" to "chdman createdvd writess corrupt CHD files on CIFS storage".
One more note: I know I have created many .CHD files from CUE/BIN CD-based images, and they all verified correctly. This makes me believe the issue is only with createdvd, and not with createcd.
mamehaze
MooglyGuy
wilbertpol
MAME version
0.267 (mame0267-dirty)
System information
Manjaro Linux, x86_64
Intel i7-6700 CPU with 8 cores/threads
Storage on a NAS device mounted over CIFS.
INI configuration details
No response
Emulated system/software
No response
Incorrect behaviour
Obtain a Playstation 2 ISO file from a game that was released as DVD.
Convert the DVD ISO file to a CHD file:
Try to verify it:
What happens:
Decompression errorCompression errorBut the behavior is always consistent on the same .CHD file.
Expected behaviour
It should be able to
verifyany CHD created viacreatedvd. It shouldn't crash either.Steps to reproduce
No response
Additional details
If I were to guess, I would think the
createdvdisn't deterministic. There might be some uninitialized data that sometimes gets corrupted and sometimes gets written wrong on the CHD file. Or maybe the file seek/write/sync/flush isn't done correctly. Or perhaps it's related to running the compression in parallel on multiple CPU cores.Or, I would guess
verifycode is buggy, as the code shouldn't ever segfault.Given the random nature of this, it may as well happen that my hardware is defective somehow. Although that's weird, given it correctly verified many CHD files, but only had trouble with a few of the ones I had created using
createdvd. The CHD files generated from CD bin/cue files worked just fine.I know debugging something random like this is a nightmare. I'm sorry about it.
As a guess after some more thought, it could be related to racing conditions on the
createdvdmethod when running in parallel. Since the storage is on a spinning disk in a network device (Synology NAS with limited CPU power), the bottleneck is probably the I/O. Some of the CPU threads might not run in the expected timing due to delays in the I/O. If this is the case, moving the ISO to/tmp(a RAM drive) and creating the CHD over there shouldn't have I/O issues. Well, still a mystery for me, and testing this takes quite some time.