search

PureDOS / DAT

ROM management DAT file of verified DOS games that can be easily loaded in a supported DOS emulator
3 stars 3 forks source link

MakePureDOSDAT fails with certain CHD files #2

Closed tomojira closed 1 day ago

tomojira commented 3 days ago

I've found that the MakePureDOSDAT utility fails when certain CHD files are included in the DOSZ, such as those for Warcraft II: Tides of Darkness + Beyond the Dark Portal. At first I thought it might be because the file size was too large - Tides of Darkness uncompressed CHD is 663 MB, Beyond the Dark Portal is 711 MB, and both need to be included. But Grand Theft Auto already has a large CHD in the DAT, so I'm not sure what the problem is. When the CHDs were deleted from the DOSZ, all the other files for Warcraft II processed correctly.

The 2 CHD files correspond with Redump:

They were created with the command chdman.exe createcd -i input.cue -o output.chd -c none

Here is some info from the chdman.exe info command if it helps:

chdman - MAME Compressed Hunks of Data (CHD) manager 0.257 (mame0257)
Input file:   .\WarCraft II - Tides of Darkness (USA) (Rerelease) (1996-10-05).chd
File Version: 5
Logical size: 708,725,376 bytes
Hunk Size:    19,584 bytes
Total Hunks:  36,189
Unit Size:    2,448 bytes
Total Units:  289,512
Compression:  none
CHD size:     695,349,504 bytes
Metadata:     Tag='CHT2'  Index=0  Length=93 bytes
              TRACK:1 TYPE:MODE1_RAW SUBTYPE:NONE FRAMES:101997 PREGAP:0 PGTYPE:MODE1 PGSUB:NONE POSTGAP:0.
Metadata:     Tag='CHT2'  Index=1  Length=89 bytes
              TRACK:2 TYPE:AUDIO SUBTYPE:NONE FRAMES:602 PREGAP:150 PGTYPE:VAUDIO PGSUB:NONE POSTGAP:0.
Metadata:     Tag='CHT2'  Index=2  Length=88 bytes
              TRACK:3 TYPE:AUDIO SUBTYPE:NONE FRAMES:16520 PREGAP:0 PGTYPE:MODE1 PGSUB:NONE POSTGAP:0.
Metadata:     Tag='CHT2'  Index=3  Length=88 bytes
              TRACK:4 TYPE:AUDIO SUBTYPE:NONE FRAMES:32387 PREGAP:0 PGTYPE:MODE1 PGSUB:NONE POSTGAP:0.
Metadata:     Tag='CHT2'  Index=4  Length=88 bytes
              TRACK:5 TYPE:AUDIO SUBTYPE:NONE FRAMES:16656 PREGAP:0 PGTYPE:MODE1 PGSUB:NONE POSTGAP:0.
Metadata:     Tag='CHT2'  Index=5  Length=88 bytes
              TRACK:6 TYPE:AUDIO SUBTYPE:NONE FRAMES:16228 PREGAP:0 PGTYPE:MODE1 PGSUB:NONE POSTGAP:0.
Metadata:     Tag='CHT2'  Index=6  Length=87 bytes
              TRACK:7 TYPE:AUDIO SUBTYPE:NONE FRAMES:4353 PREGAP:0 PGTYPE:MODE1 PGSUB:NONE POSTGAP:0.
Metadata:     Tag='CHT2'  Index=7  Length=87 bytes
              TRACK:8 TYPE:AUDIO SUBTYPE:NONE FRAMES:2836 PREGAP:0 PGTYPE:MODE1 PGSUB:NONE POSTGAP:0.
Metadata:     Tag='CHT2'  Index=8  Length=87 bytes
              TRACK:9 TYPE:AUDIO SUBTYPE:NONE FRAMES:4304 PREGAP:0 PGTYPE:MODE1 PGSUB:NONE POSTGAP:0.
Metadata:     Tag='CHT2'  Index=9  Length=89 bytes
              TRACK:10 TYPE:AUDIO SUBTYPE:NONE FRAMES:16188 PREGAP:0 PGTYPE:MODE1 PGSUB:NONE POSTGAP:0.
Metadata:     Tag='CHT2'  Index=10  Length=89 bytes
              TRACK:11 TYPE:AUDIO SUBTYPE:NONE FRAMES:16246 PREGAP:0 PGTYPE:MODE1 PGSUB:NONE POSTGAP:0.
Metadata:     Tag='CHT2'  Index=11  Length=89 bytes
              TRACK:12 TYPE:AUDIO SUBTYPE:NONE FRAMES:16201 PREGAP:0 PGTYPE:MODE1 PGSUB:NONE POSTGAP:0.
Metadata:     Tag='CHT2'  Index=12  Length=89 bytes
              TRACK:13 TYPE:AUDIO SUBTYPE:NONE FRAMES:15733 PREGAP:0 PGTYPE:MODE1 PGSUB:NONE POSTGAP:0.
Metadata:     Tag='CHT2'  Index=13  Length=89 bytes
              TRACK:14 TYPE:AUDIO SUBTYPE:NONE FRAMES:16088 PREGAP:0 PGTYPE:MODE1 PGSUB:NONE POSTGAP:0.
Metadata:     Tag='CHT2'  Index=14  Length=88 bytes
              TRACK:15 TYPE:AUDIO SUBTYPE:NONE FRAMES:6547 PREGAP:0 PGTYPE:MODE1 PGSUB:NONE POSTGAP:0.
Metadata:     Tag='CHT2'  Index=15  Length=88 bytes
              TRACK:16 TYPE:AUDIO SUBTYPE:NONE FRAMES:3188 PREGAP:0 PGTYPE:MODE1 PGSUB:NONE POSTGAP:0.
Metadata:     Tag='CHT2'  Index=16  Length=88 bytes
              TRACK:17 TYPE:AUDIO SUBTYPE:NONE FRAMES:3419 PREGAP:0 PGTYPE:MODE1 PGSUB:NONE POSTGAP:0.

     Hunks  Percent  Name
----------  -------  ------------------------------------
    35,498    98.1%  Uncompressed                            
       691     1.9%  Copy from parent                        
chdman - MAME Compressed Hunks of Data (CHD) manager 0.257 (mame0257)
Input file:   .\WarCraft II - Beyond the Dark Portal (Expansion Set) (USA) (Rerelease) (1996-10-09).chd
File Version: 5
Logical size: 760,485,888 bytes
Hunk Size:    19,584 bytes
Total Hunks:  38,832
Unit Size:    2,448 bytes
Total Units:  310,656
Compression:  none
CHD size:     745,699,968 bytes
Metadata:     Tag='CHT2'  Index=0  Length=92 bytes
              TRACK:1 TYPE:MODE1_RAW SUBTYPE:NONE FRAMES:89405 PREGAP:0 PGTYPE:MODE1 PGSUB:NONE POSTGAP:0.
Metadata:     Tag='CHT2'  Index=1  Length=89 bytes
              TRACK:2 TYPE:AUDIO SUBTYPE:NONE FRAMES:602 PREGAP:150 PGTYPE:VAUDIO PGSUB:NONE POSTGAP:0.
Metadata:     Tag='CHT2'  Index=2  Length=88 bytes
              TRACK:3 TYPE:AUDIO SUBTYPE:NONE FRAMES:29309 PREGAP:0 PGTYPE:MODE1 PGSUB:NONE POSTGAP:0.
Metadata:     Tag='CHT2'  Index=3  Length=88 bytes
              TRACK:4 TYPE:AUDIO SUBTYPE:NONE FRAMES:32387 PREGAP:0 PGTYPE:MODE1 PGSUB:NONE POSTGAP:0.
Metadata:     Tag='CHT2'  Index=4  Length=88 bytes
              TRACK:5 TYPE:AUDIO SUBTYPE:NONE FRAMES:16656 PREGAP:0 PGTYPE:MODE1 PGSUB:NONE POSTGAP:0.
Metadata:     Tag='CHT2'  Index=5  Length=88 bytes
              TRACK:6 TYPE:AUDIO SUBTYPE:NONE FRAMES:16228 PREGAP:0 PGTYPE:MODE1 PGSUB:NONE POSTGAP:0.
Metadata:     Tag='CHT2'  Index=6  Length=87 bytes
              TRACK:7 TYPE:AUDIO SUBTYPE:NONE FRAMES:4353 PREGAP:0 PGTYPE:MODE1 PGSUB:NONE POSTGAP:0.
Metadata:     Tag='CHT2'  Index=7  Length=87 bytes
              TRACK:8 TYPE:AUDIO SUBTYPE:NONE FRAMES:2836 PREGAP:0 PGTYPE:MODE1 PGSUB:NONE POSTGAP:0.
Metadata:     Tag='CHT2'  Index=8  Length=87 bytes
              TRACK:9 TYPE:AUDIO SUBTYPE:NONE FRAMES:4304 PREGAP:0 PGTYPE:MODE1 PGSUB:NONE POSTGAP:0.
Metadata:     Tag='CHT2'  Index=9  Length=89 bytes
              TRACK:10 TYPE:AUDIO SUBTYPE:NONE FRAMES:26776 PREGAP:0 PGTYPE:MODE1 PGSUB:NONE POSTGAP:0.
Metadata:     Tag='CHT2'  Index=10  Length=89 bytes
              TRACK:11 TYPE:AUDIO SUBTYPE:NONE FRAMES:16246 PREGAP:0 PGTYPE:MODE1 PGSUB:NONE POSTGAP:0.
Metadata:     Tag='CHT2'  Index=11  Length=89 bytes
              TRACK:12 TYPE:AUDIO SUBTYPE:NONE FRAMES:16201 PREGAP:0 PGTYPE:MODE1 PGSUB:NONE POSTGAP:0.
Metadata:     Tag='CHT2'  Index=12  Length=89 bytes
              TRACK:13 TYPE:AUDIO SUBTYPE:NONE FRAMES:15733 PREGAP:0 PGTYPE:MODE1 PGSUB:NONE POSTGAP:0.
Metadata:     Tag='CHT2'  Index=13  Length=89 bytes
              TRACK:14 TYPE:AUDIO SUBTYPE:NONE FRAMES:16088 PREGAP:0 PGTYPE:MODE1 PGSUB:NONE POSTGAP:0.
Metadata:     Tag='CHT2'  Index=14  Length=88 bytes
              TRACK:15 TYPE:AUDIO SUBTYPE:NONE FRAMES:6547 PREGAP:0 PGTYPE:MODE1 PGSUB:NONE POSTGAP:0.
Metadata:     Tag='CHT2'  Index=15  Length=88 bytes
              TRACK:16 TYPE:AUDIO SUBTYPE:NONE FRAMES:3188 PREGAP:0 PGTYPE:MODE1 PGSUB:NONE POSTGAP:0.
Metadata:     Tag='CHT2'  Index=16  Length=88 bytes
              TRACK:17 TYPE:AUDIO SUBTYPE:NONE FRAMES:3419 PREGAP:0 PGTYPE:MODE1 PGSUB:NONE POSTGAP:0.
Metadata:     Tag='CHT2'  Index=17  Length=89 bytes
              TRACK:18 TYPE:AUDIO SUBTYPE:NONE FRAMES:10355 PREGAP:0 PGTYPE:MODE1 PGSUB:NONE POSTGAP:0.

     Hunks  Percent  Name
----------  -------  ------------------------------------
    38,068    98.0%  Uncompressed                            
       764     2.0%  Copy from parent
PureDOS commented 2 days ago

Should be fixed now! Thanks for reporting!

tomojira commented 2 days ago

Yep, it works now, thank you!

PureDOS commented 2 days ago

I've noticed something strange, perhaps annoying. I ran the same chdman command and I got this output:

chdman - MAME Compressed Hunks of Data (CHD) manager 0.266 (mame0266)
Input file:   wc2btdp.chd
File Version: 5
Logical size: 760,485,888 bytes
Hunk Size:    19,584 bytes
Total Hunks:  38,832
Unit Size:    2,448 bytes
Total Units:  310,656
Compression:  none
CHD size:     745,621,632 bytes
Metadata:     Tag='CHT2'  Index=0  Length=92 bytes
              TRACK:1 TYPE:MODE1_RAW SUBTYPE:NONE FRAMES:89405 PREGAP:0 PGTYPE:MODE1 PGSUB:NONE POSTGAP:0.
Metadata:     Tag='CHT2'  Index=1  Length=89 bytes
              TRACK:2 TYPE:AUDIO SUBTYPE:NONE FRAMES:602 PREGAP:150 PGTYPE:VAUDIO PGSUB:NONE POSTGAP:0.
Metadata:     Tag='CHT2'  Index=2  Length=88 bytes
              TRACK:3 TYPE:AUDIO SUBTYPE:NONE FRAMES:29309 PREGAP:0 PGTYPE:MODE1 PGSUB:NONE POSTGAP:0.
Metadata:     Tag='CHT2'  Index=3  Length=88 bytes
              TRACK:4 TYPE:AUDIO SUBTYPE:NONE FRAMES:32387 PREGAP:0 PGTYPE:MODE1 PGSUB:NONE POSTGAP:0.
Metadata:     Tag='CHT2'  Index=4  Length=88 bytes
              TRACK:5 TYPE:AUDIO SUBTYPE:NONE FRAMES:16656 PREGAP:0 PGTYPE:MODE1 PGSUB:NONE POSTGAP:0.
Metadata:     Tag='CHT2'  Index=5  Length=88 bytes
              TRACK:6 TYPE:AUDIO SUBTYPE:NONE FRAMES:16228 PREGAP:0 PGTYPE:MODE1 PGSUB:NONE POSTGAP:0.
Metadata:     Tag='CHT2'  Index=6  Length=87 bytes
              TRACK:7 TYPE:AUDIO SUBTYPE:NONE FRAMES:4353 PREGAP:0 PGTYPE:MODE1 PGSUB:NONE POSTGAP:0.
Metadata:     Tag='CHT2'  Index=7  Length=87 bytes
              TRACK:8 TYPE:AUDIO SUBTYPE:NONE FRAMES:2836 PREGAP:0 PGTYPE:MODE1 PGSUB:NONE POSTGAP:0.
Metadata:     Tag='CHT2'  Index=8  Length=87 bytes
              TRACK:9 TYPE:AUDIO SUBTYPE:NONE FRAMES:4304 PREGAP:0 PGTYPE:MODE1 PGSUB:NONE POSTGAP:0.
Metadata:     Tag='CHT2'  Index=9  Length=89 bytes
              TRACK:10 TYPE:AUDIO SUBTYPE:NONE FRAMES:26776 PREGAP:0 PGTYPE:MODE1 PGSUB:NONE POSTGAP:0.
Metadata:     Tag='CHT2'  Index=10  Length=89 bytes
              TRACK:11 TYPE:AUDIO SUBTYPE:NONE FRAMES:16246 PREGAP:0 PGTYPE:MODE1 PGSUB:NONE POSTGAP:0.
Metadata:     Tag='CHT2'  Index=11  Length=89 bytes
              TRACK:12 TYPE:AUDIO SUBTYPE:NONE FRAMES:16201 PREGAP:0 PGTYPE:MODE1 PGSUB:NONE POSTGAP:0.
Metadata:     Tag='CHT2'  Index=12  Length=89 bytes
              TRACK:13 TYPE:AUDIO SUBTYPE:NONE FRAMES:15733 PREGAP:0 PGTYPE:MODE1 PGSUB:NONE POSTGAP:0.
Metadata:     Tag='CHT2'  Index=13  Length=89 bytes
              TRACK:14 TYPE:AUDIO SUBTYPE:NONE FRAMES:16088 PREGAP:0 PGTYPE:MODE1 PGSUB:NONE POSTGAP:0.
Metadata:     Tag='CHT2'  Index=14  Length=88 bytes
              TRACK:15 TYPE:AUDIO SUBTYPE:NONE FRAMES:6547 PREGAP:0 PGTYPE:MODE1 PGSUB:NONE POSTGAP:0.
Metadata:     Tag='CHT2'  Index=15  Length=88 bytes
              TRACK:16 TYPE:AUDIO SUBTYPE:NONE FRAMES:3188 PREGAP:0 PGTYPE:MODE1 PGSUB:NONE POSTGAP:0.
Metadata:     Tag='CHT2'  Index=16  Length=88 bytes
              TRACK:17 TYPE:AUDIO SUBTYPE:NONE FRAMES:3419 PREGAP:0 PGTYPE:MODE1 PGSUB:NONE POSTGAP:0.
Metadata:     Tag='CHT2'  Index=17  Length=89 bytes
              TRACK:18 TYPE:AUDIO SUBTYPE:NONE FRAMES:10355 PREGAP:0 PGTYPE:MODE1 PGSUB:NONE POSTGAP:0.

     Hunks  Percent  Name
----------  -------  ------------------------------------
    38,064    98.0%  Uncompressed
       768     2.0%  Copy from parent

~So... this is ALMOST the same but not exactly. Which means the CHD you made and the one I made won't hash the same. Which is quite a problem for what we're trying to achieve here.~

-chdman - MAME Compressed Hunks of Data (CHD) manager 0.257 (mame0257)
+chdman - MAME Compressed Hunks of Data (CHD) manager 0.266 (mame0266)
-CHD size:     745,699,968 bytes
+CHD size:     745,621,632 bytes
-    38,068    98.0%  Uncompressed                            
-       764     2.0%  Copy from parent   
+    38,064    98.0%  Uncompressed                            
+       768     2.0%  Copy from parent

~So my chdman version 0.266 makes a different CHD than your version 0.257. We will probably need to specify the exact version of chdman that needs to be used when working with this DAT project... I'll try to accumulate various chdman versions to see which one makes what kind of file.~

Edit: Oh, I guess there is a chance that the sources we are using are different. Same exact CD layout but a difference in data. I'll try to see what 0.257 makes with the source I have.

Edit2: Ok, false alarm, it turns out that all versions of chdman from 0.255 to current 0.267 produce exactly the same uncompressed .CHD. Seems we have just a different CD rip, perhaps with slight different amounts of silence in the audio tracks.

tomojira commented 2 days ago

Phew, that's a relief. I just tried 0.267 and can also confirm it gave the same result I got from 0.257.

PureDOS commented 2 days ago

Thanks for confirming!

It does leave us with another confirmation that CD rips can very easily have minute differences which could make it very hard for someone to get the exact .CHD file requested by the DAT file.

Perhaps the track listing for the DAT file should also report how many bytes of silence are at the start and end of each track. A clever tool could then try to take a mismatching rip and shift things up to make a matching CHD. Perhaps we need a MakePureDOSZ tool...