search

temisu / ancient

Decompression routines for ancient formats
BSD 2-Clause "Simplified" License
209 stars 14 forks source link

Incorrect checksum calculation for older MMCMP files #32

Closed sagamusix closed 3 years ago

sagamusix commented 3 years ago

Creagaia.zip

This is the demo song coming with Impulse Tracker 1.06, compressed with MMCMP. When checksum verification is enabled, it fails to load (actual checksum 0x418b9db4, expected checksum 0x02df3fbc). Since OpenMPT never verified checksums, I'm not sure if the checksum is simply incorrect or if the checksum calculation is wrong. As far as I can tell, the uncompressed data is correct when ignoring checksum verification.

sagamusix commented 3 years ago

Interesting, MMCMP 1.30 (which ships with that version of Impulse Tracker) crashes DOSBox when trying to uncompress that file. MMCMP 1.34 extracts it without errors. Edit: Nevermind, MMCMP 1.30 just generally crashes in DOSBox, it's not related to this file in particular.

The changelog says for version 1.31:

Change: Destination verification on decompressing wasn't quite right, and the verification scheme is changed to something much better. What it means? DON'T CARE ABOUT "Check destination" ERROR messages if you're USING MMCMP 1.30 OR OLDER!!

Sounds like something about the checksum might have changed between MMCMP versions?

I re-compressed the same module with MMCMP 1.34, and one notable change is that the version field at byte offset 10 changes from 0x130F to 0x1340, so maybe this can be used to tell various checksum algorithms apart.

Edit: Using MMCMP 1.34, it starts to warn that the result might be corrupted as soon as the version is >= 0x1320. I'm currently checking its disassembly to see if I can spot what has changed.

sagamusix commented 3 years ago

After looking at the MMCMP code a bit more, I can confirm that MMCMP 1.34 simply ignores the checksum if the file version is < 0x1320. So the checksum algorithm used in MMCMP 1.34 and ancient is not suitable for those files. I will see if I can figure out what the algorithm looks like in MMCMP 1.30.

sagamusix commented 3 years ago

Okay, even more findings... I'm getting close!

For those older files, the checksum is done by XOR-ing the output in 32-bit chunks, without any rotation. However, it's not quite clear yet what this implies for blocks with a length that's not a multiple of 4, when the start offset is not a multiple of 4, etc... some of those cases don't update the checksum correctly yet but at least for the first block I'm getting the correct checksum. Length seems to be rounded down to multiples of 4 (so e.g. if the block size is 7, the last 3 bytes are not verified).

Edit: currently my modified checksum code fails on i=5 in Creagaia.it, which is the first block containing more than one sub-block. I guess it's related to that.

temisu commented 3 years ago

Well, you were on to something here. I think I got it working

(Easiest would be just to disable checksum but that is lazy)

I have files which have version 0x1310 that do have the new checksum. So I used the 0x130f <-> 0x1310 as a threshold

The checksum is accumulated byte by byte, and full dwords are then accumulated.

sagamusix commented 3 years ago

Hah, I'm not quite sure yet why your code works and mine didn't (I was working closely with the disassembly), but great to see that it's working now. :) Can you send me the file that has version 0x1310? I'd like to see how MMCMP 1.34 behaves when altering checksums there, because I definitely saw 0x1320 being used in the code - that would mean that versions 0x1310-0x131F should be checked by MMCMP 1.34 but are in fact not.

sagamusix commented 3 years ago

I found MMCMP 1.32, which writes 0x1320 and uses the new checksum algorithm. This makes sense in retrospect because 0x1310 should be MMCMP 1.31, which is exactly the version that mentions the bugfix in the changelog. Would be great to find MMCMP 1.31 for final verification, but so far I haven't been successful.