search

sourcehold / sourcehold-maps

Reverse engineering the map file format of Stronghold
GNU General Public License v3.0
20 stars 3 forks source link

minimap in .map file: PKWare compression format? #1

Closed gynt closed 5 years ago

gynt commented 5 years ago

@metalvoidzz I saw your update on the wiki regarding map file formats and PKWare compression. This compression may apply to the image at the start of the .map file as well, as I found some code in PNGer.exe that resembles the structs in blast.c

Nice find!

janso3 commented 5 years ago

I also think the preview is compressed. If you look at the beginning of any map file, the 0xFF's and 4 additional bytes are immediately followed by one of these 12-byte long 'section headers' (I added that to the wiki as well) and some compressed data, which is probably the preview. I haven't confirmed that yet though.

gynt commented 5 years ago

Decompressing (blasting) the preview (starts at offset 20 I think) does not lead to a (by windows) readable image file straight away, there is an intermediate step. Also, I cannot find the decompressed data in the output file of PNGer.exe , it is probably compressed again. Digging further ...

gynt commented 5 years ago

I just found out that a CRC32 checksum is also computed on the blasted preview. I am still not completely sure how the final png is created.

However, the blasted preview does contain the right pixel information, with 1 byte representing a pixel (probably a color index). It is offset by 57*16 + 4 from the start of the blasted preview. Not sure why it starts there. I guess the data before that contains color data?

janso3 commented 5 years ago

Which map are you testing this on btw? For some files, I'm not getting correct results after decompression.

gynt commented 5 years ago

I updated the wiki just now.

I am using a 160x160 map generated by SH(C) map editor. https://github.com/sourcehold/sourcehold-maps/blob/master/resources/MxM_unseen_1.map

gynt commented 5 years ago

@metalvoidzz I have just released version 0.1 of the preview image extractor. Can you perhaps verify which .map files are not working for you?

janso3 commented 5 years ago

Sure, I'll test it when I'm at home. Great job!

janso3 commented 5 years ago

Yep, works for me now!

lawrencefoley commented 5 years ago

@gynt Not sure if this is the place for this, or if I should open a new issue, but I'm getting an error from the blast.exe when running preview_image_extractor.py. I'm using Python 3.7.4 on Windows 10. After some searching, it looks like I this error code comes up when a DLL is missing. Are there any files needed for blast.exe? Also, by the way I'm really excited to see this project; I've used "PNGer" and wanted to be able to create something like this myself. I would like to contribute if possible.

Traceback (most recent call last):
  File "D:/Downloads/sourcehold-maps-master/preview_image_extractor.py", line 42, in <module>
    image = extract_preview(f)
  File "D:/Downloads/sourcehold-maps-master/preview_image_extractor.py", line 9, in extract_preview
    result.check_returncode()
  File "C:\Users\Lawrence\AppData\Local\Programs\Python\Python37-32\lib\subprocess.py", line 428, in check_returncode
    self.stderr)
subprocess.CalledProcessError: Command 'resources/blast.exe' returned non-zero exit status 3221225781.

Process finished with exit code 1
gynt commented 5 years ago

Hi there!

I compiled blast.exe from here https://github.com/madler/zlib/blob/master/contrib/blast/blast.c I think I used a MSYS2 C compiler, maybe that causes the missing DLL message for you.

It is a bit tricky to debug, because I am not getting your error. I added another blast.exe that is compiled slightly differently, let's hope that works.

lawrencefoley commented 5 years ago

Good to know; Thank you for the info!

I'll be looking for your commit. If that doesn't work I might try to compile blast.c when I get home later.

gynt commented 5 years ago

@lawrencefoley Sounds good! I made some progress on the map file specification. I think we need a big database of .map files in which we systematically manipulate stuff.

What do you think?

lawrencefoley commented 5 years ago

@gynt I was messing with recompiling the blast.exe this weekend. I found out the version you compiled depended on MSYS2 to run. I was able to compile a standalone version with Cygwin, but it was returning a different error code, 2. It returns that error here, but I'm not sure why it would read past the input.

Anyways, I plan on following the debugger setup instructions you have in the develop branch when I get some time. I haven't messed with assembly code very much, so this will be interesting. 🙂

I agree with the database of map files. Right now I only have Stronghold Crusader installed; Will those map files be the same as Stronghold? Do you think we should just have different maps, with certain tiles that have certain materials and keep track of the different variations? I assume we would use those to find where the .map files change.

lawrencefoley commented 5 years ago

@gynt I was able to test out my compiled version of blast.c using the test data available in the source for it. Here is my test code that is similar to what you are using in preview_image_extractor.py. I'm able to get the expected output using this test code, but whenever I use this compiled version of blast.c with the preview_image_extractor.py it doesn't work.

Are you running your code on Windows 10 and Python 3.7? Wondering if there might be some environment differences. I've attached my complied version. blast.zip

import subprocess, io

inputData = b'\x00\x04\x82\x24\x25\x8f\x80\x7f'
result = subprocess.run("blast.exe", input = inputData, stdout = subprocess.PIPE)
result.check_returncode()

print(result.stdout)
assert(result.stdout == b"AIAIAIAIAIAIA")
gynt commented 5 years ago

@lawrencefoley I also cannot run your binary. It seems to print out 6679696 and then a part of the correct decompression result. Not sure what is going on there.

lawrencefoley commented 5 years ago

@gynt Which OS and Python version are you using?

lawrencefoley commented 5 years ago

@Gootjes I am running Windows 10 Python 3.7.4. I was actually using the 32bit version and just removed it and installed the 64bit version to be sure that wasn't the issue.

Are you able to run the preview_image_extractor.py successfully?

gynt commented 5 years ago

@lawrencefoley I released a new version that includes new binaries. Please let me know if they do not work. You can open a new issue!