Does this script still work?

[AndrewIdot]

Trying to unpack the Assets.dat on the latest version of the game, seems to be failing due to wrong asset offset values (at least based on the other issues I've seen opened up here).

Anyone know what's going on?

Traceback (most recent call last):
  File "fp-assets.py", line 477, in <module>
    assets_dir_path=assets_dir_path)
  File "fp-assets.py", line 405, in extract_all_assets
    "img_%d_meta.txt" % index)
  File "fp-assets.py", line 130, in extract_img
    img_w = struct.unpack("<H", assets_file.read(2))[0]
struct.error: unpack requires a bytes object of length 2

[BetaRelics]

Just wanted to add that I'm running into a similar issue(although at lines 476, 404, and 130) does anyone know if there's a way to update the offset values/fix this?

 Traceback (most recent call last):
  File "fp-assets.py", line 476, in <module>
    extract_all_assets(assets_file_path=assets_file_path, \
  File "fp-assets.py", line 404, in extract_all_assets
    extract_img(assets_file, "img_%d.png" % index, \
  File "fp-assets.py", line 130, in extract_img
    img_w = struct.unpack("<H", assets_file.read(2))[0]
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
struct.error: unpack requires a buffer of 2 bytes

[snickerbockers]

It looks like they've updated Freedom Planet in the time since I wrote this, and that includes a new Assets.dat file which stores its metadata block at a different offset than the old one did. I'll take a look at things and see if I can figure it out; it's been a long time since I last worked on this script (i moved on to sega dreamcast emulation shortly after writing this and I'm still working on it 7 years later), so it might take a while to get the new Assets.dat reverse-engineered.

This is the md5sum of the older Assets.dat which this script was written for: f14f24317d0323d63231cdbba511f254

It also seems that python3 very rudely changed its behavior in a way that breaks compatibility with fp-assets, so I made commit 089d2b1457f3479b850066b4d048c2fb0c8fd9bb to fix that, but that won't resolve the problem with the newer Assets.dat file.

Like I said, I'll see if I can figure it out but in the meantime it should work if you checkout the new commit I just pushed and you can get your hands on the old Assets.dat which has an md5sum of f14f24317d0323d63231cdbba511f254 .

[snickerbockers]

spent a couple hours looking into this, posting what i've found so far posterity's sake:

The md5sum of the latest version of Assets.dat from GOG for the Linux version of freedom planet: fb95f5c4809e76cae933be20ca51a660

These are the hard-coded offset to the metadata block as well as the lengths of each type of file. Note that there are apparently 0 fonts; the older version Assets.dat file had 1 font.

OFFSETS_START=0x8614
IMG_COUNT=17162
SOUND_COUNT=475
FONT_COUNT=0 # is this right???
SHADER_COUNT=95
FILE_COUNT=18
TYPE_SIZE_COUNT = 5

Something is fundamentally different about the way images are stored in the newer version of the Assets.dat file; further reverse-engineering will be required to understand how it works before fp-assets can be fixed for the newer version.

[snickerbockers]

the function that loads image data has been located; it is at address 007a8e80 in a Chowdren 64-bit x86 linux executable with an md5sum of f87a0957ad9320cfaca244342bc6e94d. I do not yet know what compression algorithm this function is using; previous versions of freedom planet used zlib to implement image compression.

this function is not aware of the resolution of the image it is loading, only its initial compressed size and its expected uncompressed size; this means that the compression algorithm chowdren is using must be a general-purpose compression algorithm and not something like JPEG that only makes sense in the context of image compression.

Another interesting detail is that this function does not know any of the image's metadata, which implies that there is only one pixel format being used.

[snickerbockers]

another interesting detail is that it allocates over 128kb of stack space which tells us two things:

• this probably uses some sort of moving window not unlike zlib's deflate algorithm
• MP2 is really careless with his stack allocation. or maybe im just too used to working on kernel-level code where allocating massive amounts of stack memory is a big no-no.

[BetaRelics]

Thank you for both looking into this and for the quick and in depth responses! Honestly, most of this is new to me so it's been fun learning about all the specifics. I'll see if I can get the old Assets.dat working with the new commit like you suggested, if I run into any issues I'll let you know.

[BetaRelics]

Ran into an issue. To clarify, I tried using the Assets.dat files from the various steam releases. First tried the 1.21.5 build which resulted in this

Traceback (most recent call last):
  File "fp-assets.py", line 476, in <module>
    extract_all_assets(assets_file_path=assets_file_path, \
  File "fp-assets.py", line 404, in extract_all_assets
    extract_img(assets_file, "img_%d.png" % index, \
  File "fp-assets.py", line 141, in extract_img
    file_dat = zlib.decompress(assets_file.read(file_len))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
zlib.error: Error -3 while decompressing data: unknown compression method

It seems to have extracted a decent amount of images but nothing else (last file is "img_16881_meta" with no associated png unlike all the other meta files in the "images" folder) Not sure if the issue is because of the asset.dat I used or if it was a weird python issue again. after a few google searches I attempted to use certutil -hashfile Assets.dat MD5 to check the md5sum which resulted in 1b77253f668ff745e0f8bf7d0e9d8b2d which does not match the md5sum you listed prior. As an extra note: the 1.21.5 build was also the only build in the steam beta releases that even got this far as the other build I tried resulted in struct.error: unpack requires a buffer of 2 bytes

[snickerbockers]

it was a problem with the Assets.dat file you used. This game hard-codes the offset of the metadata block and how much of each asset type is in the file, so fp-assets.py has to be changed to support every version. There's no way to get that from the file itself so fp-assets.py has to be specifically written for every version of Assets.dat they release.

[snickerbockers]

@AndrewIdot @BetaRelics i have successfully reverse-engineeered and implemented freedom planet's new image format. If you check out the new_assets_format branch, you'll be able to extract the Assets.dat with an md5sum of fb95f5c4809e76cae933be20ca51a660; this is the one that currently ships with the linux release on GoG; hopefully it's the same on steam and windows.

it cannot yet create a new Assets.dat because I have not yet implemented compressing images with the new format; I have only implemented extraction for now.

[snickerbockers]

i just checked windows, it seems it has a different Assets.dat file from linux but the offsets-block and the asset counts are identical to the linux version; probably all they did was change sprites or text which reference OS-specific details. I pushed out a new update to the new_assets_format branch. On this branch fp-assets.py now supports Assets.dat with any of the following three md5 checksums:

fb95f5c4809e76cae933be20ca51a660 latest version on linux (extract only) 4085c983fb918703ce459f17f69c474c latest version on windows (extract only) f14f24317d0323d63231cdbba511f254 antiquated build from 2016 that i originally designed fp-assets.py for

[BetaRelics]

The new branch seems to have worked! Thank you for taking the time to revisit and update this script! (also sorry for taking like a week to respond)

[snickerbockers]

glad it's working for you! I'll get the compression working by the end of the month; it could take a while because of personal reasons. After that I'll get this all rebased and pushed to the master branch.

[snickerbockers]

@AndrewIdot @BetaRelics i've finished implementing file compression and merged it to master, so fp-assets.py should work for you on the latest version of freedom planet for both extraction (-x) and compression (-c). let me know if there are any problems.