PhantasyServer / pso2-protocol-lib

Apache License 2.0
5 stars 1 forks source link

Update `packets.hexpat` to support PPAC 4 (zstd) #3

Open rockisch opened 2 months ago

rockisch commented 2 months ago

Thought it would be interesting to create a tracking issue for this, since I was looking into it.

One of the reasons I think parsing PPAC v4 wasn't possible is because the current implementation uses the zstd streaming API to save the files, and ImHex's zstd_decompress didn't support that.

This PR fixes that: https://github.com/WerWolv/ImHex/pull/1898

With that out of the way, it is now possible to parse the zstd data... but the way it is displayed is kinda weird. Here's the updated parts of the script:

import hex.dec;

...

struct file {
    Header header;
    if (header.Ver >= 3) {
        u8 is_ngs;
    }
    if (header.Ver >= 4) {
        u8 is_zstd;
    }
    if (header.Ver == 1) {
        Packet_v1 packet[while(!std::mem::eof())];
    } else if (header.Ver == 2 || header.Ver == 3 || (header.Ver == 4 && !is_zstd)) {
        Packet_v2 packet[while(!std::mem::eof())];
    } else if (header.Ver == 4 && is_zstd) {
        u8 packet_zstd[while(!std::mem::eof())] [[hidden]];
        std::mem::Section packet_section = std::mem::create_section("data_section") [[export]];
        hex::dec::zstd_decompress(packet_zstd, packet_section);
        Packet_v2 packet[while(!std::mem::eof())] @ 0x00 in packet_section;
    }
};

This works under the PR branch, but it seems like the way ImHex pattern scripts treats @ 0x00 in <section> is a bit weird: https://github.com/WerWolv/ImHex/issues/1899

In order to access the data you need to go to the sections tab which looks like this: image

As of right now, you can't resize this window, which makes it pretty much useless. I'm trying to see if there are any better options on the above ticket.

AntonnMal commented 2 months ago

Little background on the packets.hexpat: when I was writing the pattern file, I wasn't planning on adding compression, but when the time came, I was already using other tools to extract data (ppak_reader), so I've kinda gave up on the pattern. I didn't know that there were decompression functions. Also I believe that viewing large packet files in ImHex is not a nice experience.

Anyway, when compression was added to the format, I've written a small program (reppac) to decompress the files, so they could be viewed without trouble in other editors.

Looking through the linked issue I've noticed a mention about virtual files. In the pattern docs there is one function mentioned that is linked to virtual files: hex::core::add_virtual_file

struct file {
    Header header;
    if (header.Ver >= 3) {
        u8 is_ngs;
    }
    if (header.Ver >= 4) {
        u8 is_zstd;
    }
    if (header.Ver == 1) {
        Packet_v1 packet[while(!std::mem::eof())];
    } else if (header.Ver == 2 || header.Ver == 3 || (header.Ver == 4 && !is_zstd)) {
        Packet_v2 packet[while(!std::mem::eof())];
    } else if (header.Ver == 4 && is_zstd) {
        u8 packet_zstd[while(!std::mem::eof())] [[hidden]];
        std::mem::Section packet_section = std::mem::create_section("data_section");
        hex::dec::zstd_decompress(packet_zstd, packet_section);
        Packet_v2 packet[while(!std::mem::eof())] @ 0x00 in packet_section;
        hex::core::add_virtual_file("a", packet);
    }
};

In the virtual files section a new entry appears: image

And it can be opened as a reguar file. However it requires slightly modified hexpat to parse. Maybe there is some way to copy data in the pattern between files or views.