search

HearthSim / UnityPack

Python deserialization library for Unity3D Asset format
https://hearthsim.info/
MIT License
720 stars 153 forks source link

assets: compute CRC32 #75

Closed jnsnow closed 5 years ago

jnsnow commented 5 years ago

This computes the CRC32 that Unity uses as the Asset Bundle Manifest CRC32, for purposes of comparing a file against a known CRC32, to make sure that the bundle was downloaded properly.

This is ... a very quick and dirty hack, because this will only compute the asset bundle CRC32 for compressed assets that were added a certain way. It's probably desirable to compute the CRC32 in all cases if you wish to add it at all, but I wanted to submit some sample code as an RFE instead of just begging for free development.

It looks like for non-compressed assets we don't pre-read them on parse, so would you mind suggesting a better location in the code to calculate this CRC32?

thanks, --js

jleclanche commented 5 years ago

Yeah the buffering code is a little nasty, which makes adding things like these cleanly a bit problematic.

I'm fine merging this, but if you want it for all types of files, I think you might have to do a little refactoring. Easiest looking at the code right now would be to make ._buf a BytesIO, and initialize the BinaryReader in the parsing code instead. Then set .crc32 somewhere in there using _buf.getvalue().

jnsnow commented 5 years ago

I'll take a look. I've definitely gotten good mileage out of this project so far, so it's probably worth it.

jnsnow commented 5 years ago

NACK for now.

I'm looking at the asset bundle manifest provided for The Pokemon Trading Card Game Online and the CRCs there correlate sometimes to the headered and sometimes to the headerless decompressed versions of these compressed asset bundles.

(How does the stock client know how to differentiate when to use which? Does it check for either-or? does it check at all? I need to do more research.)

It'd be useful to have both headered and headerless CRCs, but since it makes opening the files quite slow, it should be an optional post-hoc call, I think. I'll rework this.

jleclanche commented 5 years ago

@jnsnow Alright, no worries. You can ping me on discord as well if you want to discuss this: jleclanche#0001 or https://discord.gg/hearthsim-devs

I've also been looking at reworking some bits of Unitypack. The overall logic is solid but the messy unity asset format has caused a lot of cruft and hacks to accumulate. It breaks on 2 Point Hospital assets, which I'd like to fix without breaking the rest.