AoE2:DE Assets Support

[sandsmark]

Might as well start an issue for this (I hope that's okay).

Just downloaded it and started looking at the data files, and seems they have changed the graphics format again, now it is .smx files. Haven't gotten very far yet, just diffed a couple with dhex, and it seems like the header is now 53 4d 50 58 02 00, so 'SMPX' followed by 0x2 0x0 (that seems to be constant). Then there's two int32s, it seems, that varies in different graphics for the same unit, so I assume it is frame numbers.

I'm not sure if they did this with AoE:DE, but now they actually have a Tools_Build folder with various tools for modding, including a tool (DEAssetTool.exe) for converting SLPs or PSDs to SM(P)X files, which will probably help a lot. A fun fact is that they also include AGE/genieutils, which I guess indicates that the .dat formats haven't changed (I don't see the source anywhere for it, so I assume they haven't modified it, it being GPL and all). :-)

They also seem to do the terrain blending and hill stuff in shaders (I think they did the same in AoE1:DE), and also the conversion from SMP to RGB.

But now it's getting late, just wanted to get this down somewhere.

List:

• [ ] Wwise audio formats for sounds
• [x] SMX/SMP for sprites
• [x] SMX/SMP damage masks
• [ ] DDS for terrain files
• [ ] Particle effects (JSON config files)
• [ ] A bunch of blend masks are now configured by JSON files too

[sandsmark]

As for the other types of assets, they now use wwise for audio. And for some reason they seem to be including some ffmpeg stuff, but they also include lame so might just be for streaming support, I think they integrate with Mixer directly? it doesn't run in wine yet, so haven't actually run it.

[heinezen]

SMX conversion support will be added on Sunday, probably :) We've already figured out how it works, so documentation will come along with it to extend the already existing SMP file specs. The .dat file has one new entry (blood sprite ID, int16, comes after dead unit ID) that is xclusively used by the Konnik.

Fun fact: Microsoft right now violates the GPL with DE2 because they do not include a GPL license file for genieutils and AGE3 in the Tools Builds folder :laughing:

[simonsan]

@sandsmark Here are instructions how to get AoE2DE running on Wine: https://www.reddit.com/r/aoe2/comments/dwuplr/how_to_run_age_of_empires_2_definitive_edition_on/

[sandsmark]

SMX conversion support will be added on Sunday, probably :) We've already figured out how it works, so documentation will come along with it to extend the already existing SMP file specs. The .dat file has one new entry (blood sprite ID, int16, comes after dead unit ID) that is xclusively used by the Konnik.

Goddamn you're fast. :-)

I think reversing (technically) all the data formats in less than 10 days must be some kind of record. Though I guess it's kind of cheating when they share a lot of the code with something people have been picking apart for 20+ years. :p

Fun fact: Microsoft right now violates the GPL with DE2 because they do not include a GPL license file for genieutils and AGE3 in the Tools Builds folder

Hah, true. I was thinking about contacting them because I have a fork of AGE with a couple of new features (among other things).

I think they just put in whatever is in the zip file Tapsa publishes on aokheaven though, so it's an easy mistake to make. But then again I would assume their lawyers would be pretty wary of anything GPL (most of our lawyers are, except for the one who's an openbsd user).

[sandsmark]

@sandsmark Here are instructions how to get AoE2DE running on Wine: https://www.reddit.com/r/aoe2/comments/dwuplr/how_to_run_age_of_empires_2_definitive_edition_on/

Thanks! The timing of that post is a bit creepy though, that was almost exactly when I opened the exe in ghidra to debug it. :-P

edit; if you've tested it with proton, putting a link here would probably be nice; https://www.protondb.com/app/1017900 (I think that's where most people find information about stuff like that)

[sandsmark]

Just manually tried to parse the wav header (i. e. open one in dhex) in the wwise .wem files, and it seems to be AAC compressed. In addition to being annoying to decode even with another lib dependency, it seems like it uses channel coupling which faad2 doesn't support.

Reading this doc, though, it seems like Microsoft's decoder just skips frames with "channel coupling elements" which I take is what faad2 means. So faad2 refuses to continue, but just making it skip the frames with it should probably be OK.

edit; ignore me, ww2ogg seems to parse them just fine, so probably me screwing something up..

[brisvag]

edit; if you've tested it with proton, putting a link here would probably be nice; https://www.protondb.com/app/1017900 (I think that's where most people find information about stuff like that)

Yesterday I sent a report with a working solution (taken from some guy at the issue tracker relative to aoe2de here). The game works just fine :)

[heinezen]

1193 for SMX is now ready :)

[TheJJ]

smx support has landed!

[sandsmark]

What asset files are missing now? Just the sounds? (Wondering if I should rename this.)

[heinezen]

@sandsmark I think it would be good to have a checklist in your top post. The new formats are

• Wwise audio formats for sounds
• SMX/SMP for sprites (already supported by us, except for the damage mask)
• DDS for terrain files
• Particle effects (JSON config files)
• A bunch of blend masks are now configured by JSON files too

And maybe more. I didn't check every folder yet.

[sandsmark]

DDS for terrain files

groan... I looked into those for the special editions of monkey island, not a fun format.

any idea how they do sloping of terrain now? just in a vertex shader or something? (I haven't even been able to find a usable disassembler for HLSL, so it was just a guess from my side in my first post, based on the names etc.)

[heinezen]

groan... I looked into those for the special editions of monkey island, not a fun format.

RenderDoc has a parser for the format:

https://github.com/baldurk/renderdoc/blob/6d1d30249136e13f742f03c32ef18e09647be116/renderdoc/common/dds_readwrite.cpp

Can't recommend this program enough for people interested in rendering.

[sandsmark]

Sweet, thanks! That's the cleanest code I've seen for parsing DDS, maybe I can pick up the monkey island stuff as well.

Regarding the sounds, forgot to write it here; the files ww2ogg generates aren't playable, but vgmstream manages to convert just fine as wwise vorbis files. And vgmstream's wwise vorbis support is from ww2ogg, so probably just some fix there that could be backported.

I was originally planning to hack in support for the "broken" vorbis packets into stb_vorbis, but for openage it probably makes more sense to just fix the vorbis streams and store it as normal .oggs? Or do you want to convert everything to opus? Assuming you want to do as much of the asset conversion in Python I think just rewriting the wwise files to normal ogg files is probably easier (shouldn't be much Python code to recreate the broken/"replaced" packets).

[sandsmark]

FWIW; to look at the shaders you can use vkd3d-compiler to convert it to spirv, and then spirv-cross to convert it to glsl (or whatever you want). Seems to work fairly okay (though there seems to be some unrolling in the ConvertSMPToRGB shaders that's annoying to read).

edit; and before anyone else waste their afternoon; most of the huge blocks in CombineTerrainSpriteSMP_ps_Default.pso seem to be RGB to YUV conversion (the Y calculation is very obvious, and some of the other repeating blocks look like the yuv hlsl shader in mpv after going through spirv).

[TheJJ]

Currently the audio system only has opus playback support, and I'd prefer to keep it that way for now (we only link against opusfile). If we want to support multiple codecs and whatever formats (which we should, once e.g. users want to add taunts :grin: ), we have to consider how we implement that properly (and with cross plattform support). opus uses ogg as container.

[sandsmark]

yeah, I understand not wanting to link against multiple libraries, is there a huge difference in quality between opus and vorbis for the AoE sounds? Otherwise just using vorbis for everything might be easier.

and ref container; wwise is just a semi-custom wav/riff container + various codecs, but for DE they use vorbis. the problem is just that they mangle the vorbis stream a bit in their container, so to convert to "normal" .ogg you just need to load the .wwise (the container is more or less just standard RIFF but with an extended header afaik, recreate the missing vorbis header and slightly unmangle packets, and write it out to an .ogg file like any normal vorbis stream. tada, normal ogg vorbis.

[simonsan]

is there a huge difference in quality between opus and vorbis for the AoE sounds

Yes, opus is the superior format for audio and speech. To use ogg vorbis seems like a step back in this regards. https://opus-codec.org/comparison/

[TheJJ]

I guess we won't be able to hear the difference, especially since we convert it from $source to opus, and the quality of source can't be magically better after conversion. As vorbis is also by xiph and our audio system is of course prepared to handle multiple codecs, we could add it quite easily. Then we can perform your conversion from above. Alternatively, we could just read the file (with whatever codec necessary) and still convert it to opus, just because we can :) We selected opus as target codec because we want to be hipster and the source files were wav only :stuck_out_tongue:.

[heinezen]

The only other codec we maybe should prioritize is FLAC I think because it's losless like .wav. Other formats are still useful if someoe wants to import their own music playlist or mod sound files with other codecs in, but .opus and FLAC should cover 99% of use cases.

[simonsan]

the quality of source can't be magically better after conversion

Just some thoughts: This will not happen wit a simple conversion, but maybe with something like an music enhancement algorithm (Fréchet Audio Distance: A Metric for Evaluating Music Enhancement Algorithms) that we might be able to use in the future. Would be also quite interesting to use software to "automagically" reverse engineer the music and rearrange it to our liking but keeping some stuff (as much as we are allowed to from licensing reasons, see Kraftwerk vs. Moses P) as in the original.

[simonsan]

AoE2DE supports Opus now as well, does it change anything for the Wwise audio formats for sounds spec in here?

[sandsmark]

AoE2DE supports Opus now as well

I saw they used ffmpeg, but I assumed that was just for the mixer integration, but maybe they use it for more.

does it change anything for the Wwise audio formats for sounds spec in here

Their official assets still are in wwise's format, so I don't think so (unless they change to opus in a future release).

[duanqn]

I have AOE2:DE installed via Microsoft Store and it seems the files are very well protected. I can see the files but I have no ways to read/write them. Has anyone seen this before? Or are you all looking at the Steam version?

[heinezen]

@duanqn Apps from the MS store can be protected by UWP. You need to dump the data first, before writing/reading them with other projects. UWPDumper should do the trick.

[duanqn]

@heinezen Thank you. That tool looks really useful but AoE2:DE is nearly 20GB.

[sandsmark]

Apps from the MS store can be protected by UWP. You need to dump the data first, before writing/reading them with other projects. UWPDumper should do the trick.

If someone feels up to it I guess uwpdumper compatibility could be added to the asset converter (not very useful until aoe2:de assets are supported though).

Seems fairly straightforward (famous last words), once you're in the right part of the uwpdumper code you seem to have more or less normal access to the files: https://github.com/Wunkolo/UWPDumper/blob/master/UWPDumper/main.cpp#L91

(for those that don't know c++, that code seems to just list a directory and copy out all the files there normally, like any other program.)

[heinezen]

It would probably make sense to mount the files instead of dumping them, so that we don't need so much disk space. That would require more work though (we would need a Cython/Python interface).

[TheJJ]

That would be a new backend to the fslike subsystem :) (We originally started this to transparently mount the install CD)

[TheJJ]

Here ya go: #1247. Shouldn't be highly priorized though, it's just an extension for another "source" of data, the hard work is the DE-converter itself.