cnexabyte
commented
2 years ago Big Thanks
Open emoose opened 2 years ago
cnexabyte
commented
2 years ago Big Thanks
emoose
commented
2 years ago Huh, just found out PlatinumGames Bayonetta did actually have some symbols leaked, https://archive.org/details/Bayonetta20090524 includes a nice 60MB PDB with it, which IDA loads in fine with https://github.com/emoose/idaxex (and PDB_PROVIDER = PDB_PROVIDER_MSDIA set), seems to name pretty much every function & gives plenty of structs/local types too 😸
Haven't looked too deep into it yet, but seems despite similar names not many functions are actually shared with RE4, classes like cModel/cParts/etc seem to contain completely different fields inside too...
I guess the PG dev-team maybe had to start from scratch with a new engine but also wanted to keep things familiar with the CPS4 engine, or maybe figured the way CPS4 did things with stuff like Rno / model parts / Esp was the best way for them, who knows.
IMO it's probably not that helpful for RE4, but anyone working on PG games should find it really useful, wish I knew about this when I was doing work on NA >.>
DoomUsr
commented
2 years ago Hi, its garbage.
i'm cnexabyte(ツ)
(E2: updated with some quick info about PS2 debug build) (E: updated with some things about the PC UHD release too)
Saw there was some recent interest in reversing the console versions, I looked into a bunch of different builds/ports to gather info for re4_tweaks over the past year, so figured a write up of the things learned there might come in useful.
There's a good chance I might have misremembered something here though, if you notice anything wrong feel free to post any corrections 😸, besides that I hope this might help someone out in some way!
(I was mainly working on recovering symbol names at the time too, so this info is mostly focused around symbols, but hopefully should also be able to help show where code for certain things can be found)
Sadly other than these I don't really think we'll get to see much else from the actual RE4 engine for a long while, been teased by a few different groups with the idea of builds that have even more debug-symbols included, but there doesn't seem to be any intention of those being released any time soon ;_;
Oh well, there's still quite a few builds out there publicly that can reveal a lot about the game engine, here's a quick overview of some interesting ones:
Gamecube (debug)
Unlike the PC "Ultimate HD" (UHD) version, the code for maps/characters/weapons/etc are split into separate REL overlays, a little similar to DLL files on Windows, helping to save memory by only loading in code for things that are actually in use.
Bio4.sym file contains symbols for the main .dol executable, SYM files are also included for each of the REL overlays too. Some of the REL overlays can be found in the raw filesystem, but most are stored in DRS container files (usually as the last file inside the DRS)
SYMs are also available for RELs which weren't included on the debug disc 1 ISO, disc 2 was private for a while but thankfully got released eventually, should be able to match it up with those fine (although there are still a few SYM files for things like Em06 that aren't included, might be worth checking the other prototype builds for them...)
RELs also contain unnamed imports from the main .dol, but your REL loader should hopefully expose the address of the import, so it'd be possible to match those addrs against the main Bio4.sym to name them in your disassembly.
SDA & TOC/SDA2 note When loading PowerPC dol/rel files you should also make sure to set the SDA & TOC/SDA2 addresses if your disassembler allows it, to help it to track down more variable usages. In IDA these can be set via "Processor specific analysis options" button on the initial file loading screen.
You can usually find the values of these somewhere near the start/main function of the main executable - r13 holds the SDA value while r2 holds the TOC value.
For the debug build these should be TOC = 0x8032BEE0, SDA = 0x8031BEE0.
PS2 (retail)
It's been known for a long time that the retail PS2 version contains symbols, but the symbols included are very incomplete, and seem to only be for functions that get used by overlay RELs - though these symbols do provide partial prototypes for certain functions, revealing the names of some structures used by the engine.
Similar to GC version REL overlays are used, though these are stored inside the main AFS file now (and also still inside DRS containers too), and the format used is different to DolphinOS, instead using a unique ProDG format used that I couldn't find much info about (ProDG SDKs do contain something similar, but IIRC it wasn't an exact match) - symbols inside the PS2 RELs are usually very poor though, hardly even worth trying to load in, main executable symbols are a lot more interesting.
While some things (graphics/audio related) were rewritten to work on PS2 hardware, the majority seems to be using the same code as GC (though with some enhancements due to the additional scenario that PS2 added)
PS2 (debug) (Aug 27 2005)
A PS2 debug build was finally publicly released in Oct 2023, and I totally missed the release of it, thanks to zatarita for directing me to it.
.mdebug stabs can be converted to objdump compatible split-stabs via https://github.com/uyjulian/stab_debuginfo_utils, objdump can then dump them to C header-like file (might be possible to load in the split-stabs-converted ELF into disassemblers too, but hadn't tried it yet)
I'd been given an elf file from a PS2 debug build in the past (thanks to someone awesome), the .mdebug symbols included structures which filled in a lot of blanks, though without any of the corresponding data files for that build to check against I wasn't sure whether the REL files might have had symbols for the expansion code too. (interestingly, the elf I'd been given before doesn't seem to match this new release, that elf was dated earlier at Jun 2005 while this build is Aug 27 2005, so I guess there's still another PS2 proto out there...)
Sadly in this build at least it seems SNR2 REL files don't have much more than the retail ones did, some symbol names for a few exported major functions, but local functions were left unnamed, so we're still missing a lot of symbols related to expansions - who knows, maybe a proto of one of the 7th gen ports might show up one day with something for them.
Really wish I'd documented the method I used to convert the mdebug symbols & dump them out before (which resulted in a nice chonky ~140MB header file), once I have some time I'll try retracing my steps for this new build and update this post with a method for it, maybe some better way has shown up for mdebug since I looked too.
Oculus Quest (RE4VR)
Despite all the UE4 boilerplate code this is probably one of the best sources for finding UHD info, they did seem to remove code for a lot of things though, but in most cases the removed functions still had a stub function left in the code, giving us the function name and partial prototype for free. (or it was only the call to the function removed, with the actual function still left in place)
Unfortunately a lot of the code that used to be included in REL files doesn't seem to be included in the symbols here, making GC debug the best for completeness, but RE4VR still gives a lot more info via function prototypes.
Seems they might have ran some kind of strip tool on the ELF since there's sadly no DWARF structs or anything like that, could have saved us a lot of work if those were included, ah well.
Bonus: Wii (debug, incomplete)
The PC UHD release actually seems to include REL files for a debug Wii version, likely one of the last versions before they branched off for the X360 port.
Sadly the main executable for that version isn't included so there's not much chance of ever getting to actually boot those files, but the RELs themselves seem pretty interesting, might be easier to match missing RELs from the GC debug to these rather than the GC retail.
UHD does seem to still read these RELs for some reason, removing them seems to cause some strange gameplay effects with certain cutscenes being skipped too - guess there's data being used from the REL somehow, but haven't found where it's read from / what it gets used for yet.
Similar engine: Viewtiful Joe 2 (PS2 prototype)
The .mdebug section contains a huge amount of structures/function names/prototypes, but seems this is based on a very early version of whatever engine RE4 was based on (maybe started with P.N.03?), so a lot of the stuff RE4 makes use of isn't included here, or is included but in a very early state with a lot missing from it.
A dump of the symbols from this build can be found here: https://github.com/emoose/re4-research/issues/3#issuecomment-1061447972
Similar engine: P.N.03 (GC prototype)
Maybe earliest form of the engine used by RE4, not hugely useful because of how early everything is, but can give some idea about certain things used by the GC debug version which didn't have much information available (eg. the sound engine)
IIRC the MAP symbols didn't actually match up to the main executable though, but .SYM should match fine, the MAP is still useful for function prototypes.
PC UHD release
It's probably worth mentioning what I know about the UHD release here too, this looks to be based on the X360 port of the game, which itself seems to be based on the original Wii release, AFAIK QLOC mainly worked on the PC-specific enhancements - doesn't really seem that anything was shared with the older 2007 PC Ubisoft port.
The PC version was compiled with VS2010 (likely because that was the last version that the X360 SDK supports, which the PC ver. still seems to tap into for compression code), and all builds I've looked through seem to be incremental, meaning most functions are called through small thunks to make incremental linking easier - this does carry a small performance penalty AFAIK, but also makes hooking things for re4_tweaks a lot easier too, so I won't complain about it much.
What I'd like to complain about is how almost all floating-point-related code makes use of the older x87 instruction set (though there are some small sections that make use of SSE as well), from some tests to try generating similar code I found that this seems to be default behaviour in VS2010, where project settings need to be explicitly set to generate SSE/AVX code - a shame they never changed this themselves, since even the ancient SSE1 which all post-2000 PCs include can give a performance boost in vector/matrix related code in the game. (re4_tweaks allows replacing some of this code with SSE-enabled equivalents, but only affects some math related functions and not the entire game code, so the improvement is only minor)
This port seems to make use of a GX emulation layer that was introduced with the X360 release, which exposes itself to the game code with the same API as Gamecube/Wii's GX API, but then translates those calls over to DX9, pretty neat stuff, though sadly seems the emulation might not be that accurate since certain GC effects (Filter03...) don't seem to work properly after being re-enabled, even when using code recreated from the GC build.
For some reason certain graphical effects were left disabled/incomplete during the X360 port, and so also were left disabled for the PC release too - re4_tweaks tries filling in the gaps based on the GC version to re-enable these effects, which works well for some (the transparent item pickup screen overlay) and not so great on others (the low-resolution depth-of-field effect) - in both cases the cause of those being missing seemed to be pretty much a single line/block being absent in the code, maybe something that was commented out during development but forgot to be re-enabled.
This port also added a 60FPS mode to the game (which originally only ran at 30FPS on all earlier releases), this seems to work by allowing the game code to run twice as fast (for it to achieve 60 frames/second), but with an internal game-speed variable set to 1/2 - in the original releases not many things actually respected this game-speed variable though, so a lot of things had to be updated to make use of it in order for everything to run at the normal speed. Of course from the end result we can see that many spots were accidentally left out of these game-speed changes, resulting in things like falling items/water animations/others to run at double speed - fortunately re4_tweaks has been able to fix most of these left-out spots.
Many in the RE4 community don't seem to think too highly about this port, besides the much higher system requirements compared to 2007 it's also been plagued by issues since release, and Capcom only seems to have supported it for a brief few months before giving up on it - though the game did receive an update many years after initial release to add Chinese localisation support.
(A debug version was also accidentally released during that Chinese localisation work, but sadly is nowhere near as fully featured as the earlier GC debug version, almost seems to just be a Release build with a couple debug toggles enabled, not too incredibly useful - it did at least help to show us what was needed to re-enable certain debug menus in the normal non-debug builds.)
Conclusion
The engine was used for quite a few early-00 Capcom titles, starting with P.N.03 as the earliest game I could find (2003), lasting over ~5 years up until Clover Studio was closed down & Capcom focused on MT Framework instead (even then, some MT Framework games do seem to include some tiny pieces of the RE4 engine inside them), of course later ports of RE4 would allow the engine to live on even longer.
Strangely there do seem to be parts that make use of re-implemented PS1 SDK functions, while that could imply the engine was started for PS1 games I don't really see that being the case, more likely some small component used in the game was ported from a PS1 title and they didn't want to recode it all to work again, easier to just emulate the few PS1 functions it uses instead - many years later this was also the same route used by the HD edition team to emulate GX API calls too.
There's a good chance I've left out some other offshoots of the RE4 engine here, maybe another game similar to Viewtiful Joe will come to light eventually with even more information, until then I guess we're stuck manually reversing the rest out of what we have.
Spent nearly a year working on an IDA database for the UHD version, including trying to recreate structs, still a lot more work to be done though: https://github.com/emoose/re4-research/issues/3