Feature: Modding files in nested archives

[nex3]

ME1 and ME2 (through the built-in mod loader extension) worked by allowing mods to provide a kind of overlay filesystem where the modded games would check paths within that filesystem in preference to the game's own internal virtual filesystem. This provides the valuable ability to swap out individual resources without needing to directly patch the game executable.

However, many of the files used by these games are themselves archives containing multiple files. Some of these archives can be quite large—for example, in DS3 menu/01_common.tpf.dcx is almost 27 megabytes, which means that if a mod wants to add even one new item with a new icon the mod itself has to be at least that large. I suspect it would be substantially more complex than just intercepting virtual filesystem paths, but if it were possible for a mod to add mod/menu/01_common.tpf.dcx/MENU_Icon_03001.dds or whatever as an individual file that the mod loader overlays onto the game's data automatically, I think it would be hugely valuable for mod authors.

[garyttierney]

cc @vswarte, any of the VFS code you've looked into recently deal with this or does it sit at a higher layer?

[vswarte]

Def at a high layer. Earliest I've seen individual resources (like the FLVER from a chrbnd or an FXR from an sfxbnd) is in FileCap impls. FileCaps only momentarily contain a pointer to the resource's bytes and pretty much immediately create ResCaps from the bytes, after which the FileCap usually pretty much immediately yields the files bytes back for dealloc. The parsing and ResCap creation happens at FileCap::vftable[10] (I checked with ER 1.13.0).

I think what is being asked here is not that hard. My biggest concern is figuring out what bnd/tpf something comes from when hooking ResCap creation.

[vswarte]

Also I didn't look too deep into this, maybe there's a simpler hook we can think of if we look into it a bit more.

[garyttierney]

Maybe a wild but potentially simple idea:

• Replace all I/O done by the game with an fstools backed DvdBnd reader
• Transparently repack BNDs as they're requested by the game
• Perhaps use some sort of derivative cache to avoid the overhead of doing this (if it's significant)

This working well hangs on a few assumptions:

• It's trivial to prevent the game from doing any IO at all. fstools will want to mmap archives in.
• The game will accept uncompressed archives where it previously loaded compressed ones, otherwise we decompress in fstools, recompress in me3, and decompress in the game.
• The game can handle loading BNDs from an incomplete stream. We don't want to keep two in-memory copies for every file load request.