Refactor SpecialLumpClasses

[snake-biscuits]

The current way SpecialLumpClasses are loaded from .bsps involves feeding the raw lump bytes to a function However, in the code we thing of SpecialLumpClasses as classes, but LumpClasses have .from_bytes an alternate __init__ This also messes with generating documentation for SpecialLumpClasses with a fixed structure (e.g. 1x struct LevelInfo)

It would be more useful to think of SpecialLumpClasses as a single, often dynamically sized, structure We already refactored LumpClasses to have their own __init__ methods for easier use, but SpecialLumpClasses must be fed bytes

Creating a blank of any lump type & populating it after is far more useful, and doesn't require memorising stubs

Required Changes:

• SpecialLumpClass __init__ -> from_bytes @classmethod for each SpecialLumpClass:

  @classmethod
  def from_bytes(cls, raw_lump: bytes) -> SpecialLumpClass:
      ...  # old __init__

• All unique _preload_lump BspClass methods need to calling SpecialLumpClass .from_bytes(), rather than __init__
  • Same for ExternalLumpManager.__getattr__
  • And RespawnBsp .ent (shared.Entities)
• Replacement __init__ methods for each SpecialLumpClass (if nessecary) Calling with no arguments should create a valid "empty lump" super.__init__() should be good enough for list subclasses, but test that theory just in case

All Special Lump Classes:

import functools
import bsp_tool

all_branch_scripts = functools.reduce(lambda a, b: set(a).union(set(b)), bsp_tool.branches.scripts_from_file_magic.values())
source_slcs = {slc for d in [d for bs in all_branch_scripts for d in bs.SPECIAL_LUMP_CLASSES.values() if "version" in bs.LumpHeader._mapping] for slc in d.values()}
quake_slcs = {slc for d in [d for bs in all_branch_scripts for d in bs.SPECIAL_LUMP_CLASSES.values() if "version" not in bs.LumpHeader._mapping] for slc in d.values()}
all_slcs = source_slcs.union(quake_slcs)

{print(x) for x in sorted([slc.__name__ if slc.__name__ != "from_bytes" else slc.__self__.__name__ for slc in all_slcs])}

NOTE: all scripts are in bsp_tool.branches

• [x] id_software.quake.MipTextureLump
• [x] id_software.quake2.Visibility
• [x] id_software.quake3.Visibility
• [x] respawn.apex_legends.LevelInfo
• [x] respawn.titanfall.EntityPartitions
• [x] respawn.titanfall.Grid
  • [x] respawn.titanfall_x360.Grid_x360
• [x] respawn.titanfall.LevelInfo
• [x] shared.Entities
• [x] shared.PakFile
• [x] shared.TextureDataStringData
• [x] valve_physics.CollideLump
• [x] valve_physics.Displacement
  • [x] strata,strata.PhysicsDisplacement

Phase 2: Game Lump Classes

We need to do the same conversion for the following GameLump & Lump Classes:

• [x] valve.source.GameLump_SPRP
  • [x] valve.source.StaticPropv4 -> source.SPRPv4
  • [x] valve.source.StaticPropv5 -> source.SPRPv5
  • [x] valve.source.StaticPropv6 -> source.SPRPv6
  • [x] valve.left4dead.StaticPropv8 -> left4dead.SPRPv8
  • [x] valve.left4dead2.StaticPropv9 -> left4dead2.SPRPv9
  • [x] valve.orange_box.StaticPropv10 -> orange_box.SPRPv10
  • [x] valve.orange_box_x360.StaticPropv4_x360 -> orange_box_x360.SPRPv4_x360
  • [x] valve.orange_box_x360.StaticPropv5_x360 -> orange_box_x360.SPRPv5_x360
  • [x] valve.orange_box_x360.StaticPropv6_x360 -> orange_box_x360.SPRPv6_x360
  • [x] valve.orange_box_x360.StaticPropv10_x360 -> orange_box_x360.SPRPv10_x360
  • [x] valve.sdk_2013.StaticPropv10 -> sdk_2013.SPRPv10
  • [x] valve.sdk_2013.StaticPropv11 -> sdk_2013.SPRPv11
  • [x] valve.sdk_2013_x360.StaticPropv8_x360 -> sdk_2013_x360.SPRPv8_x360
  • [x] valve.sdk_2013_x360.StaticPropv9_x360 -> sdk_2013_x360.SPRPv9_x360
  • [x] valve.sdk_2013_x360.StaticPropv10_x360 -> sdk_2013_x360.SPRPv10_x360
  • [x] valve.sdk_2013_x360.StaticPropv11_x360 -> sdk_2013_x360.SPRPv11_x360
• [x] nexon.vindictus.GameLump_SPRP
  • [x] valve.source.StaticPropv6 & nexon.vindictus.StaticPropScale -> vindictus.SPRPv6
• [x] respawn.titanfall.GameLump_SPRP
  • [x] respawn.titanfall.StaticPropv12 -> titanfall.SPRPv12
• [x] respawn.titanfall_x360.GameLump_SPRP_x360
  • [x] respawn.titanfall_x360.StaticPropv12_x360 -> titanfall_x360.SPRPv12_x360
• [x] respawn.titanfall2.GameLump_SPRP
  • [x] respawn.titanfall2.StaticPropv13 -> titanfall2.SPRPv13

TODO: do we align GameLumps closer to lumps.BspLump?

• [x] Refactor all GameLump classes to work the same as SpecialLumpClasses
  • [x] __init__ creates an empty lump for population
  • [x] .from_bytes() @classmethod replaces old __init__
  • [x] StaticPropClass is part of class definition (similar to lumps.BspLump)
• [x] .from_bytes() should map a lumps.BspLump to reduce memory use (see #80)
• [x] Use GameLump subclasses, instead of lambdas wrapping __init__
• [x] Verify StaticPropClass in .as_bytes()
• [ ] Use a bytearray for StaticPropClass if undefined (derive length from prop_count)

[snake-biscuits]

respawn.titanfall.Grid & LevelInfo (and their children) are both base.MappedArray/Struct already. Their lumps are one occurrence of each struct. All SpecialLumpClasses should be treated like this, even though most are dynamically sized Same goes for GameLumpClasses.

__init__ with no args should create a valid "empty" lump on as_bytes() call What args specifically to use for __init__ will vary depending on the lump (user-only interface) It'd also be neat to implement a common __eq__ that compares Lump.as_bytes

if bsp.SPECIAL_LUMP == bsp.branch.SpecialLump():
    ... # lump is basically empty

[snake-biscuits]

Broke a lot of stuff doing this refactor today Need to run a MegaTest & some casual use before closing this issue

Also haven't done much in the way of docs Autogenerated docs can do without the special cases for from_bytes methods The new __init__ methods for list based SpecialLumpClasses have type hints at least.

[snake-biscuits]

We should also be testing all SpecialLumpClass __init__ & as_bytes methods

16 should take care of from_bytes tests (so long as test maps represent all mapped lumps)

[snake-biscuits]

GameLumpClasses (GameLump_SPRP) should be counted as SpecialLumpClasses

GameLumpClasses also need a refactor (for apex_legends at least) to hook lumps.BspLump for dynamic prop loading

Should address the absurd RAM use present in #80

[snake-biscuits]

.from_bytes() should map a lumps.BspLump to reduce memory use

This means we'll be resolving #80 as part of this issue

Allow automatic conversions; might need to remap some member names

Ties into #73

[snake-biscuits]

Also haven't done much in the way of docs Autogenerated docs can do without the special cases for from_bytes methods The new __init__ methods for list based SpecialLumpClasses have type hints at least.

When closing this issue, we need to start putting together docs for how to use the new __init__ methods when The ideal use case would be creating a .bsp w/ bsp_tool (e.g. compiling, cross-branch conversion) Haven't tried anything like that yet though, so we'd have to prototype & try a test case

[snake-biscuits]

A .from_stream(stream: io.BytesIO) alternate __init__ would actually make a lot of sense iirc every .from_bytes(raw_lump: bytes) creates a bytestream internally

We could just wrap .from_stream when we call .from_bytes Though we do make assumptions about the stream ending with the lump Limiting streams to lump size is nice for security too... (not to mention checking bounds errors)

I'm not even sure it's possible to determine some special lump lengths without the header

Could be relevant to #21

[snake-biscuits]

Use a bytearray for StaticPropClass if undefined (derive length from prop_count)

The new GameLump_SPRP are so nice & tidy now that I don't want to mess them up w/ this idea Plus it's not too hard to whip up a quick MappedArray to start building a new StaticPropClass

A bytearray utility class w/ hex editor views etc. for whipping up a quick MappedArray could be cool tho Could come in handy for reversing infinity_ward.modern_warfare structs etc.