Create bsp class from bytes / stream

[snake-biscuits]

All functions / classes that return a base.Bsp subclass should accept either an open binary file or bytestring in the place of a filename, this would allow for loading .bsps from within archives without unpacking

And other sources I'm sure

as part of such a system a path handler for indexing files inside archives would be needed nice to have

“game/archive.ext/maps/target.bsp” ->
archive = game/archive.ext
bsp = bsp_tool.load_bsp(archive.read(“maps/target.bsp”))

Changes would also have to be made to how bsps record and relate to their “folder” & “filename”

Opening an archive only once and scanning it for .bsp files would also be useful for tests

Good luck writing back to the archive, unless you want to add a Read-Only mode...

[snake-biscuits]

Some archives are nested, for example: mappack.7z/surf.rar/maps/surf_exquisite.zip/Fortress Forever/custom/mymod/maps/surf_exquisite.bsp.bz2 have fun walking those, will definitely want to implicitly tell the lump system to just keep the whole file in memory

oh and don't forget the .cdi / .iso / .cue + .bin format for Quake III on the Dreamcast w/ nested .zips

We could drastically simplify building the MegaTest library if we walked these paths, at the cost of execution time

[snake-biscuits]

Indexing into Dreamcast & Xbox360 disc images could get hairy Quake III (Dreamcast): quakeiii.cdi/quakeiii.16.iso/QUAKE3/BASEQ3/MAPS/DC_MAP01/BSP.ZIP/maps/dc_map01.bsp Three nested unique archive formats! (.cdi, .iso & .zip)

Orange Box (Xbox360): HalfLife2_TheOrangeBox.iso/ Source Engine Xbox 360 .iso XISO files are different to regular .iso, though can be interpreted as such Assets (models, textures, audio etc.) are stored in a special .zip format for Little-Endian Maps are just in the XISO filesystem

[snake-biscuits]

Good luck writing back to the archive, unless you want to add a Read-Only mode...

Read-Only mode could be really powerful for faster tests!

It turns out that part of the RAM & time consumption of reading Apex Legends .bsp files comes from how python handles files file.close() has to flush a buffer the size of the whole file, which is ridiculous for Apex Legends ~400 MB .bsp Especially when you consider that .bsp_lump files roughly double that size, making a single map close to 1GB!

Unsure if using the with keyword is any faster But if a read-only mode speeds up the MegaTest it'd be 100% worth the effort

[snake-biscuits]

Look into using python's builtin bytearray (or bytearray-like behaviour) for RawBspLumps The mmap module also looks convenient for handling files

TODO: test mmap read & close speeds on large files

Faking in-memory files w/ the io module is also pretty handy

Python file read costs in general could use more research, but also testing different approaches for read, write & close speeds

[snake-biscuits]

please

[snake-biscuits]

This ended up being way easier to implement than I expected

>>> from bsp_tool.id_software import QuakeBsp
>>> from bsp_tool.branches.id_software import quake
>>> QuakeBsp.from_file(quake, "E:/Mod/Quake/Id1/pak0/maps/e1m1.bsp")
<QuakeBsp 'e1m1.bsp' id_software.quake (version 29)>
>>> from bsp_tool.extensions.archives.id_software import Pak
>>> pak0 = Pak("D:/SteamLibrary/steamapps/common/Quake/Id1/PAK0.PAK")
>>> import io
>>> QuakeBsp.from_stream(quake, "pak0.pak/maps/e1m1.bsp", io.BytesIO(pak0.read("maps/e1m1.bsp")))
<QuakeBsp 'e1m1.bsp' id_software.quake (version 29)>

Still running tests to see if anything broke yet but it's a promising start