This PR is primarily motivated by https://github.com/itb-community/ftldat-rs/issues/1, which (among other things) asked for ftldat-rs to be published as a standalone package that other Rust projects could interact with.
This represented an opportunity to reduce duplication in both ftldat-rs and itb-io projects, since both of those projects had to include their own copies of files that allowed them to export Lua bindings.
While preparing ftldat-rs for publishing, I also overhauled the way it loads files. During reading, files are now memory-mapped, rather than read into a buffer. Also, the in-memory representation of the .dat file now only stores a list of entry headers, rather than their full contents, which dramatically reduces memory usage. In practice, there aren't any noticeable speed improvements, but ITB's memory usage no longer spikes into over 350MB range during startup.
The fact that the archive file is now memory-mapped means that is important to properly destroy each FtlDat instance (by invoking FtlDat:destroy), as it actually holds system resources now.
Since files cannot be overwritten while there exists a lock on them, we cannot overwrite resources.dat while an FtlDat instance exists. To resolve this issue, when writing to a file, FtlDat instances first write to a temporary file, release system resources, and then replace the destination file with the temporary file. This means that once the FtlDat:write method is invoked, that instance becomes invalid and cannot be used anymore.
This also means that when there are multiple FtlDat instances created from the same source file at once, it is impossible to overwrite their source file. In practice, this shouldn't be an issue, as the mod loader is the only client of FtlDat, and provides hooks that mods use to apply their desired changes.
This PR is primarily motivated by https://github.com/itb-community/ftldat-rs/issues/1, which (among other things) asked for
ftldat-rsto be published as a standalone package that other Rust projects could interact with.This represented an opportunity to reduce duplication in both
ftldat-rsanditb-ioprojects, since both of those projects had to include their own copies of files that allowed them to export Lua bindings.As a result, I created https://github.com/itb-community/itb-rs-lua, which includes
ftldat-rsas a dependency (crate), and fully incorporates https://github.com/itb-community/itb-io-rs, with an improved project structure that should be easier to extend in the future.Changes in file sizes:
While preparing
ftldat-rsfor publishing, I also overhauled the way it loads files. During reading, files are now memory-mapped, rather than read into a buffer. Also, the in-memory representation of the.datfile now only stores a list of entry headers, rather than their full contents, which dramatically reduces memory usage. In practice, there aren't any noticeable speed improvements, but ITB's memory usage no longer spikes into over 350MB range during startup.The fact that the archive file is now memory-mapped means that is important to properly destroy each
FtlDatinstance (by invokingFtlDat:destroy), as it actually holds system resources now.Since files cannot be overwritten while there exists a lock on them, we cannot overwrite
resources.datwhile anFtlDatinstance exists. To resolve this issue, when writing to a file,FtlDatinstances first write to a temporary file, release system resources, and then replace the destination file with the temporary file. This means that once theFtlDat:writemethod is invoked, that instance becomes invalid and cannot be used anymore. This also means that when there are multipleFtlDatinstances created from the same source file at once, it is impossible to overwrite their source file. In practice, this shouldn't be an issue, as the mod loader is the only client ofFtlDat, and provides hooks that mods use to apply their desired changes.