Game building tools (master ticket)

This is a master ticket for tasks related to decoupling game compilation from the Editor. The final goal is to have a minimal set of standalone tools enough for compiling a game data package from the game project sources. The previous ticket on this topic: #872, it contains a discussion and records first steps in this direction. The general consensus was to write standalone command-line programs in C++, as that's something that may be compiled and run almost everywhere without big dependencies, and may reuse parts of the existing engine's code. There have been a number of tools created since (see Tools dir in our repo), pretty much enough for compiling the game scripts, but not covering all required tasks for creating game data. Several tools more have to be done, and some of the existing ones may need adjustments.

Please note that this ticket is dedicated to writing tools, not replacing a build process in the Editor. That should be a separate task. In theory, replacement may be done in steps too, so long as that is convenient; but that's a topic for another discussion.

All the sub-tasks for this goal are assigned to a "Minimal standalone tool set" milestone. There's also a github "project" created, as an experiment for tracking the related tasks: https://github.com/adventuregamestudio/ags/projects/1 A wiki page I wrote, with a table of all the build steps, input and output data, which may help understanding the building process: https://github.com/adventuregamestudio/ags/wiki/AGS-Game-Build-process-(3.5.*)

Above explains ags3 workflow, ags4 may have certain changes, primarily due to change in room format.

Following are lists of required tasks, separated into categories.

Priority tools (required for both ags3 and ags4)

These are must have for building a game from a source.

[ ] Build main game data from Game.agf (#1188). Must parse Game.agf, create game structs in memory, and write game28.dta.
[ ] Export lists of game contents necessary for running script compiler and game packer (ticket TBD). Must parse Game.agf, and create simple text files with lists of stuff: scripts to compile, files to gather into package, etc. Something that may be used by any external script (bash, py, make, etc) which runs whole process.
[ ] (OPTIONAL) Merge tools that export generated scripts from Game.agf (ticket TBD). These are tools named agf2autoash and agf2glvar, I did them two separate programs, but I think it's more convenient to have one that does one of or both things depending on command args. Because they both do thematically same thing: translate game contents to script as required for the scripts to compile.
[ ] Export script headers from the engine plugins (ticket TBD). This tool must load the engine plugin up, pretending to be an IDE, and query it for the script header, then write one into a file. Note that such tool is also required if making a 32-bit game build in a 64-bit editor, or vice-versa, because plugins cannot be loaded by editor program of different bitness.

Secondary tools

These are tools that may be skipped when just building the game from "full" project state, but they are still required in order to get the project into this "full" state.

[ ] Spritefile builder (ticket TBD). The purpose of the tool is to parse Game.agf for a list of sprites, generate sprites from their source files according to their import settings, and write a new spritefile (acsprset.spr) and optionally a sprite index (sprindex.dat). Note that for this tool we need a image loading library in C++, as source files may come in various formats (BMP and PNG at minimum, but ideally everything that AGS Editor supported).
[ ] Default config writer (ticket TBD). The purpose is to parse Game.agf for a component called "Default Setup" and generate default game config file (acsetup.cfg).

Tools specific for ags4

[ ] Room file compiler (ticket TBD). Must parse room's data.xml and compile room.crm out of it, packing all room assets together (backgrounds, masks, compiled room script). This has to be thought through first, as there may be few more intermediate steps. For example, room bgs and masks may be stored as PNGs, but has to be converted to raw bitmap data and compressed using LZW. Which means there has to be a PNG loading code in C++. Maybe we could make a separate dedicated tool for working with images, as we also might use one for compiling spritefile (see above); or maybe linking image-read/write functions to a room-compiling tool will be okay. Also note that since 3.6.0 we do not have to package compiled room scripts inside the room.crm, and may place them as separate assets insides (roomN.o). I suggest to use this option, as that will simplify the room compiling tool.
[ ] Translation compiler from PO source (ticket TBD). We have a translation compiler ("trac") which supports old-style TRS files, but ags4 now writes TRS in PO format. Regardless of whether we keep original "trac" in ags4 branch or not, I suggest to give new tool a new name (e.g. "trac2" or "tracpo") so to distinguish them inside a collection of all tools.

Above lists may be amended if more tasks are required.

Auxiliary tasks

There's an issue of extra dependencies of game structs declared in Common dir. These structs, and their serialization functions presumably should be reused by the tools in order to avoid code duplication. But there's a number of things these structs are connected to, which ideally should not be linked in tools.

First of all, this is Bitmap class, that is used for drawing, and which brings a large part of Allegro graphics library. There are two ways of dealing with this (that I see):

[DONE in #2435] Introduce a simpler struct that would hold image metrics and pixel data, but won't have any drawing methods. E.g. "BitmapData". Use this struct in image loading/saving functions. Make it so that Bitmap class is constructed by either copying pixel data from BitmapData, or wraps it; in the latter case Allegro's BITMAP struct should be initialized by us, having its pixel array pointers directed to BitmapData's contents.
[NOT NEEDED ANYMORE?] Write a separate variant of Bitmap class which drawing methods do nothing, and include & link it in programs that do not need full drawing support.

Then, some classes, such as GUIs, have methods that are meant for runtime: drawing and update. These will be useless in tools, and again bring more dependencies. Possibly, we could have structs containing only GUI and GUI control's serialized fields, as suggested in this draft on engine refactor: https://github.com/adventuregamestudio/ags/wiki/AGS-4-Draft:-major-engine-refactor#game-data-classes

For related problems see also: #1833, #1834, #2058.

adventuregamestudio / ags