Decoupling the game module build tools from the editor

[edmundito]

Hello, here's another side project I started after talking and interacting with the number of AGS users who create commercial games:

• Separate the functionality of loading and compiling the game (I will call this the build tool) away from the editor in order to provide more power to users who develop commercial games and modules.

Rationale

1. If the functionality to build a game or a module is decoupled from the editor, you can then create a command line interface (CLI) for AGS that can open and build the game project
2. With the AGS CLI build a game via command line, it can enable AGS users to generate a build via the build server
3. Developers can build modules via the command line without having to manually do all the export steps
4. Decoupling the builder from the UI also allows to make some improvements in how the editor builds, and improve the UI experience without reworking the compilation steps.
5. Using the .NET core (https://dotnet.microsoft.com/download), you may also be able to provide the build tool for other platforms such as Linux and Mac
6. With the build tool becoming available in multiple platforms it may make the process of building the game in multiple platforms easier

Work in Progress

I started pushing some changes into my fork. So far it doesn't seem very difficult other than time consuming and at least the first steps give me an idea of what areas to separate and where some refactoring may be needed. The primary bulk of the work is decoupling UI calls from the builds steps.

https://github.com/adventuregamestudio/ags/compare/release-3.5.0...edmundito:edmundito/decouple-compiler (should probably rename to decouple-builder)

With the work so far, here's what I'm thinking:

1. Right now we have AGS.Types and AGSEditor solutions
2. I would introduce new solution that is used by AGSEditor (AGS.Core?) that is able to open the game and manage the native compiler to create an executable (more or less what the editor is currently doing)
3. AGS.Core can be then used by AGSEditor or other projects (AGSBuilder?)

I am seeking some feedback and to talk about next steps. Send your thoughts in the comments 👇

[morganwillcock]

I'm not sure why there are two script compilers, or what was wrong with the C# one, but I think the only one that is known to work, has tests, and is actively being refactored in a PR, is the C++ one. I think (just because I stumbled across this part recently, but I've not looked too closely) that the broader assembly of all of the parts when they are joined together is done in C#.

So maybe this is less straight-forward that it first appears, but I also think it is absolutey the right thing to do. I'd suggest that it should also be expanded to handle resource packing (i.e. build the sprite cache and vox files for distribution). I also don't see any long term future for the Editor as it is works now and using a custom plug-in for Atom or VSCode, plus a separate compiler, looks a lot more suitainable. I guess I'm saying, portablility and integration with something else is probably more important that how it integrates with the existing Editor. I would like to say I'd meet you in the middle but it is difficult for me to commit when issues around the project's overall goal and how it operates in general remain unresolved.

[rofl0r]

I'd suggest that it should also be expanded to handle resource packing (i.e. build the sprite cache and vox files for distribution).

the command line tools of agsutils can already do that

I think the only one that is known to work, has tests, and is actively being refactored in a PR, is the C++ one

it would certainly be preferable to have a stand-alone compiler written in C++ than in C#. so linux users can use it too without messing with mono. it could even emit assembly in an agsutils-compatible form, so the compiler could leave the work of building export/import/data sections, as well as fixups to the assembler of agsutils which already does all that.

[sonneveld]

the command line tools of agsutils can already do that

True but it shouldn't be too hard to port the existing C# code to C++.

[ericoporto]

I started learning dotnet core last night, it can do C# without mono @rofl0r :) and it builds binaries with no dependencies apparently - haven't boot a clean machine to test. Anyhow, if you don't know, libvirt, lots of cool things using it, I particularly use lxd.

[rofl0r]

i tried to get both mono and dotnet core to compile on my linux distribution, but both are a pain in the ass with dozens of dependencies and 100's of MB of space usage (and full of unportable assumption, i.e. designed for GLIBC). it's kinda like installing a second OS inside your primary OS. no thanks.

[sonneveld]

it's kinda like installing a second OS inside your primary OS. no thanks.

I get that you have a reluctance to use anything more than make and a C compiler. And your OS requires compilation of everything. But this is primarily a C#/C++ project (with a sprinkle of python scripts). How far should we accommodate you?

[fernewelten]

As far as the "C# compiler" is concerned, the point is moot IMO. Last time I looked, there wasn't any C# compiler to speak of.

There's code for pre-processing the sources, and that code is being used indeed, but all the scanning, tokenizing, and parsing code is from non-existent to rudimentary and not used.

I'm currently rewriting the C++ compiler, although I took a break last month in order to participate in the MAGS June competition. I'm working within AGS 4, but the compiler is stand-alone, so once it does work it could be integrated into 3.5 fairly easily.

[rofl0r]

How far should we accommodate you?

it's not only about me. it's a hassle for all linux (or mac, for that matter) users having to install a huge package like mono to run .net stuff. but for those compiling from source, e.g. gentoo users, it's even more annoying.

I'm currently rewriting the C++ compiler ... the compiler is stand-alone

oh? nice. it could be added as a subproject and then editor code could simply be changed to run the external compiler and the old compiler phased out. having it outside of the main AGS source tree would also relief us from the refactoring spam in your previous PR (which tbh, would be kind crazy if it was merged due to the sheer amount of work it needs to review).

[sonneveld]

it's not only about me. it's a hassle for all linux (or mac, for that matter) users having to install a huge package like mono to run .net stuff.

This isn't strictly a Linux or Mac issue, Windows also requires the .net runtime. From what @ericoporto is saying, you can produce standalone binaries which don't require a separate runtime at all. For reference, the .net core binaries are less than 30 meg. (although I'm not sure if that includes enough for gui support)

[edmundito]

I got a few hints that the compiler itself was native and after digging a little deeper in the code I realized that the Compiler C# solution is only used to provide a few types to the Editor. My goal here to use whatever compiler is available, which I now realize is the C-based compiler that the editor invokes from the Native DLL.

Primarily what I'm trying to do here is separate the logic of the editor that can load and build a game project. I updated the top post to make this more clear.

[rofl0r]

i ripped off the script compiler code from ags repo and made it available as a standalone command line tool: https://github.com/rofl0r/ascc only C++ compiler and GNU make required to build.

[ericoporto]

That's very interesting, how does one runs the generated binary file? Also, what your thoughts in making it in two pieces, a library (.so) that is as close to be compatible to the original compiler, and the command line executable (that depends of this library). This would enable writing frontend for the compiler. With the proper ifdefs, it would be code that could still reside in the AGS repository (that has the C# front end of sorts).

[rofl0r]

That's very interesting, how does one runs the generated binary file?

the compiler creates only the binary script, which by itself cannot be run (except in agssim - with some restrictions) - it needs to be injected with agsinject into an existing room file (roomX.crm) or game data file (ac2game.dta). this can be easily scripted though.

Also, what your thoughts in making it in two pieces, a library (.so) that is as close to be compatible to the original compiler, and the command line executable (that depends of this library).

it's simply a matter of taking all object files except main.o and putting it into a .so file (g++ *.o -shared -o libascc.so) and then using the same functions as in main.c to access it. but then, you could just as well link it directly to your own frontend.

With the proper ifdefs, it would be code that could still reside in the AGS repository (that has the C# front end of sorts).

that would be possible, yeah. there are not many changes against the vanilla upstream sources. of course it would take a lot longer to compile if it uses the "full" upstream sources.

[ericoporto]

So, this is just two ideas, first, if this can be adapted to be PR'd back in this repository, it would be in sync with the AGS compiler, and it could benefit from @fernewelten refactoring once he is done with it.

The thing about the frontend and building it as a library, is, because this code doesn't depend on anything, it would not be hard to turn it in JS using emscripten, imagining an usage of creating a webpage that explains AGS Scripting using this (and agssim).

@rofl0r can you remove the license from agssim and just use the regular license from ags instead? The no C++ clause turns it really hard to reuse.

Ah, I like to write plug-ins, and they usually pack along a header that is used to link their functions to the plug-in functions at build time by the Editor in some way, if you have ideas, I could change it so it would be readable by this in some way, for now this header is if def to be windows only.

[rofl0r]

So, this is just two ideas

oh, sorry. i didn't see your post in notifications for some reason. trying to answer now...

if this can be adapted to be PR'd back in this repository, it would be in sync with the AGS compiler

yeah, as i said before, the sources are almost identical. it's easy to run a diff on the files and import the patches to ags repo.

and it could benefit from @fernewelten refactoring once he is done with it.

tbh, i don't think the refactoring will be beneficial. the current script compiler is stable and well-tested, and the refactoring is IIRC already at 20 KLOC changed (which will certainly create new bugs), while providing no advantages over the current version that i can see. @fernewelten even goes to great lengths to ensure his refactored compiler creates the identical, utterly inefficient code... imo the only major improvement to the script compiler (apart from untying it from the editor) is a complete rewrite by someone very familiar with how compilers work, with state-of-the-art optimization passes added. but this is unlikely to happen. also the bytecode of ags is really inefficient as well. for example, every instruction "prefix" is saved in a 32bit, variable, even though there are less than 200 of them. there's also a lack of general purpose registers (even though a full 32 bit number is wasted for every register number in the bytecode), and the incapability of all registers expect mar to access memory results in at least 3 instructions dispatches for every memory access. a comparison of a register with an integer literal takes 3 dispatches too, since there are no instructions like "branch to X if R < NUM". i could go on and on. maybe ags4 should instead of focus on getting lua or another well-tested, fast VM+language embedded than trying to polish this heritage.

The thing about the frontend and building it as a library, is, because this code doesn't depend on anything, it would not be hard to turn it in JS using emscripten, imagining an usage of creating a webpage that explains AGS Scripting using this (and agssim).

even though i dislike bloated JS web stuff, this actually sounds quite appealing.

@rofl0r can you remove the license from agssim and just use the regular license from ags instead? The no C++ clause turns it really hard to reuse.

why does the license make it hard to use ? it explicitly allows linkage against C++ code, and the fact that it's written in C, which is a much simpler language than C++, actually makes it easier to use.

Ah, I like to write plug-ins, and they usually pack along a header that is used to link their functions to the plug-in functions at build time by the Editor in some way

yeah, i've seen there are headers lying around in the ags plugin dir, and they seem to be automatically preprocessed by the script compiler when compiling a file. i didn't look in detail how the linking mechanism works yet. i personally dislike the existing plugin system, because 1) i don't want to run other people's binaries, 2) it's tied to a specific platform, and 3) many game authors only care about windows. rather, game authors that need extended functionality should cooperate with AGS maintainers to get the functionality into upstream. another good solution to this could be to ship plugins as portable C code, which is then compiled on the fly and JIT'ed at runtime using something like http://github.com/vnmakarov/mir .

btw, i made some exciting improvements. for example, support for char* pointers, use of custom preprocessor, specifying the .ash headers to include, etc...

https://github.com/rofl0r/ascc/commit/3ea0637a5b027d346f482cb573e737353a761c4a

[ericoporto]

@rofl0r I literally make the plug-ins I write multiplatform and open source. You can check all of them, you could compile them yourself - but they are built on a continuous integration system and uploaded to GitHub Releases directly. The basic idea is have plug-ins with functionality that is not available on AGS itself, and the idea is, if people actually use the thing, then think on how to put the functionality in, but the main focus be, to not bloat the AGS source code. So far no one that's not me has used anything, so there has been no reason to PR anything.

I think you need to know, 1, I find C not very expressive, so I could not understand what the commit meant, 2, again, the thing I am asking is an interface in the compiler that could pull the header from the plug-in, now this interface doesn't exist, they have been made for the AGS Editor only which have been ifdef out of the build, so if you could, evaluate if the current interface available on Windows should be ifdef in the Linux build too, or if a new interface is needed to retrieve the header.

[rofl0r]

i managed to do a benchmarks of the engine using the factored out compiler.

this well-known benchmark program here (taken from shootout benchmark game):

#define MAX_N 64

int max(int a, int b)
{
    if (a > b) return a;
    return b;
}

int fannkuchredux()
{
    int n = 8;
    if (n > MAX_N) {
        return -1;
    }
    int perm[MAX_N];
    int perm1[MAX_N];
    int count[MAX_N];
    int maxFlipsCount = 0;
    int permCount = 0;
    int checksum = 0;

    int i;

    for (i=0; i<n; i+=1)
        perm1[i] = i;
    int r = n;

    while (1) {
        while (r != 1) {
            count[r-1] = r;
            r -= 1;
        }

        for (i=0; i<n; i+=1)
            perm[i] = perm1[i];
        int flipsCount = 0;
        int k;

        while (1) {
            k = perm[0];
            if(k == 0) break;
            int k2 = (k+1) >> 1;
            for (i=0; i<k2; i++) {
                int temp = perm[i]; perm[i] = perm[k-i]; perm[k-i] = temp;
            }
            flipsCount += 1;
        }

        maxFlipsCount = max(maxFlipsCount, flipsCount);
        int tmpflip;
        if (permCount % 2 == 0) tmpflip = flipsCount;
        else tmpflip = -flipsCount;
        checksum += tmpflip;

        /* Use incremental change to generate another permutation */
        while (1) {
            if (r == n) {
//                print_int(checksum);
//                puts("");
                return maxFlipsCount;
            }

            int perm0 = perm1[0];
            i = 0;
            while (i < r) {
                int j = i + 1;
                perm1[i] = perm1[j];
                i = j;
            }
            perm1[r] = perm0;
            count[r] = count[r] - 1;
            if (count[r] > 0) break;
            r++;
        }
        permCount++;
    }
}

when compiled with gcc without any optimizations to native code: real 0m0.024s

when compiled with gcc with -O3 to native code: real 0m0.016s

with agssim (agssim compiled with -O3) real 0m3.535s

agssim is pretty much optimized for speed, apart from doing safety checks before every memory access.

with ags engine (hacked some code into the script interpreter that upon encountering a sourceline statement runs clock_gettime(CLOCK_MONOTONIC, &now); to save a timestamp with nanosecond precision, and upon the next ret shows the difference) ret from 1337: took 4612390654 i.e. 4.6 seconds.

python 2.7 fannkuchredux program with n=8: real 0m0.446s

lua 5.1 version: real 0m0.186s

so ags engine + its script compiler is 10x slower than python, 25x slower than lua, and 288x slower than native code.

i'm showing these numbers here so the audience gets an impression how inefficient both the script compiler and the bytecode are. the speed of ags could be improved by approx 100-150% with a highly optimizing compiler, but it will never get even close to python due to the bytecode design.

[loquimur]

the speed of ags could be improved by approx 100-150% with a highly optimizing compiler

Well, I can only click my mouse so many times per microsecond. The Adventure Game Studio is a dedicated environment for creating point & click adventure games, so the engine just needs to be fast enough to keep pace with that. In my opinion, it is.

For me, the thing that is much more limiting to the adventure game coder is the power of the language itself. For instance, AGS doesn't have either multidimensional arrays or brace initializers, so things that are easy to code in C are frustratingly onerous to code in AGS. For an example, typical C code:

struct Location { int X, Y; } PoisonSpots[] = 
  { { 12, 17, }, { 23, 7, }, { 59, 223, }, { 99, 99, }};

Typical AGS code:

struct Location { int X, Y; };
#define PoisonSpotNum 4 // "readonly" wouldn't work, can't be used for array defn
Location PoisonSpots[PoisonSpotNum]; 
// We can't initialize the array here in global space, 
// so the initializers end up at a completely different location than the definer.

… much later in the code …
function room_load()
{
   …
    if (Game.DoOnlyOnce("Room2.room_load()")
    {
        … 
        // irky and verbose
        PoisonSpots[0].X = 12;
        PoisonSpots[0].Y = 17;
        PoisonSpots[1].X = 23;
        …
        PoisonSpots[4].Y = 99;
    }
   …
} 

Or another matter, AGS insists on doing "one-pass runtime linking" (instead of two-pass bundling-time linking) with the modules. Therefore, each function etc. must be defined at a location that lies before its first call, making recursive algorithms impossible to code.

These “rustic” elements of the language have sometimes annoyed me when coding adventure games, although there are certainly good ways to get around them, and overall, the requirements of the adventure game coder are catered for very well.

But I've never been hampered with speed limitations in my point & click adventure games yet.

[loquimur]

As far as I remember, the speed limitations that have come up recently seem to be in the library part of the engine which is coded in C++ und runs in native machine language. For instance, huge sprites aren't served fast enough to the viewport in movie animations for 720p games. Speeding up the Bytecode engine or switching to Python wouldn't help any to solve these problems, would it?

[ghost]

Sorry for barging in, but it seems the discussion above is about to go off-topic again (or maybe already had).

Just wanted to remind that this ticket was dedicated to decoupling a script compiler from the editor program. In this context IMHO is desired to have at least one release where we have a compatible compiler, ideally that supports everything current embedded compiler supports, and optionally has some perfomance improvements.

Total compiler rewrite, script language speed or switching to another script language, - these all are separate issues. For example, script language change has been discussed several times before, and mostly with the same results (some people stating it's slower than Lua, while others say that current speed is enough for them personally). So it's kind of going in circles. In any case, if you really like to discuss this, please open the new ticket.

IMO the problems and questions related to standalone compiler are:

• where the source code should be located in this project (same or separate repo).
• what is the input data, and how it is passed into stand-alone compiler;
• what input arguments compiler executable should support;
• what is the output and how is it used afterwards to build a game;
• distinct folder for intermediate output that is not interesting to the end-user (coming from compiler and used by further game assembly tool);
• making certain that the game project has all available input files ready for compiler, or editor at least capable to generate missing ones (because right now some of the data is either stored in big game XML, or generated by Editor in memory, or generated by plugins). It may be worth to open separate tickets for each of these subtasks. Recently opened: #1007, #1008.

[ghost]

@ericoporto

2, again, the thing I am asking is an interface in the compiler that could pull the header from the plug-in, now this interface doesn't exist, they have been made for the AGS Editor only which have been ifdef out of the build, so if you could, evaluate if the current interface available on Windows should be ifdef in the Linux build too, or if a new interface is needed to retrieve the header.

I don't believe this should be compiler's job to retrieve these script interfaces from plugin. And AFAIK in the Editor it was not compiler itself which was doing this.

We might need to find another way to handle this. It may be not the bad option to simply provide *.ash files alongside with plugins, so that these headers could be passed into compiler directly. Such approach would also exclude plugins from script compilation.

[ericoporto]

Agreed, alternatively maybe a tool extract the plug-in header. But yes, it should also be possible to just pass a header along, I just don't know how the compiler doesn't complain if there's nothing implementing the things it is importing.

I believe currently a __Plugin.ash of sorts is created at compile time by the Editor here.

[ghost]

I just don't know how the compiler doesn't complain if there's nothing implementing the things it is importing.

No, it does not. Linker does, but linking is currently a part of interpreter (engine). It's where it registers functions from all available sources (engine API, compiled script, plugins, built-in subs) and matches them to declared functions.

Something I forgot to mention, as an optional solution, Editor may just get header from plugin and export it before compilation too (similar to #1007). This may be a way if it's wanted to keep plugin management strict, where plugins must be registered in project (but that may be separate question).

[rofl0r]

For me, the thing that is much more limiting to the adventure game coder is the power of the language itself.

i removed some of the limits in ascc. for example it now has proper support for char, short and int* pointers, which allows to implement low-level stuff and work around some of the limitations. struct initializing etc would be nice to have, but i don't think i want to shoehorn that into the existing compiler.

As far as I remember, the speed limitations that have come up recently seem to be in the library part of the engine which is coded in C++ und runs in native machine language. For instance, huge sprites aren't served fast enough to the viewport in movie animations for 720p games. Speeding up the Bytecode engine or switching to Python wouldn't help any to solve these problems, would it?

it depends on your game, naturally. if your script is just "if click was here, show this message" then yeah, the speed is good enough. other more elaborate games like operation forklift or last n'furious, or quest for infamy spend a lot of time in the script processor. for example, op. forklift runs at about 14 FPS on my machine, after applying the optimization passes of "agsoptimize" from agsutils, it runs at least at 21 fps, which makes it playable. switching to python isn't a good option either way, because it's one of the slowest "industry-tested" language available. lua otoh is a lot faster, and dino even moreso. one of those two is what i would pick, if i was involved in designing ags4.

But I've never been hampered with speed limitations in my point & click adventure games yet.

right, you probably use scripts pretty sparingly or your users all have 3 GHZ rockets under their table. but if you want to target mobile or embedded platforms, mentioned game titles are unplayable.

desired to have at least one release where we have a compatible compiler, ideally that supports everything current embedded compiler supports, and optionally has some perfomance improvements

exactly what ascc does, then. :)

what is the input data, and how it is passed into stand-alone compiler;

well that part is pretty clear to me: ascc GlobalScript.asc -H header1.ash -H header2.ash -o GlobalScript.o

what input arguments compiler executable should support;

it should behave like other command line compilers do, pretty much as i implemented it in ascc (i happen to maintain 2 cross-compiler toolchains and a distro that focuses on compiling from source, so i'm pretty familiar with compilers).

what is the output and how is it used afterwards to build a game;

the output should be an object file with SCOM header, just like what the compiler already emits (but indirectly by shoving it into a "gamepack"), i.e. what ascc does. agsutils already provides a "linker" to put these object files back into a room or game data file, but it's a little hackish because ags wasn't designed to be used this way.

making certain that the game project has all available input files ready for compiler

right, this is the biggest issue at the moment. i could successfully compile your "last n furious" sources if some scripts wouldnt rely on constants defined inside the game xml data. (it's possible to work around it at the moment by disassembling compiled code and looking which constants are passed around, but that's clearly not something users would do).

[fernewelten]

I just don't know how the compiler doesn't complain if there's nothing implementing the things it is importing.

That's the standard way of handling code compilation. Visual Studio or the GCC suite do it that way too:

• The compiler's job is to compile each compilation unit, separately, concentrating on that unit only. To handle entities that are defined in other units, it gets a list of those entities as its input, and as part of its output, it lists a lot of code locations that reference those external entities.
• The linker's job is to resolve the references to external entities, i.e., to add the information “The first byte of that entity is there”.

The AGS approach is to defer the linking process completely to the time when the game is actually run. So it treats all the compilation units as a sort of “DLL”. This is one of the common approaches; it isn't a special quirky thing that only AGS does. The downside of that approach is that the Engine might only find out at runtime that something vital is missing.

Of course, nothing would stop the studio from doing a “static tryout linking” as part of the bundling (f7) to find out at this point, too, whether something important is missing. But AFAIK, this isn't implemented.

[ghost]

Hey, I decided to experiment with github Projects, and made a project as a way to track the bigger task: https://github.com/adventuregamestudio/ags/projects/1

@edmundito Your last comment in this thread states:

Primarily what I'm trying to do here is separate the logic of the editor that can load and build a game project. I updated the top post to make this more clear.

Were you able to get any progress in that? There's number of things which need to be changed or added in the Editor's workflow, - and some of these may not be immediately obvious, - so I think it's best to keep detailed plans written down here on the tracker, preferably split into subtasks, so that everyone interested realized the bigger picture (this is why I created a project board).

EDIT: another topic of concern here is to not have all the related changes done as a single bulk, which may be difficult to review and merge without breaking something.

PS. Not meaning to push or rush anyone, only wish everything related is tracked here for the reference.

[ericoporto]

https://github.com/edmundito/ags/tree/edmundito/decouple-compiler https://github.com/sonneveld/ags/tree/ags--script https://github.com/rofl0r/ascc https://github.com/fernewelten/ags/tree/Script_compiler_refactoring

These are all the efforts in the compiler I could find. It seems we need to find steps to advance that are bite sized and can be PR'd in, because doing a lot at once is hard to finish/merge. I suggest starting with figuring the directory to receive code reggarding command line tooling.

---

The compiler is already somewhat separated, in the Compiler.Lib vcxproj, in the ags/Compiler directory. There's also a different CSharp one in the ags/Editor/AGS.CScript.Compiler directory, this one despite the name appears to be the preprocessor since the preprocessor parts in the cpp compiler are either commented or removed.

The preprocessor being built with C# appears to be useful for the editor since it needs to preprocess stuff to figure out what should be available in the autocomplete. It's also very well separated, except by the fact it picks the AGS Version from AGS.Types (it's the single reference to AGS.Types in it if we don't count a spurious using call that can be removed without causing trouble). Can we pick the Version from somewhere else?

We use AGS.Types for some things in the build process but AGS.Types is linked to things in the panels (Windows Exclusive Winforms!). AGS.Types is also what is used to define the big Game.agf project descriptor.

There is some important questions above I will try to reply inline here.

IMO the problems and questions related to standalone compiler are:

• where the source code should be located in this project (same or separate repo).

In this repository. There's a top directory called Compiler/, we could instead have a BuildTools or AGSToolChain or AgsSDK, add a readme to clarify things there, and move the Compiler/ dir inside of it. AGS.CScript.Compiler should be move there too. This would give a directory to put cmdline frontends to these libraries and new stuff.

• what is the input data, and how it is passed into stand-alone compiler;
• what input arguments compiler executable should support;

For now let's try to work with what we have already. There's a mix here of Script compiler and Game builder being discussed in this thread. The script compiler could take the script file it's building plus a list of header files, the compiler then reads the files directly. I would suggest using the ascc design, but using AGS.CScript.Compiler as a separate preprocessor instead (building a command line frontend for it too). The game builder will have to be able to build for Release and Debug.

• what is the output and how is it used afterwards to build a game;

I don't know how it works today but at first this should work just like it's today. If we need file name suggestion for the compiled script I would go with .aso (ags script objects).

• distinct folder for intermediate output that is not interesting to the end-user (coming from compiler and used by further game assembly tool);

In the ags Project, it could be the Compiled/Intermediate directory.

• making certain that the game project has all available input files ready for compiler, or editor at

least capable to generate missing ones (because right now some of the data is either stored in big game XML, or generated by Editor in memory, or generated by plugins).

[ivan-mogilko]

The preprocessor being built with C# appears to be useful for the editor since it needs to preprocess stuff to figure out what should be available in the autocomplete. It's also very well separated, except by the fact it picks the AGS Version from AGS.Types ... Can we pick the Version from somewhere else?

Depends on how the version is used. It could be passed as a function/tool argument.

distinct folder for intermediate output that is not interesting to the end-user (coming from compiler and used by further game assembly tool);

In the ags Project, it could be the Compiled/Intermediate directory.

I have to disagree, "Compiled" was meant for final files (and some users already make mistake and upload whole contents of it with "Data", duplicating the game size).

[ericoporto]

What about Intermediate/obj/ then, on top level of the project for the result of Script compilation? I just thought of adding obj/ in case we decide to reuse the directory for other things (like, room crm files after the text rooms are done).

[morganwillcock]

When I was trying to think how to do this, for purposes of separating components but mainly to be able to compile a game from the command line, I think that I eventually concluded that there needed to be a new type defined that represents a project. This Project type would need to know how to load/save/edit(+upgrade)/compile and then the Editor and a new command tool operate using this new type. This is pretty large task though, as project editing code is scattered everywhere and the Editor has a very poor concept of what a project is - to the point that when using the sprite manager it doesn't even know for sure which game it is editing.

It was around this time it seemed much more appealing to look at redesigning the Editor and data formats and compilation outside of AGS 3 and once something decent exists look at how existing projects would be migrated. Also, outside of AGS3 you shouldn't have to worry about getting it right first time, particularly if no-one has run any data through it yet...

[ivan-mogilko]

@ericoporto I guessed you will suggest "intermediate" in the root, but this is going to be an obscure name for AGS users, as most are infamiliar with compilation technique and don't know what's going on under the hood. For that reason I'd propose using terms like "building", "temp" and similar, that clearly define what the folder is related to.

[ericoporto]

Temp is short and clean. We can use that :)

@morganwillcock, I opened #1092 which I think can be somewhat related to what you mentioned.

Does anyone knows if I can put arbitrary code in the AGS Engine and have it run? Like, supposed I built a script with the ascc compiler, could I run it with the AGS Engine itself?

[ivan-mogilko]

I think you should also define what is for 3. branch and what is for ags4 branch. For instance, this is roughly what was planned for 3.5.1 update: https://github.com/adventuregamestudio/ags/milestone/6 What is the goal for 3. in regards to standalone tools, and which milestone it belongs to?

Does anyone knows if I can put arbitrary code in the AGS Engine and have it run? Like, supposed I built a script with the ascc compiler, could I run it with the AGS Engine itself?

I am not sure if I understand the meaning of this question. But if a code were loaded into script interpreter, it could run it either by finding a function by name, or starting with certain random bytecode.

[ericoporto]

Thanks @ivan-mogilko ! I was thinking if it would be possible to run the bytecode produced by the compiler from the Engine so one could check the behavior of the code.

[ivan-mogilko]

I was thinking if it would be possible to run the bytecode produced by the compiler from the Engine so one could check the behavior of the code.

There are undefined details in this: what kinds of behavior are you going to test, and how do you plan to load and run this.

For instance, you won't be able to test many API function calls without loading full game data. Engine also currently does not have arguments and logic to launch any random bytecode from startup (or load one from file at runtime). If all you want is to test that script runs at all, I believe that rofl0r had a interpreter simulator "agssim" in agsutils that only runs basic operations and stubs API calls (afaik).

[ericoporto]

Thanks! I will use agssim for now.

Ah, unless I am missing something, if we had a tool that understands ags project files, so it could read and "orchestrate" the other tools, it could build a game using only the tools from agsutils. Like, sprite packing done in a command line tool instead of the Native.dll way that I think we do. The tools could live in a bin/ directory in the editor.

If this is indeed true, I think we should make ags versions of the relevant ones using the ags code that already exists (like the Common library) as much as possible to avoid code duplication.

[ivan-mogilko]

Ah, unless I am missing something, if we had a tool that understands ags project files, so it could read and "orchestrate" the other tools

Yes, that is indeed the point of this ticket.

Don't think such tool should literally command other tools though, rather prepare an input for them, as the whole group could be run simply from a shell script.

EDIT also, if I understand correctly, @morganwillcock 's suggestion was to treat whole dir as a project in future versions, which means file list could be taken directly from the dir, not project file (this may have unwanted consequences, so needs to be thought through beforehand).

[ericoporto]

Don't think such tool should literally command other tools though, rather prepare an input for them, as the whole group could be run simply from a shell script.

OK, it's alright too, just we need to make a tool to write some output files for other tools (erh, I imagine it basically pick pieces of the Game.agf xml file and spits the relevant parts in a separate one). This can be windows only and use AGS Types for simplicity.

This forum message goes into the project as directories idea, which is a good one, don't know how to get there right now.

Edit: removed some things from this comment to move to a new issue to avoid cluttering this discussion.

[ericoporto]

Uhm, just in case I am understanding something wrong, the sprite files when they get into the game data, is it copyied directly from the spritecache (like concatenated with other data that goes there - in the game.ags file) or is it read from the source files and put into the data "stream" , erh, the binary blob part of the sprite files in the game data that is read by the engine.

[ivan-mogilko]

Sprite file is packed into game data as a whole. Sprites are not taken from source files when compiling the game, they are stored in sprite file when imported by the editor.

This action if it's understood as doable, could spawn a new ticket

Yes, please, a ticket for separate task/discussion is a proper thing.

[rofl0r]

i checked out the writeup @ivan-mogilko mentioned in #1121 and it seems the logical next step to go on with this would be to write a command line C++ tool that reads the game.agf xml file and extracts certain things, for starters it could generate the header files that currently exist only in memory during the gui-triggered game build process, and subsequently change the build process to 1) invoke the tool 2) read the generated headers for compile from disk, so the code lives only in a single place to avoid code duplication. should i open a new ticket for this?

[ivan-mogilko]

Yes, there are several things that may have to be read from Game.agf, done by one tool that can execute several operations, or several tools, but using similar xml parsing.

I was thinking to start writing the tickets for this table too, since we mentioned that, but haven't got there yet, so certainly, do this if you want.

[ericoporto]

I had a good experience with Cereal in the past : https://uscilab.github.io/cereal/

It's header only and builds on anything using c++11, it may be useful to deserialize and serialize obects. It comes with json and XML out of the box but it's easier to plug something like yaml in it too.

[ivan-mogilko]

@rofl0r @ericoporto ok, I wrote one: #1146, please comment if you have further suggestions.

[ericoporto]

I have a feeling that it's better to close this and rewrite a new ticket for this - the discussion started at a time we had not researched much about this, it was a good kick in the butt to move a bit in this direction, but there isn't much in terms of actual practical design of what would be the code that fixes this.

Probably with references to https://github.com/adventuregamestudio/ags/wiki/AGS-Game-Build-process-(3.5.*)

[ivan-mogilko]

Right, I propose the new ticket should have following points:

1. The task goal and rationale.
2. The minimal requirements: what a user should be able to accomplish in order to consider this task completed.
3. The minimal necessary parts for covering these requirements.
4. Current situation overview (what is ready now).
5. What is left to do, preferably with a table of subtasks and ticket references (if available).

As a reminder, all this time there have been this "github project" which grouped related tickets: https://github.com/adventuregamestudio/ags/projects/1 and a milestone for the same purpose: https://github.com/adventuregamestudio/ags/milestone/10

[ivan-mogilko]

Closed in favor of #2316.