Improve the performance of the GDScript VM

[reduz]

Describe the project you are working on

Godot

Describe the problem or limitation you are having in your project

Many users have complained for a long time about the performance in GDScript.

For the GDScript 2.0 rewrite, a lot of work was put into making sure the VM has all the information required to generate optimized code, including type information, strict type checking, better addressing modes and the ability to access every operator, utility function, etc. as a C function pointer. This can be evidenced by the new codegen class used to emit code.

Unfortunately, just the changes to the parser and compiler ended up being very significant so almost no optimization work could take place for the VM.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

The idea is to use the the new type information as well as all the C function pointers to engine to generate direct executable code and bypass the VM, allowing for much greater performance when typed code is used.

This is challenging to do in a practical way, however. Because of it, this is more of an open discussion on evaluating different alternatives.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

There are a few general approaches that could be taken to optimize GDScript:

• JIT
• real-time AOT
• offline compilation.

The problem with JIT and real-time AOT is that there are many platforms that do not support real-time code generation, such as iOS, WebAssembly or consoles. Because of this, if we followed this approach, Godot would not run well on those.

As a result, the ideal way to compile GDScript code to binary would be to compile the whole script to a shared object (DLL on Windows, .so on Linux, .dylib on MacOS, etc) that can be bundled together with the exported game.

Fortunately, GDExtension provides pretty much everything the compiled script needs:

• A standard C ABI that allows accessing all the engine API and C function pointers.
• The means to bundle the shared object with the game on export.

The question that remains is how to build cross-platform code from the editor. There are a few alternatives to doing this:

• Require the user to install an external compiler (like GCC or LLVM) and basically generate the C code for the GDExtension, which the GDScript VM will optionally use to run instead of the interpreted code.
• Integrate some compiler infraestructure like LLVM or QBE to generate this.

External compiler

This is one of the easiest solutions, but it can be quite bothersome for the user to have to do this usability wise. It would make distributing Godot harder if the intention is to make sure code just runs fast. For some platforms like WebAssembly, I am not sure if there is any other way though.

There are tiny and embedded compilers that may be of use, but none really seems fit our needs:

• TCC: tiny, actively developed and embeddable C compiler that seems to meet all our needs. But license is LGPL.
• ChibiCC: Seems abandoned.

LLVM / QBE

Another alternative would be to integrate LLVM or QBE into Godot and generate the binary on the fly. LLVM looks like a lot of work though, since supposedly you have to implement the C ABI code yourself and its codebase is huge. QBE sounds like a much better project for this (it already has full C ABI support) but, while active, its developed by a smaller team and they don't use GitHub.

Lower level binary geneator

There are lower level libraries like SLJIT that do cross-platform codegen. This could work as an AOT compiler for platforms that can do this, and the code generated could be dumped to an shared object in advance on export for the platforms that do not support it (this would need some work).

The downside of this is that these libraries perform no optimization, so GDScript would need its own SSA form optimizer. This may not be terrible, but its some work.

So, what do you think? how would you tackle this problem in a way where the usability and maintainability balance is good in our favor?

If this enhancement will not be used often, can it be worked around with a few lines of script?

N/A

Is there a reason why this should be core and not an add-on in the asset library?

N/A

[reduz]

If we can't find a single solution that works, it may be possible we can just do an AOT compiler for platforms that support codegen, and have alternative write to C/GDExtension for platforms that do not.

[vpellen]

Disclaimer: My knowledge on compiler infrastructure and build systems is limited.

For me personally, the concern is less that GDScript is slow, and more that the alternatives are miserable to work with. My language of choice for many years was C++, and so GDExtension looks nice enough, but having to download external tools and build systems and thread everything together gets in the way of what I want to do, which is write performant code to solve my problems. I don't feel like I gain any meaningful control from choosing a compiler infrastructure or refining the build system.

I also don't really care about editor integration as much as I care about ease of use. If I could write a single trivial .cpp file and drag and drop it over some random executable that did all the boilerplate garbage and spat out a DLL for me to plug into my project, I wouldn't give a damn about editor integration.

So perhaps a distinction needs to be drawn. Are we trying to: A) Improve net performance for the average user, or B) Reduce friction for advanced users trying to accomplish specific ends?

[reduz]

@vpellen The goal is to improve performance for the average user, so definitely this is not for you.

[AThousandShips]

I think I'd be in favor of having a good GDScript centric optimizer, working against the codegen format of GDScript, with the option of exporting in executable format via some method. It's a big undertaking but I feel it ensures that GDScript behaves well in all cases (including dynamically created scripts), and makes prototyping easier (not having to export each time to get the optimization but only missing the boost from executable format)

[jabcross]

If we go with LLVM, I want to recommend using MLIR. It's somewhat of a more powerful, more generic evolution of LLVM IR that could allow some other things in the future, like compiling GDScript to compute shaders directly.

There's a lot of research at the moment and it's pretty powerful. (It's used by TensorFlow, for instance).

There are already abstractions for most of the common programming language features like structured control flow, functions, etc. And it already has a lot of stuff implemented, like the propagation of debug symbols. So line-by-line debugging would just be a matter of plugging into LLDB.

[reduz]

@jabcross The problem with LLVM IMO is that its enormous and that you are on your own for a lot of stuff to generate IR code. Given GDScript demands a ton of C ABI interoperabilty, C should work better and be simpler than an IR, but of course things are more complicated from there. This is why QBE seems more interesting than LLVM in the fact that its designed around C ABI interop.

I also don't think GDScript has a ton of optimizer potential that LLVM can take advantage of, so another possibility is to do simpler optimization manually and then run on something like SLJIT.

[chamakits]

(Not a core developer, just a big fan and casual/average user of Godot, mostly through GDScript, dabbled with Mono) Would this new solution mean to replace the current GDScript implementation, or to exist side by side permanently?

I'm asking because GDScript being a simple included VM is to me one of the biggest advantages of Godot for people to get started. Not requiring a compile step makes iterating on code changes very quick, and having everything included in the editor without any external install makes it the most pleasant Game Engine I've used. I think Godot's GDScript VM is what gives Godot such great ergonomics, at least for average users (like myself :D ).

If this is an alternative to the VM that lives side by side, then I'm a bit less concerned about the ergonomics being impacted. But I'd still be concerned if that increases the workload on the team. It seems it'd be effectively another whole back end (or set of back ends) that is officially supported, and may make future development much more complicated as it requires supporting many back ends. If the team has the bandwidth, then that concern is also addressed.

Do average users run into performance issues frequently enough with GDScript that it merits addressing with this large a shift in implementation strategy? Is there any usage pattern that can be identified that would be better addressed providing some other tooling (for example, nodes that abstract away the concept of resource pooling if a frequent performance concern is creating too many entities)?

[bruvzg]

TCC: tiny, actively developed and embeddable C compiler that seems to meet all our needs. But license is LGPL.

It's x86 and Linux + Windows only.

Cross-building with any external compiler will be a huge pain. Any version of clang should be able to generate code for any target, but linking the final shared library and any use of C/C++ standard libraries won't work without access to the platform SDKs for each target. But I guess the same will be true for the integrated LLVM / QBE as well.

[jabcross]

@reduz MLIR addresses a lot of these issues, but you're right that, at least for development of the engine itself, LLVM is quite big. But the compiler could dynamically link to the LLVM library once it's done, so the end users wouldn't be exposed to that.

The GDScript optimizer could be made as a separate module that most engine devs wouldn't have to compile.

MLIR provides a number of ways to deal with C ABI interop. (I had to learn them for my MsC project). There's also a C/C++ MLIR dialect being worked on that will greatly simplify connecting directly to C code at any level of abstraction, so we won't be limited to the ABI forever.

[vpellen]

@vpellen The goal is to improve performance for the average user, so definitely this is not for you.

Hey, I'll have you know I'm remarkably average!

Jokes aside, the reason I bring it up is because robust maintainable large-scale integrated performance improvements are probably going to be exceedingly difficult to implement. The average user is not going to be well versed in build systems and compiler infrastructure, which means if you want a good solution it'll have to be integrated with the existing editor in a pretty seamless way.

From what little knowledge I have on the subject, LLVM would probably be the most robust and future proof, but the overhead concerns are real, and I feel like Godot has a history of rejecting existing external codebases in favor of home-grown ones (Bullet is the most recent example of this I can think of). I have a sneaking suspicion that the larger codebases will be deemed excessive, while the smaller ones won't quite fit the needs, which will lead to the development of a custom solution that will take three years to develop and be miserable to maintain.

I'm hoping my concerns are unwarranted.

[Splizard]

Something not mentioned yet but I think is worth seriously considering here is using WASM itself as the GDScript intermediary format. There are plenty of fast and actively maintained FOSS WASM runtimes, (AOT, JIT and interpreters). https://github.com/appcypher/awesome-wasm-runtimes

Wasmer (AOT & MIT), WAVM (JIT & BSD) and wasm3 (interpreter & MIT) stick out to me as being the most performant in their respective categories. They are all close to native (WAVM even claims to exceed native performance in some cases).

Using WASM for this would also enable a better extension experience. No more hassle with making sure that the correct binaries are built and included for all possible platforms ahead of time. Just drop in your one WASM file and use any language that supports a WASM target to write Godot extensions, the WASM runtime will take care of executing the extension efficiently on any of the export targets.

[reduz]

@Splizard WASM is overkill IMO and the memory isolation for something like this really gets in the way. It also does not solve the problems mentioned above (export to platforms that need native AOT).

[badlogic]

As someone who actually build a commercial AOT compiler from JVM bytecode to x86 and ARM machine code using LLVM, I think compiling to C is the best option for various reasons:

• It's way simpler than learning and dealing with LLVM's ever changing APIs.
• The emitted code is much more easily inspected and debugged.
• It's easier for contributors to bug fix as the complexity and needed knowledge is a lot lower than contributing to an LLVM based machine code emitter.
• You can target new architectures/platforms with ease.
• You still get all the optimizations (mostly) you'd get when emitting machine code directly.

The one downside is that it's harder to emit debugging information, if it's a goal that the AOT compiled code should also be debuggable via the GDScript debugger. I don't think that's a requirement for a V1.

The other downside you mentioned is requiring a C compiler. I don't think this is a problem though. Godot could bundle or on-demand download Clang on all platforms. That's not really much more in download size than bundling the LLVM binaries. If needed, a customized Clang build could be maintained that strip away anything not needed.

[Zireael07]

Seconding @badlogic in that LLVM is not only big, but also changes A LOT.

[reduz]

@badlogic I think the AOT route via C is useful, but its still more hassle than what Godot users are accostumed to. IMO this is something that we should have to support anyway at some point (else Wasm, iOS or console optimization will not happen) but, as you say, for development this is a hassle.

To me C AOT (basically compiling GDScript to a C code that uses the GDExtension API) is something that is very simple to do (likely few thousands loc) and maintain.

But for development, my feeling is that we should have something that works out of the box, which is why I still think something like SLJIT makes more sense. The only thing you miss is the optimizer pass, but given the nature of GDScript I doubt this will be much a problem and the performance gain will still be very significant.

[reduz]

For reference, either as LLVM, C or SLJIT, all that needs to be done is provide a VM back-end: https://github.com/godotengine/godot/blob/master/modules/gdscript/gdscript_vm.cpp That builds its data from the generator: https://github.com/godotengine/godot/blob/master/modules/gdscript/gdscript_byte_codegen.h

Either spitting C or building an SLJIT execution buffer based on this is not a lot of work (order of thousands of lines of code). The reason is that most of those instructions, if you look closely, already call C functions to do most of the heavy loading (as an example, all built-in types, operators, constructors, and all the entire engine API are available via C function pointers). Eventually, significant optimization can be achieved by providing specialized versions of the instructions when the types are known (the codegen.h already lets you know this, so this work is already done). Even adding an SSA optimizer with most common optimizations would likely not be very hard.

In contrast, writing an WASM, LLVM or QBE back-end would probably be orders of magnitude more complex, because you need to replace all this with inline code in IR format. Of course, this has more optimization potential than the previous methods, but the amount of work required and the maintenance requirements are enormous.

[badlogic]

@reduz I don't think a C generator + compiler requirement is a hassle. Users already have to download export templates for all platforms. Providing the user a single click download inside the editor to fetch (a trimmed down) Clang seems to be pretty OK to me in terms of hassle.

As you said, you will need an AOT solution for iOS and WASM anyways, where SLJIT won't work. It's also questionable if an SLJIT generator can actually achieve any performance improvement compared to the current bytecode interpreter.

[Splizard]

@reduz

It also does not solve the problems mentioned above (export to platforms that need native AOT).

WASM runtimes totally support native AOT.

Wasmer 1.0 introduces “wasmer compile”, a new command for precompiling Wasm files. Our customers and developers use “wasmer compile --native” to precompile WebAssembly modules into native object files like .so, .dylib, and .dll.

(https://medium.com/wasmer/wasmer-1-0-3f86ca18c043)

the memory isolation for something like this really gets in the way

Reference types in WASM are a thing (also supported by wasmer), I'm not sure about the performance profile of say keeping all packed arrays and strings as reference types but it's certainly a solution. Modding is certainly an area where WASM runtime isolation could be helpful.

As @vpellen indicates, I don't think you can beat the user experience of dropping a single WASM file into your project and being able to export this seamlessly to mobile devices, consoles, etc. Let the WASM runtimes focus on performance!

[reduz]

@badlogic It is a hassle if this is a requirement to even use the engine. Godot is a tiny download that just works and a large amount of the Godot community largely appreciates this. The export templates are only downloaded at the time of exporting the game, but you don't need them to start working quickly.

I agree with you that this is likely the path we will have to take at some point, but it should only be an optional step for games that only really need this extra performance. The current bytecode interpreter still has a good amount of room for optimization (addressing needs to be further simplified and typed instructions need to be added), but as the intepreter is a gigantic jump table, it still has a big toll on the L1 caches and branch predictor, which should be significantly improved with something like SLJIT, and that works out of the box with little work.

[reduz]

@Splizard Its good to know you can handle AOT but, as I said, I still think the isolation sucks and creates a lot of glue friction that I would be happy to avoid. Additionally, you still need some sort of compiler so, in the end, the advantage provided is minimal vs the drawbacks.

[YuriSizov]

For the reference, there is an existing proposal for a WASM runtime with a decent amount of support, but not much discussion: https://github.com/godotengine/godot-proposals/issues/3370

[xsellier]

I know it is an unpopular opinion, what about dropping GDScript and using Python instead?

I would like to clarify my point. As a matter of fact, writing a programming language is the work of a full-time team. I find it ambitious for Godot Engine to create a game engine with a whole new approach with a new programming language.

Perhaps Godot Engine should do with GDScript the same thing that happened with VisualScript. At the same time switching to another language with a syntax close to GDScript, such as python. It should close a lot of tickets directly related to GDScript and allow us to focus on the core of Godot Engine, i.e. being a game engine.

I'm aware that this is only my opinion, and that I'm not the person who maintains GDScript, nor the one who will be in charge of switching to Python, and even less the one who maintains Godot Engine in a broader sense.

[YuriSizov]

@xsellier Juan just recently recalled the creation of GDScript on Twitter, and mentioned why Python was not a good option. It is also covered in part in our official documentation.

We like GDScript because it allows us to tightly integrate into the engine, to remove a lot of overhead both conceptually and in terms of the interop. Maintenance of our own tool does come at a cost, but so does maintenance of a blackbox solution that we must embed into our project. Take our approach to physics for example, you see the same thing. We opted to abandon Bullet as the main backend because it is very hard to keep that bridge between the two parties sound. Same applies here.

Aye, it will close some bugs in our codebase. But it will also open new bugs which may not be within our reach to fix. So we'll have to bother other projects with our issues, projects which may not share our goals or don't see our problems as worthwhile. But our responsibilities before our own users won't go anywhere, so we'd have to either take the blame or ship some hacks and ad-hoc fixes, if we can. That's not great at all.

[reduz]

@xsellier Python is a terribe option. Letting aside that there is not a good VM that we can use and that all the nice integration to the editor would be lost, the VM is not designed to work standalone. Even if you make it work standalone, you lose access to the rest of the ecosystem, or you bloat massively your exports as a result.

Python is designed so applications are parts of its ecosystem and not the opposite. It definitely is nice to have Python support for some things (if you work on scientific or academic fields as an example) but it's not meant to be used for something like Godot.

[xsellier]

I get your point, Python might not be the best choice because of the size of its VM.

My point is still relevant, writing a programming language is a full-time job for a whole team of dev. Meaning, either hiring more devs to work on GDScript, or dropping it and take another already existing programming language.

I'm quite happy about what's GDScript is becoming, First class functions, static methods, new built-in types... but for the last 2 years, progress reports about GDScript are quite sparse ( https://duckduckgo.com/?sites=godotengine.org%2Farticle&q=GDScript&ia=web ). I mean I'd expect more update on GDScript since it is a programming language with quite some bugs ( https://github.com/godotengine/godot/issues?q=is%3Aopen+is%3Aissue+label%3Atopic%3Agdscript ). Some bugs are more critical than others and lasts for years ( https://github.com/godotengine/godot/issues/7038 ). I'd like to state it is not a complain, because I use GDScript and I like the way it is, it is more like a realistic statement.

My point is more, creating a programming language is too much work for Godot Engine atm, it might be better to think about dropping it, to focus on what's imprtant for the engine. You can focus on Python if you want, I was talking about dropping GDScript in favor of something else.

[reduz]

@YuriSizov Its probably not a lot of work to have a Wasmer module that can load GDExtensions inside the runtime compiled for Webassembly. There is likely not a lot of merit either, but using it for running GDScript is definitely not a good solution. GDScript uses engine types for everything, so the isolation means the glue needs to either spend a lot of time converting the engine types to communicate with the engine, or it needs to keep types as external references and access to them is slower.

This is what happens in C# as an example, where you need to be conscious on using engine types or native types depending on what you need for the most part (accessing data or passing it), which is fine for this language as users are expected to be more experienced. But for something easy to use like GDScript that is meant to "just work", it will cause a lot of friction.

[YuriSizov]

My point is more, creating a programming language is too much work for Godot Engine atm, it might better to think about dropping it, to focus on what's imprtant for the engine. You can focus on Python if you want, I was talking about dropping GDScript in favor of something else.

@xsellier Besides a couple of links to address your specific proposal at the start the rest of my comment was not about Python, it was exactly about the cost of dropping a bespoke solution which is GDScript for something else. You keep implying that using an off-the-shelf language would somehow be free, or at least cheaper to maintain. I made a point that it's not the case.

[reduz]

@xsellier The problem with your logic is that, what I believe you fail to understand (likely due to lack of experience in the field) is that Integrating another programming language to Godot and keeping the fantastic "just works" usability GDScript is even more work than developing GDScript.

Writing glue code is difficult and there are a lot of corner cases:

• How do you handle type conversions to engine conversions efficiently?
• How do you handle threads?
• How do you handle a class model that resembles and references the same as the engine?
• How do you handle exporting and packing?
• How do you handle reference assignment and garbage collector with engine types?
• Etc.

I already integrated other languages such as Lua, Squirrel and Python to Godot. It was a ton of work and generally there is more code to do the glue than there is GDScript runtime code, and inefficiencies everywhere when getting data one way or the other (same reason why I think Wasm as script runtime is a terrible idea). This without even getting into all the nice editor integrations GDScript has to the engine, which would be a massive amount of work to do in another language (where in GDScript they are tiny).

Building software is not stacking lego bricks where you add something existing and it magically works. Getting the pieces to fit and keeping them together often requires as much effort as writing the pieces yourself.

[xsellier]

@xsellier The problem with your logic is that, what I believe you fail to understand (likely due to lack of experience in the field) is that Integrating another programming language to Godot and keeping the fantastic "just works" usability GDScript is even more work than developing GDScript.

Wow, that's quite a strong argument you got here. I agree there are corner cases, it has to be, and I never said it would "just work", but integrating an already existing programming language, like you did with dotnet didn't create more bug than gdscript had. Again, I might be wrong, but I would never question your abilities or knowledge because we don't agree.

What I say is, creating a programming language takes more manpower than using an existing one.

[somebody1121]

Why not translate gdscript into C# on export, since the engine already has optional support for it? The end user could use a keyword like C#Translate to chose what files to convert, like heavy scrips that run faster on C#.

[reduz]

@xsellier Again you are wrong here, C# was an inmense amount of work to integrate, with far more developer time spent on it than GDScript. It is also still far from perfect and there is glue friction and worse engine integration everywhere, but developers bear with it because they are often more experienced (just like they do in Unity) and because currently the performance tradeoff is for many of them a win.

As I said, in my experience, integrating something existing that works as well as GDScript is far more work, hands down.

[reduz]

@somebody1121 Because of the same argument made in the points above. It would add a lot of friction and worsen engine integration. At that point, just use any other language that runs on the .NET VM, which is why UnityScript failed.

GDScript is a domain specific language that is all about tight integration to Godot. Its strength is that it "just works" with zero user friction. This needs a specialized parser and VM.

[AThousandShips]

@xsellier even if the point that integrating an existing language would take significantly less work than creating a new one (I too say that it absolutely does not, assuming you want it to work well and not be a major bottleneck) we're not creating a new one, we already have GDScript, there needs to be a strong reason to abandon an already established language (not resting entirely on "sunk cost" here but there still needs to be a strong reason to abandon a lot of work), creating a huge change for end users, and all the issues already presented here about such an integration.

Aside from the claim that it would be easier to integrate and maintain, what are the real arguments for abandoning GDScript? What would be gained?

Yes there are bugs in GDScript, but it isn't like it's unusable, quite the opposite, and what reason is there to believe integrating an existing language would yield fewer and not more issues?

The reason I'm asking is that I don't see any arguments in favor, only "it isn't as hard as you say".

[xsellier]

@AThousandShips I would have loved to have a constructive discussion about why it is or isn't feasible, trying to understand the other's point of view without being defensive, like in Stephan Zweig's 24 hours in the life of a woman. At no stage did I want to offend anyone, without questioning the competence of the interlocutors. So I prefer not to continue the discussion.

[AThousandShips]

@xsellier I'm sorry if I came across as defensive or upset, nor to question your competence, I'm sorry if I've caused any offense, you surely haven't caused any to me. Have a nice day!

[xsellier]

@xsellier I'm sorry if I came across as defensive or upset, nor to question your competence, I'm sorry if I've caused any offense, you surely haven't caused any to me. Have a nice day!

Your post was fine, it wasn't your post that made me think that. I had already dropped the conversation before your intervention.

[reduz]

@xsellier Apologies from my side, I did not meant to question your competence, since I honestly don't know you. I only meant to say that this is a topic you really need to get familiar with to understand the tradeoffs and very few people are as obsessive as to waste so much time on areas like this.

I can't offer much more in the way of an explanation than telling you that the friction you find when adding a large third party dependence is much larger than what common sense may imply. I can only cite out of my head a few complex scenarios but in reality there are hundreds I stumbled upon. I did this for more than two decades and with different programming languages, so I am quite convinced of what I saw and the types of problems you find.

If you don't want to believe and pretend that I have no idea, that's fine and, if you are still really interested, I would be happy to go into a lot more technical detail to convey my point better (even if we are derailing the OP quite a bit), but there is not a lot more I can offer.

[touilleMan]

Regarding the External compiler part, Ziglang has done an incredible work So downloading the ~40mo zip of the Zig program gives you a C/C++ compiler (based on LLVM/Clang) that basically support all cross-compilation platform you can dream of 🥳 🎉 (and of course you get for free a zig compiler, but that's not the point 😄 )

Unrelated point: It would be interesting to provide a way to expose the C compiler to 3rd party module: I'm myself considering shipping a C compiler as part of Godot-Python in order to provide an out-of-the-box Cython compilation experience (Cython code get converted into C code by the Cython compiler).

[reduz]

@touilleMan This is interesting, but is it not the same as downloading a standalone LLVM? Regarding calling a compiler, Godot 4 has now hooks that let you alter any single resource during export so you should be able to gather your resources and do whathever you want with them and the exporter will cache this.

[touilleMan]

@reduz

but is it not the same as downloading a standalone LLVM?

You can consider zig as an optimized distribution of LLVM. For instance all the headers files has been de-duplicated between the supported platforms so that it takes only a fraction of the size it would need (~380mo for linux only clang vs the 45mo for zig with all platform support)

From the article I linked:

Bundling a C compiler as part of a larger project

With Zig tarballs weighing in at under 45 MiB, zero system dependencies, no configuration, and MIT license, it makes for an ideal candidate when you need to bundle a C compiler along with another project.

For example, maybe you have a programming language that compiles to C. Zig is an obvious choice for what C compiler to ship with your language.

Or maybe you want to make a batteries-included IDE that ships with a compiler.

So it seems our current usecase is something the Zig team consider there tool is made for ;-)

We could have a "download C cross compiler" option in the Godot editor just like we have one to download the export templates, then compile all the C we want for any platform \o/

[reduz]

@touilleMan that is super interesting, mindblowing that zig can make a smaller cross compiler than anyone else. Technically all we need is a C compiler and not even system headers (and it it could be small enough to be embedded we are all set), which is why TCC is so interesting.. but LGPL license sucks for us and as @bruvzg mentioned, no Apple platform support.

[touilleMan]

I guess we can ping @andrewrk to congratulate him for this incredible achievement 🎉 😄

[reduz]

@touilleMan I have to be honest that Zig is probably the only of the new languages that is of interest to me. It's like they read my mind on anything I would like out of a modern language.

That said, sounds like we have a very good candidate if we want to do the "write to C on export" path.

[jabcross]

Zig first class support confirmed

[reduz]

@jabcross zig should be one of the easiest languages to support via GDExtension since its fully C compatible. No idea if anyone is working on it, though.

[jabcross]

There are GdNative bindings but they seem to also suffer from the C ABI issues. https://github.com/thimenesup/GodotZigBindings

[bruvzg]

Cross-building with any external compiler will be a huge pain. Any version of clang should be able to generate code for any target, but linking the final shared library and any use of C/C++ standard libraries won't work without access to the platform SDKs for each target. But I guess the same will be true for the integrated LLVM / QBE as well.

Done some testing, upstream LLVM-14 clang / lld toolchain is perfectly capable of building working dynamic libraries for macOS / iOS / Windows and Linux on any platform and without OS specific SDKs, as long as it's done with -nostdlib -ffreestanding, and linker is called manually. So it's not as bad as I was thinking. And the minimal usable part of the LLVM for this seems to clang and lld executable + libLLVM and libclang-cpp libs only.

[JoNax97]

Would any of these approaches work for scripts not included in the main executable, but instead exported into a PCK file and loaded dynamically at runtime? That's major use case for us and it would be nice to have the performance boost.

[reduz]

@JoNax97 The way I originally thought about this is, besides of course being optional, by doing it entirely as GDExtension. On export, you dump/compile an extension that gathers all scripts and generates a .so file that registers function pointers that existing scripts can use to run instead of the bytecode if they are available.

So, I suppose in your case, the PCK would contain or come with an extension of its own.

[JoNax97]

Nice, I didn't know you could include extensions/ dll-like files in packages. As long as there's some API that allows all this process to be automated we're golden