godotengine / godot

Godot Engine – Multi-platform 2D and 3D game engine
https://godotengine.org
MIT License
88.89k stars 20.15k forks source link

Proposal: GDVM / standalone GDScript #17846

Closed juan-garcia-m closed 6 years ago

juan-garcia-m commented 6 years ago

TL;DR

Expose the GDScript VM and re-implement GDScript source to bytecode compiler (tooling) in GDScript itself in a different git repository.

Motivation

GDScript is one of the best features of Godot. It let's you quickly run the editor, start coding and have a simple prototype in minutes. This is extremely useful for those who want to work in a multi-platform game but do not know how or are not willing to setup and maintain a cross platform C++ tool chain.

However, there is a trade-of for such convenience: there is only one language to choose from, and adding new features sometimes divides the community on if it is or not needed and how it should be implemented. For example, things like optional typing, which it might seem a minor thing, will never be implemented in a way that everybody will agree, while other languages features might be to great interest of only a few people and just skipped. All this just adds to the complexity of what it is just a huge difficult task to improve and maintain a community driven game engine. And the maintainers have been doing a great job on do it all that, with a great community that has also made great contributions.

But I think that all this can be improved, and maybe it can now be a great time for doing so.

Proposal

If the GDScript module get refactored in way that its VM gets exposed to the editor with the ability to use scripts in bytecode formats, interesting things can be done with it.

This can have many interesting advantages, and we should also consider its possible disadvantages. Bellow are just some ideas, and not conclusions.

Possible advantages

Possible disadvantages

Conclusion

Even with all the disadvantages, I think that implementing this proposal would have a positive impact in the future of Godot. It would be great to see how other Godot users and the developers see this proposal, either if accepted or rejected, see what new ideas, info, etc. can be discussed before any of us attempt to come up with a implementation and a PR.

reduz commented 6 years ago

GDScript is so simple, so small, so maintainable and hassle free in general (just works)… that it's difficult to justify anything else or any modification as better.

On Thu, Mar 29, 2018, 10:58 Juan Garcia notifications@github.com wrote:

TL;DR

Expose the GDScript VM and re-implement GDScript source to bytecode compiler (tooling) in GDScript itself in a different git repository. Motivation

GDScript is one of the best features of Godot. It let's you quickly run the editor, start coding and have a simple prototype in minutes. This is extremely useful for those who want to work in a multi-platform game but do not know how or are not willing to setup and maintain a cross platform C++ tool chain.

However, there is a trade-of for such convenience: there is only one language to choose from, and adding new features sometimes divides the community on if it is or not needed and how it should be implemented. For example, things like optional typing, which it might seem a minor thing, will never be implemented in a way that everybody will agree, while other languages features might be to great interest of only a few people and just skipped. All this just adds to the complexity of what it is just a huge difficult task to improve and maintain a community driven game engine. And the maintainers have been doing a great job on do it all that, with a great community that has also made great contributions.

But I think that all this can be improved, and maybe it can now be a great time for doing so. Proposal

If the GDScript module get refactored in way that its VM gets exposed to the editor with the ability to use scripts in bytecode formats, interesting things can be done with it.

This can have many interesting advantages, and we should also consider its possible disadvantages. Bellow are just some ideas, and not conclusions. Possible advantages

-

GDScript can be rewritten in GDScript itself, to produce bytecode output. This could enhance its maintenance as there are many people interested in GDScript that want to contribute but, for any reasons, can't do so in C++.

GDScript can have it's own repository, adding additional maintenance improvement to the current state. The main Godot repository can clone from an stable version or, better yet, download the bytecode compiled tooling during build time, so that the default release is still as simple as user friendly as ever.

New independent languages can be created by other developers and outside Godot. This can be from simple DSL, GDScript unofficial extensions, or other full languages. Those could be provided to the editor through plugins.

The scripting backend (the VM module) can be improved, optimized, bug fixed and extended without having to worry about other things like syntax, keywords, etc., and also tokenizing, parsing, etc.

For better or not, close source libraries can also be created this way. When this might sound insulting in an open source community (and not been a fan myself of proprietary solutions), this could be a new way to open new lines of contribution for those willing to provide commercial solutions with bundled library assets.

The main Godot repository will be freed of maintaining GDScript, which will help on ease the amount of issues and PRs, which are now way too many.

GDScript can evolve past the problem of breaking compatibility by providing multiple versions if the game developers needs them.

New build tools could be made. For example a 'require' or @Grab like per file dependency management, or a DSL build script.

Possible disadvantages

-

If GDScript is moved to a new repository, the main repository must be kept in sync somehow by pointing to an specific commit or release. In which case GDScript becomes a dependency. This could also be done by downloading the GDScript tooling when the editor first starts up.

GDScript tooling must be bootstrapped, as we have the chicken-egg compiler problem. This should be OK if we keep the first compiled version in the GDScript tooling in the git repository, and subsequent versions as build artifacts. The first compiled tooling to be kept in the git repository must be able to compile itself and it have to be developed in a version of Godot with embedded GDScript (as it is now). Maybe first developing a self-interpreter https://en.wikipedia.org/wiki/Meta-circular_evaluator can be a temporary option for testing.

Many incompatible, unpolished and with unclear features GDScript forks may spawn, but hey, this is the bazaar.

The nodes will have to point to the bytecode compiled classes files instead of the *.gd files when referring scripts. Maybe this can be handled the way the the collada files are handled now by using import files, so that the bycode files can be .gitignore'd and the source committed.

This will need more coordination than other more usual features PRs, as at least 2 developers, ideally contributors, should work on it, one on the C++ part of the VM APIs and editor exposure, and other in the GDScript tooling (source to bytecode compiling).

Conclusion

Even with all the disadvantages, I think that implementing this proposal would have a positive impact in the future of Godot. It would be great to see how other Godot users and the developers see this proposal, either if accepted or rejected, see what new ideas, info, etc. can be discussed before any of us attempt to come up with a implementation and a PR.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/godotengine/godot/issues/17846, or mute the thread https://github.com/notifications/unsubscribe-auth/AF-Z2xljkspJUzI24zBVoVUGCssBl5P2ks5tjSDMgaJpZM4TAvXP .

karroffel commented 6 years ago

Actually @hpvb and I have been talking about this quite a bit at FOSDEM and GodotCon, instead of downloading at build time, a git submodule could be used instead.

So our plans so far have been:

Another really huge advantage would be that we can play around with the language a lot more. We could develop new language features and people could opt-in to use those, since a language update would be as easy as updating the standalone compiler which generates still-valid bytecode.


All in all I'm very much in favor of this proposal, I think it can open up many new nice opportunities that are hard to do right now.

ghost commented 6 years ago

is this kind of how crystal-lang works? would be actually really fascinating to get that kind of performance from just gdscript. would be great for server development as well, getting that kind of performance w/ the same language used for game creation.

the only real downside i see is from point 4. surely, this would be an option since not everyone would have their stuff compiled yet? absolutely great idea @juan-garcia-m

e: also, can we use this as an example of how to write proposals. because the formatting is perfect, easy to read and i understood everything! :D

groud commented 6 years ago

I think I like the idea of having access to the bytecode. I understand that it could be used to develop or extend new languages. Also maybe we could compile VS into bytecode too ?

Sslaxx commented 6 years ago

Would (or could) this be accessed ala how GDNative/NativeScript plugins are now? If so, then this could be an automatic dependency for Godot (which could be later removed if)?

reduz commented 6 years ago

As Endragor said before, if you want all this right now you can use Nim

I am against making GDScript more complex than it is now (save for optional typing, which is being worked on), and much less a separate library.

"would be cool to" != practical or useful

vnen commented 6 years ago

While I like the idea, I don't think it's worth it. GDScript is very easy to maintain in the current state, going with this idea will make it much harder than it needs to be. And I don't think GDScript has the power to be used outside of Godot (there are so many languages already), so going through all of this work only to use it in Godot seems like wasted effort for me, it's simple to have it tightly integrated.

I agree GDScript has lots of room for optimization, but I don't think this proposal is required for that.

efornara commented 6 years ago

Personally, the only feature I would find valuable in this discussion would be the ability to generate a C / GDNative version of the scripts. That of course makes sense only after optional typing has been added.

I see no need for a separate library/project for this. C code generation should be fairly easy and can be integrated into the Editor. C Compiler integration might be trickier, depending of how deep you want to do it. For me, it would be enough some basic awareness of the .c / .dll/.so files and the ability to start an external tool (e.g. make) for .dll/.so generation.

juan-garcia-m commented 6 years ago

Thank you to all of you for your feedback!

Things like C generation or Nim, while interesting, are totally missing the point. The idea is not to have to depend of external tooling. Maintaining a multiplatform C toolchain is not something everybody want or are able to do.

It is more or less the same situation like a web browser: there was Flash, Java and even native clients, but none of those were as convenient as JavaScript, and all failed at the end. And now JavaScript, while been a decent language, it is not just perfect nor definitive for everything or everyone. Compiling to JavaScript and hacks like asm.js were useful workarounds but still all was hinting to the need of a common virtual machine. And WebAssembly is finally the answer. Why it took that long? Because many opinions said JavaScript was enough and more complexity wasn't needed.

One of the great things of using JVM for example it is the variety of langues you can choose from, knowing that you can easily (enough) change from Linux to Windows to Mac without having to install a different compiler tool chains. You don't have to provide or request sudo, you can have a portable self contained workspace that will not mess with your system, etc. If you wanted to implement a new language for the JVM you don't have to recompile the JDK for all platforms and you don't need to provide native libraries for each operating system.

I see that for many C devs might see this proposal as not interesting. But in the other side, other devs coming from other backgrounds like web programming and new users would appreciate the Godot zero install and the ability to choose and install additional languages from the assets store for example.

Also this will be interesting for those using GDScript, as many interesting proposals and feature request are been left open in limbo, some of them from 2014, and PRs not getting merged. It looks like GDScript it is just not getting enough attention to favor things like GDNative, that the majority of users won't use because it is not convenient.

And the idea of GDScript not been useful outside Godot, I disagree. If this proposal is accepted and implemented, first thing I will do is to port GDVM to the JVM, so that some of the same code can be used on the server.

vnen commented 6 years ago

The JavaScript scenario is a bad analogy. JS is enough for most applications, and WebAssembly is not a replacement to JS, but a complement (the same way GDNative can be seen as a complement to GDScript).

Also this will be interesting for those using GDScript, as many interesting proposals and feature request are been left open in limbo

Many proposals increase the complexity of GDScript only for convenience, or because of a background in other languages. A few things could really be added and should be checked out, but only because there's a proposal it doesn't mean it'll automatically be accepted. While GDScript may lack a clear roadmap (which could be solved), externalizing a VM won't make those features be accepted anyway.

It looks like GDScript it is just not getting enough attention to favor things like GDNative

That's a common misconception. @karroffel simply decided to work on GDNative. If he hadn't done it, GDScript would remain the same way, even without GDNative (same could be said about C#). Just because one people is working on one feature, it doesn't mean all other parts of the engine are being neglected.

You could say that GDScript needs more attention, but it has nothing to with GDNative.


Every language ecosystem has pros and cons. For instance, you started with JavaScript, which is an interpreted language, then went to the JVM, which is a bytecode interpreter. They are different from each other, because they have different target usage. There's not a better overall solution. While it sounds cool and it theoretically could have some benefits, I don't think that in practice it will be worth the effort.

efornara commented 6 years ago

@juan-garcia-m I also see the more general point, but I really don't see much value in it. If your proposal really grows as much as logic would lead it to grow, aren't you just ending up spending years to reimplement CLR badly? Mono/C# is kind of becoming a first class citizen already.

@reduz and @vnen hinted at the complexity. I'll try to elaborate on that, the way I see. This VM would be built on top of variant, which is a sort of a kitchen sink class. Nice and convenient, but not really general purpose. RID? AABB? NodePath? Plane? I don't see it taking off outside the game engine world. Which brings me to the main issue. What about these basic types? When you are proposing to take the VM out of the main repository, I really read it like proposing to take out two repositories: core (string, color, matrix, etc...) and VM (bytecode + basic tools). Core is a very hard dependency for the engine. You would end up with a "logical commit" becoming 2-3 "physical commits".

juan-garcia-m commented 6 years ago

I see that there is no much interest on this, which it is fine.

Let's say that instead I decided to write a VM module with no dependencies, or just those that can already be found in GDScript. Would such a module have a possibility to be included by default in Godot, like GDScript and GDNative is now?

juan-garcia-m commented 6 years ago

Closed as it doesn't seem to match GDScript design goals.

ghost commented 6 years ago

very sad. oh well, jit compiled gdscript is on the roadmap, so there's that sorry @juan-garcia-m