Implement Neural Networks and Reinforcement Learning Modules

[csstej]

Describe the project you are working on

Trying to implement a basic Artificial Neural Network in gdscript.

Describe the problem or limitation you are having in your project

Implementing a neural network from scratch can be tiresome and lead to bugs which can be complicated to track, also neural networks are best implemented in c++ using gdnative, but the implementation complexity stays the same.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

Using mlpack Library (BSD License), which already has implementations of all the basic Machine learning Algorithms, Deep Learning Architectures and Reinforcement Learning models and implementing them by Binding this library to Godot.

For example, in a library like Tensorflow, creating a neural network can be done in a few lines of python code like:

model = Sequential()
model.add(Dense(10, input_shape=(8,)))
model.add(Dense(5))
model.add(Dense(1)) 

which creates a basic neural network.

The main reason for this proposal is to make AI in games easier with less code, even for someone who isn't experienced in reinforcement learning algorithms or neural network models can use this module. By creating bindings in gdscript, it should be able to create an abstraction layer, so beginners should have just the basic understanding (ignoring the math), and be able to create AI at ease! This library could also be implemented as a GDNative plugin.

Also this feature could not only be useful for game developers but also Researchers, Data Scientists and ML engineers, who would like to test out their work in an isolated environment. eg: Testing out an algorithm for a self driving car, Robotics etc;

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

Implementation of Neural Network Regression using the mlpack library.

After Binding the module to godot using it in gdscript should closely resemble that of python from the tensorflow example, (pseudo-code):

model = Sequential.new()
model.add(trainX.n_rows)
model.add(10)
model.add(5)
model.add(1) 

The Algorithms can be implemented either as a node or a Resource.

Training multiple agents like in unity's ml agents:

If this enhancement will not be used often, can it be worked around with a few lines of script?

Can't be implement in a few lines of code.

Is there a reason why this should be core and not an add-on in the asset library?

From the reasons stated above.

[Calinou]

This library could also be implemented as a GDNative plugin.

In this case, it should remain a GDNative plugin to not bloat the core with a feature most developers won't use :slightly_smiling_face:

Machine learning libraries are rarely compact in terms of binary size, so we need to be careful about what goes into core Godot.

[Jummit]

So this should be closed then? Or should the GDNative plugin be discussed here?

[Zireael07]

IIRC we had a proposal to do with NN already? So most likely it can be closed...

[Calinou]

So this should be closed then? Or should the GDNative plugin be discussed here?

This repository is designed to discuss features that should be maintained officially. While GDNative add-ons can be official (example), I doubt we'll take on the task of maintaining bindings for a machine learning library ourselves. The community will likely scale better for such tasks.

[YuriSizov]

So this should be closed then?

There is no rush to close proposals, unless you can point for a direct duplicate or they don't follow the template at all. Even failing to satisfy the "can't be done as an addon" rule is not a sign to immediatelly close it. Besides, we were asked recently to keep proposal open for a minimal time. 🙂

[mohsenph69]

I createed a GDNative library for neural network and there is youtube tutorial for that https://github.com/mohsenph69/Godot-Neural-Networks

[csstej]

@mohsenph69 Nice, i'll check it out. Also which library did you use to implement the neural network architecture, or was it built from scratch?

[fire]

The two approaches that work today are:

1. https://github.com/lupoglaz/GodotAIGym
2. https://github.com/touilleMan/godot-python

(note that my tensorflow-lite godot module isn't maintained.)

[Xrayez]

So this should be closed then? Or should the GDNative plugin be discussed here?

This repository is designed to discuss features that should be maintained officially. While GDNative add-ons can be official (example), I doubt we'll take on the task of maintaining bindings for a machine learning library ourselves. The community will likely scale better for such tasks.

This was previously discussed in godotengine/godot#30613 (see also godotengine/godot-proposals#1577).

It depends on user support and developer base. If this approach becomes mainstream for implementing AI-like behavior, then this proposal will become more relevant in the future. But yeah I kind of doubt that Godot would have this out of the box in either case, unless reduz would want to delve into this topic himself. 🙂

[ghost]

I'm making a simple 8x8 board game, single player vs single cpu character + some other entities like bombs, hearts, etc. For starters, I've collected the (input, output) data by bruteforcing the cpu character to do random moves and selecting the ones that yield benefits for said character. After that I designed a sequential model using python and keras which uses only Dense layers so I could collect the weights and biases once the model is trained to eventually run them through a simple forward pass algorithm inside Godot. And here is where my problems started.

After hours & hours of training, reading and studying I've finally realized that I actually need to use Conv2D nets instead of just plain Dense sequential layers to achieve more than ~78% accuracy. A forward dense layer is easy to implement in Godot (it's just basic linear algebra after all) but Conv2D networks are a totally different beast. I'm still trying different configurations and figuring out ways to use just plain Dense layers with a sequential model but I'm definitely running out of options here.

Here the deal: I don't really need to train anything inside godot, the frameworks are doing all the heavy lifting in my case, but I definitely, desperately need an advanced library to run predictions, that's it.

[skaiware]

Hi all, Bringing a minimum of NN (not speaking of RL) in godot have been proposed dozen of times in the last 2 or even 3 years. None of them have been sadly (even partially) accepted and consequently, as today (without risky addons/unofficial projects):

• there is no way in gdscript to infer a simple Feed Forward Network
• there is no way in gdscript to infer just 1 linear layer (general input matrix- weight matrix multiplication + broadcasted addition of bias)
• there is no general Tensor (a n-dimensional array with linear algebra) class in gdscript to do simple linear algebra (gemm, reductions, convo, activation, ...).

During that time, godot competitors did not wait. In 2017, Unity officially released 'Barracuda', the 'Unity-Inference-Engine', to at least run inference on a NN in the official Unity : https://docs.unity3d.com/Packages/com.unity.barracuda@1.0/manual/index.html https://github.com/Unity-Technologies/ml-agents/blob/main/docs/Unity-Inference-Engine.md

I suppose some, not understanding the domain are perhaps just afraid of adding a new huge risky dependency to Godot core. If so, let's just remind what @postmortem-x has just confirmed :

• the large majority of us (game developers) do not need to train a model in godot in the realtime final released game. That is usually done offline using our preferred full external framework (pyTorch, TF, ...).
• most of us just need, at least, to run the 'inference', sometimes called forward pass, at least on some typical NN layers. Technically, this just need a generic 'Tensor' class. Most NN fwd layers would then be easy to implement at low (c++) or even high level (gdscript) via a simple wrapper.

Kind regards S.

Ref: an example of application which just needs to be able to infer a simple 10KB linear NN : http://theorangeduck.com/page/phase-functioned-neural-networks-character-control https://www.youtube.com/watch?v=Ul0Gilv5wvY

the eigen Tensor class, core of TF: https://eigen.tuxfamily.org/dox/unsupported/eigen_tensors.html

the ATen Tensor class, core of pytorch: https://pytorch.org/cppdocs/

[gilzamir18]

Hi all, Bringing a minimum of NN (not speaking of RL) in godot have been proposed dozen of times in the last 2 or even 3 years. None of them have been sadly (even partially) accepted and consequently, as today (without risky addons/unofficial projects):

• there is no way in gdscript to infer a simple Feed Forward Network
• there is no way in gdscript to infer just 1 linear layer (general input matrix- weight matrix multiplication + broadcasted addition of bias)
• there is no general Tensor (a n-dimensional array with linear algebra) class in gdscript to do simple linear algebra (gemm, reductions, convo, activation, ...).

During that time, godot competitors did not wait. In 2017, Unity officially released 'Barracuda', the 'Unity-Inference-Engine', to at least run inference on a NN in the official Unity : https://docs.unity3d.com/Packages/com.unity.barracuda@1.0/manual/index.html https://github.com/Unity-Technologies/ml-agents/blob/main/docs/Unity-Inference-Engine.md

I suppose some, not understanding the domain are perhaps just afraid of adding a new huge risky dependency to Godot core. If so, let's just remind what @postmortem-x has just confirmed :

• the large majority of us (game developers) do not need to train a model in godot in the realtime final released game. That is usually done offline using our preferred full external framework (pyTorch, TF, ...).
• most of us just need, at least, to run the 'inference', sometimes called forward pass, at least on some typical NN layers. Technically, this just need a generic 'Tensor' class. Most NN fwd layers would then be easy to implement at low (c++) or even high level (gdscript) via a simple wrapper.

Kind regards S.

Ref: an example of application which just needs to be able to infer a simple 10KB linear NN : http://theorangeduck.com/page/phase-functioned-neural-networks-character-control https://www.youtube.com/watch?v=Ul0Gilv5wvY

the eigen Tensor class, core of TF: https://eigen.tuxfamily.org/dox/unsupported/eigen_tensors.html

the ATen Tensor class, core of pytorch: https://pytorch.org/cppdocs/

I'm developing a framework (with visual appeal) that has the features you suggest: https://github.com/gilcoder/ai4u. It's not complete yet. The goal is that the environment can be modeled in Godot, visually as much as possible. For that, I created an agent-based framework.

[edbeeching]

I have been reading up on the numerous requests for NN inference inside Godot. I am the author of the Godot RL agents library. Currently we perform training in python but for version 0.2.0 would like to load the trained model inside Godot for inference. I am looking for C# libraries for model loading (ideally ONNX format) and inference in Godot. Do you have any recommendations?

[GeorgeS2019]

@gilcoder @Xrayez We work as a team recently at Microsoft Hackathon and we combine the best of both worlds e.g. PyTorch training involving Gym ( OpenAI RL) & Godot Physics engine environment and THEN export the trained model to ONNX format for consumption using c# in Godot . It is a straight forward to use GDScript to call the c# wrapper of ONNX directly all within Godot without involving python server.

=> Here is the link to Godot ML Agent repo

[GeorgeS2019]

This seems to be an interesting .NET Reinforcement Learning library potable to Godot

[fire]

I am currently evaluating https://catboost.ai/ for non deep learning machine learning and OpenVINO https://docs.openvino.ai/latest/index.html for deep learning. If anyone was interested, contact me.

[GeorgeS2019]

Training multiple agents like in unity's ml agents:

@csstej You posted in March, Have you found Unity ML Agent-like solution for Godot?

Here we provide a (possible) WORKFLOW for Godot which is very similar how Unity ML agent train and deploy RL agent.

What need to be done NEXT!!

• The main limitations now are the Godot UI parts (no longer the RL models) to consume the trained Reinforcement models
• Through this approach, It is possible (as almost similar workflow as Unity ML Agent) to consume Unity ML Agent trained models (this has to be investigated and I need comment if there is licensing issue using the Unity ML Agent models in Godot).

Why? This workflow is using the same source of RL library (Stable Baseline SB3) used by Unity ML Agent.

How important to Godot community?

The path to Godot option for users to do similar things like Unity ML Agent will still take many months, if not years. If the Godot community can come together to do the What need to be done NEXT!! to make Godot a viable option, we will be reduce the time to perhaps less than 6 months (especially now there is a WIP workflow (as approach as Unity ML Agent => using the same MIT license RL python library))

@fire => You are absolutely right with your assessment . Now the community need to come together to TRAIN and DEPLOY Unity ML Agent like solutions ====> ALL within Godot

[ashtonmeuser]

I built a small addon for TensorFlow inference in Godot available here.

Currently, only built for and tested on macOS. The README includes instructions for building from source and I'd welcome any PRs targeting other platforms.

[gilzamir18]

ML-Agents has its merits, but I believe it has a wrong design. At the current stage of DRL in games, which is very experimental, Barracuda is very limited. For example, try getting the return of the state value estimate function and you'll find it's not easy if you try to incorporate that into the agent logic in C#. It is difficult to get any customization of the neural network that goes a little beyond the traditional feature-extractor --> dense/lstm/gru layers --> output layers. Trying to achieve compatibility with MLAgents is limiting yourself. DRL isn't as mature as to be boxed in as the Unity folks seem to think.

[gilzamir18]

@gilcoder @Xrayez We work as a team recently at Microsoft Hackathon and we combine the best of both worlds e.g. PyTorch training involving Gym ( OpenAI RL) & Godot Physics engine environment and THEN export the trained model to ONNX format for consumption using c# in Godot . It is a straight forward to use GDScript to call the c# wrapper of ONNX directly all within Godot without involving python server.

=> Here is the link to Godot ML Agent repo

I stopped messing around with Godot for a while, mainly because I have a project running in Unity and I have little time to deviate from that path.

[GeorgeS2019]

ML-Agents has its merits, but I believe it has a wrong design. At the current stage of DRL in games, which is very experimental, Barracuda is very limited. For example, try getting the return of the state value estimate function and you'll find it's not easy if you try to incorporate that into the agent logic in C#. It is difficult to get any customization of the neural network that goes a little beyond the traditional feature-extractor --> dense/lstm/gru layers --> output layers. Trying to achieve compatibility with MLAgents is limiting yourself. DRL isn't as mature as to be boxed in as the Unity folks seem to think.

@gilcoder I read what you have written. I try once more to understand ai4u.

I stopped messing around with Godot for a while, mainly because I have a project running in Unity and I have little time to deviate from that path.

So the case is close for you.

[gilzamir18]

ML-Agents has its merits, but I believe it has a wrong design. At the current stage of DRL in games, which is very experimental, Barracuda is very limited. For example, try getting the return of the state value estimate function and you'll find it's not easy if you try to incorporate that into the agent logic in C#. It is difficult to get any customization of the neural network that goes a little beyond the traditional feature-extractor --> dense/lstm/gru layers --> output layers. Trying to achieve compatibility with MLAgents is limiting yourself. DRL isn't as mature as to be boxed in as the Unity folks seem to think.

@gilcoder I read what you have written. I try once more to understand ai4u.

I stopped messing around with Godot for a while, mainly because I have a project running in Unity and I have little time to deviate from that path.

So the case is close for you. I'm sorry, I don't have time to migrate for now, but I'll keep watching and particularly I think Godot with DRL has a great future ahead. I just gave you some advice not to repeat what I think is a MLAgent design mistake. Copy Barracuda is a mistake :)~. Thanks.

[GeorgeS2019]

Copy Barracuda is a mistake :)

Thx, now I get it. I tend to get second opinion beyond one source. When bringing ideas from different sources, that is when I understand and learn.

[fire]

I posted a standalone ONNX prototype of https://github.com/V-Sekai-fire/ML_fire_juan_mosaic that load ONNX model "filters".

The unfinished design is to use operating system native GPU drivers and put into a GDExtension or a Godot Engine C++ module.

1. Use Directml and CPU on Windows for the widest GPU support.
2. Use CPU and Cuda on Linux
3. Use CPU and AMD on Linux
4. Use CPU and ??? on Mac