Problems with AIController3D

[Miner34dev]

Hello, i'm trying to create a simple RL 3D platformer, but it never works. I fixed the errors in my code but then the script "sync.gd" started having a problem during initialization. Can somebody help me?

[edbeeching]

Perhaps it is this issue, mentioned at the bottom of the custom env tutorial?

NOTE: If you receive an error Invalid get index '0' (on base: 'Array[Node]') click on AIController3D and add an AGENT group by clicking on Node, clicking on Groups, typing in AGENT and clicking Add to add the new group.

[Miner34dev]

Ok, now sync.gd works, but i still have problems, now on get_observation(). It says "Attempt to call function 'get_observation' in base 'null instance' on a null instance". Here is my AIController3D extension code:

extends AIController3D

var fw_bw_move = 0.0
var r_l_move = 0.0
var jump = 0
var raycast

func _ready():
    raycast = $root/Node3D/Player/Raycast

func get_obs() -> Dictionary:
    var obs = raycast.get_observation()
    return {"obs":obs}

func get_reward() -> float: 
    return reward

func get_action_space() -> Dictionary:
    return {
        "fw_bw_move" : {
            "size": 1,
            "action_type": "continuous"
        },
        "r_l_move" : {
            "size": 1,
            "action_type": "continuous"
        },
        "jump" : {
            "size": 1,
            "action_type": "discrete"
        },
        }

func set_action(action):    
    fw_bw_move = clamp(action["fw_bw_move"][0], -1.0, 1.0)
    r_l_move = clamp(action["r_l_move"][0], -1.0, 1.0)
    jump = clamp(action["jump"][0], 0.0, 1.0)

I renamed the raycast sensor to "Raycast".

[Miner34dev]

And there is a screenshot of my object hierarcy:

[edbeeching]

Hmm strange. Can you try making your raycast sensor a child of the AIController?

I realize you probably don't want to do this as you want to move relative to the parent, I may have to revert to a change that was made. Originally AIController3D inherited from Node3D, but it was changed at some point and I think it was a mistake.

[Miner34dev]

Ok, if i set the raycast sensor as a child of the AIController it can be detected, but when i start gdrl in my terminal it outputs this:

Error:  No module named 'ray'
Error:  No module named 'sample_factory'
No game binary has been provided, please press PLAY in the Godot editor
waiting for remote GODOT connection on port 11008
connection established
action space {'fw_bw_move': {'action_type': 'continuous', 'size': 1}, 'jump': {'action_type': 'discrete', 'size': 1}, 'r_l_move': {'action_type': 'continuous', 'size': 1}}
observation space {'obs': {'size': [81], 'space': 'box'}}
Using cpu device
/home/miner34/Documenti/Godot/AI/pyvenv/lib/python3.10/site-packages/stable_baselines3/ppo/ppo.py:137: UserWarning: You have specified a mini-batch size of 64, but because the `RolloutBuffer` is of size `n_steps * n_envs = 32`, after every 0 untruncated mini-batches, there will be a truncated mini-batch of size 32
We recommend using a `batch_size` that is a factor of `n_steps * n_envs`.
Info: (n_steps=32 and n_envs=1)
  warnings.warn(
Logging to logs/log/PPO_2

Error:  No module named 'ray'
Error:  No module named 'sample_factory'
No game binary has been provided, please press PLAY in the Godot editor
waiting for remote GODOT connection on port 11008
connection established
action space {'fw_bw_move': {'action_type': 'continuous', 'size': 1}, 'jump': {'action_type': 'discrete', 'size': 1}, 'r_l_move': {'action_type': 'continuous', 'size': 1}}
observation space {'obs': {'size': [81], 'space': 'box'}}
Using cpu device
/home/miner34/Documenti/Godot/AI/pyvenv/lib/python3.10/site-packages/stable_baselines3/ppo/ppo.py:137: UserWarning: You have specified a mini-batch size of 64, but because the `RolloutBuffer` is of size `n_steps * n_envs = 32`, after every 0 untruncated mini-batches, there will be a truncated mini-batch of size 32
We recommend using a `batch_size` that is a factor of `n_steps * n_envs`.
Info: (n_steps=32 and n_envs=1)
  warnings.warn(
Logging to logs/log/PPO_2

And godot says:

Invalid get index '0' (on base 'float')

(At line 37 of AI logic.gd). Why if on fw_bw_move and r_l_move the same command worked? P.S. I tried making jump a float or the clamp a int since there was another error, but nothing changed.

[Miner34dev]

I fixed it changing line 37 from

jump = clamp(action["jump"][0], 0.0, 1.0)

to

jump = clamp(action["jump"], 0, 1)

[Miner34dev]

But what does the terminal warning mean?

[edbeeching]

Most of the examples there are 16 copies of the env, to speed up training. In your case you just have one, so the batch size is probably too large for this configuration. Either make copies of the env or reduce the batch size.

[edbeeching]

I'm glad that you are making progress!

[Miner34dev]

Ok, just the thing i though, i need to configure the AIController. Oh, and i have an idea on how to make the raycast sensor follow the player: I think if you set an AIController as scene root it will work!

[Miner34dev]

Wait, how do i change these values? The AIController properties menu is almost empty!

[Miner34dev]

Ok, AIController as root works, and the raycast sensor follows the player.

[Miner34dev]

But where are those settings? I can never find them.

[edbeeching]

Which settings are you referring to?

[Miner34dev]

Batch size, n_envs...

[edbeeching]

n_envs is calculated automatically from the number of copies of the environment you have made in Godot. See the examples for an example of how to do this.

Batch size is configured in python. If you are using the sb3 backend you have refer to their docs to config it. https://stable-baselines3.readthedocs.io/en/master/

[Miner34dev]

Ok, but to change n_steps and fps?

[edbeeching]

If you want more fine grained control, refer to this [example script] (https://github.com/edbeeching/godot_rl_agents/blob/main/examples/stable_baselines3_example.py)

For the fps, there is a speedup option to enable higher than real time training

[Miner34dev]

No, the examples ran at 60 fps while the my is at 7.5, and this should be configured in godot because i used the same command for both of them (gdrl).

[edbeeching]

The reason you see 7.5 is because of action_repeat. The action is repeated 8 times, so you have an FPS of 60. You can also use --speedup=8 if you want to train at faster than real time.

[Miner34dev]

Ok, but where is that setting? in sync.gd?

[edbeeching]

It is an export of the sync node, but you can also pass it as an argument to gdrl. For example:

gdrl --env=gdrl --env_path=examples/godot_rl_JumperHard/bin/JumperHard.x86_64 --viz --speedup=8

I strongly recommend that you read the Advanced tutorials in the readme and I think that should answer most of your questions.

[edbeeching]

@Miner34dev , did you manage to resolve this? Can I close this issue?

[Miner34dev]

Why is my fps (the fps shown in the terminal) still at 7 after changing action_repeat?

[edbeeching]

Do you change it from the command line or in the editor?

[edbeeching]

@Miner34dev perhaps this is relevant to you: https://github.com/edbeeching/godot_rl_agents_plugin/pull/13

I will push the change to the plugin.

[Miner34dev]

How to update the plugin?

[edbeeching]

You can search for it on Godot assets.

However a simpler solution for you would be to change the following lines in sync.gd on your machine: https://github.com/edbeeching/godot_rl_agents_plugin/pull/13/files#diff-f787f8c54d9f84053c0eb814c447443bcd5f657ea525b827e375ceb51db71477

[Ivan-267]

Hello @Miner34dev, one other way to update the plugin to the latest version available is to download it from the repository, and then copy the addons folder to the Godot project folder (replace the old addons folder):

https://github.com/edbeeching/godot_rl_agents_plugin/tree/main

You may also need to replace the file: Godot RL Agents.csproj with the csproj file in your project (or just change the content of the file to be the same as in the plugin, if your project uses other NuGet packages, you may need to make sure that <PackageReference Include="Microsoft.ML.OnnxRuntime" Version="1.15.1" /> is present in your file).

I will close this issue for now, but feel free to open a new issue if you run into any other issues.