Export model to ONNX

[yaelatletl]

this is a suggestion/request in which I want to contribute, I have started work on this feature (which I have committed to my fork), but I am not well versed on Torch code, though I have gotten to the point where the model gets loaded from the checkpoint, I get an error saying I need to pass a Tensor of shape [...,8] to the torch.onnx.export function

[edbeeching]

Nice work, if you are using rllib this may be of help https://docs.ray.io/en/latest/rllib/package_ref/algorithm.html?highlight=onnx#ray.rllib.algorithms.algorithm.Algorithm.export_policy_model

By the way, I am updating Godot 4 on this branch in addition to more examples and better support for stable baselines and sample-factory: https://github.com/edbeeching/godot_rl_agents/tree/godot4

[yaelatletl]

I made a pull request for this #25 I'd like to help backport features from the godot4 branch, as I actually need to work on godot 3.X anyways Is loading ONNX modules through C# ok?

[edbeeching]

Yes C# is fine, this is great work. Have you managed to run the ONNX model inside Godot?

[yaelatletl]

I've run another model inside Godot before, ResNet50. I just commited the latest changes to the Pull Request, I've loaded the produced Model, though it isn't running yet. And also changed the version output, I was using 15 but the runtime wouldn't recognize it, so I changed it to 9.

[yaelatletl]

The model runs now, and I was able to produce this output: Is this output correct?

[yaelatletl]

Made a GDScript wrapper for this, made it a generalized class and also exported the rest of the checkpoints; the only one I couldn't export was the Space Shooter, it doesn't even run on my end.

Right now the classes are only on the JumperHard example as I still need to integrate it

[yaelatletl]

@edbeeching I've been trying to implement the ONNX model to directtly handle the agents, but it's a bit obscure.
I was actually able to relate the outputs from the model to the outputs from the checkpoint and training.

Godot recieves:setting action 4content: {jump:0, move:[-0.965605], turn:[-1]}
ONNX outputs: {output:[1.830403, -1.823679, -0.965604, -2.522008, -1.960869, -2.00579], state_outs:[0]} 

Godot recieves: setting action 7content: {jump:1, move:[-1], turn:[0.057112]}
ONNX outputs: {output:[-0.059597, 0.059326, -1.319151, -1.642016, 0.057112, -1.164303], state_outs:[0]}

Godot recieves: setting action 9content: {jump:0, move:[-0.582598], turn:[-0.251385]}
ONNX outputs: {output:[3.487688, -3.483186, -0.582597, -2.297049, -0.251385, -2.50098], state_outs:[0]}

The index 2 relates to move and index 4 relates to turn, but I can't wrap my head around the other values, question here is, what are the other values supposed to mean? Which one represents jump?

I'm not sure if the output is complete, but the results are deterministic and coherent so far, so maybe there's something I'm missing? Any advice would be appreciated

[edbeeching]

Hey, I am on paternity leave so I have limited time. I have not forgotten this and I should get an hour to take a look on Saturday evening.

[yaelatletl]

Don't worry, there's no hurry! :D

[yaelatletl]

Technically this has been completed already and can be expanded on, I'll make a new issue on the plugin repo so we can discuss implementation