search

Unity-Technologies / ml-agents

The Unity Machine Learning Agents Toolkit (ML-Agents) is an open-source project that enables games and simulations to serve as environments for training intelligent agents using deep reinforcement learning and imitation learning.
https://unity.com/products/machine-learning-agents
Other
17.18k stars 4.16k forks source link

Getting all RL output values in Unity/Python #2527

Closed Zilch123 closed 5 years ago

Zilch123 commented 5 years ago

Please forgive for unsuitable vocabulary.

Is it possible to get all the values of the nn output ? In unity/Python. Is there an unity/Python function to get all the outputs. What mean as output is the each action which maximises the reward. I would also need other action values for which reward might be less or zero. image

Zilch123 commented 5 years ago

https://github.com/Unity-Technologies/ml-agents/blob/master/docs/Python-API.md

https://github.com/Unity-Technologies/ml-agents/blob/master/docs/Learning-Environment-Design-Agents.md

chriselion commented 5 years ago

Sorry for the delay, glad you were able to find the answers.

Zilch123 commented 5 years ago

Sorry for the delay, glad you were able to find the answers.

No Problem @chriselion, I've a solution of implementing an RL algorithm in GYM and use that to get all the output from a NN. I was wondering if you can point me a better way, I mean directly getting them in Unity by tinkering the of output function or something?

chriselion commented 5 years ago

If you mean where the actions are decided, this happens in this part of the code: https://github.com/Unity-Technologies/ml-agents/blob/fba79899eba43442af00524acab8ee220d800087/ml-agents/mlagents/trainers/tf_policy.py#L113-L130

For PPO, the implementation is here: https://github.com/Unity-Technologies/ml-agents/blob/34300b906c4dcca4ab0161f8eafd8158ca060fd6/ml-agents/mlagents/trainers/ppo/policy.py#L139-L169

So the actions, memory, and value will be contained in the run_out dictionary.

Zilch123 commented 5 years ago

Thanks a lot @chriselion, I wasn't expecting you to answer me. Thank you! The above will be useful, if I'm to use python to control unity, I was hoping to do it in unity itself through c#. I want to get the raw output of a trained RL agents, just today I came to know that the inference engine is doing this job. https://github.com/Unity-Technologies/ml-agents/tree/master/UnitySDK/Assets/ML-Agents/Scripts/InferenceBrain I will figure out by reading all these Scripts. In case there is any documents of the SDK and inference engine that would be great, please point me. Thank you again!

chriselion commented 5 years ago

The .nn file (and some other files) are produced here: https://github.com/Unity-Technologies/ml-agents/blob/fba79899eba43442af00524acab8ee220d800087/ml-agents/mlagents/trainers/tf_policy.py#L213-L235

The .nn file is a Unity-specific format for the Barracuda inference engine. But the frozen_graph_def.pb is a more standard tensorflow format that you should be able to visualize with standard tools.

Zilch123 commented 5 years ago

Someones have already figured it out #2027 Have to implement import the .nn file (of a game) in C# and implement something similar to this. The Barracuda documentation was what I was looking for. For more of how to use Barracuda Inference Engine look into the Inference Brain . Thank you, Zilch

github-actions[bot] commented 3 years ago

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.