sini
commented
4 years ago I spoke with the author responsible for this original change and he explained that the reason we removed GetValueEstimate is because it's unavailable when using Heuristic / Player control. It would also require a hook into mlagents_envs that is not necessarily useful when using this API directly.
Basically, value estimates are not part of the typical RL loop.
I know we closed your last request on this issue. I'll bring it up for more discussion in our next team meeting.
Phong13
ruanrothmann
TectonicIP
github-actions[bot]
I have been using ML-Agents to train models for a physically simulated character controller. The project makes extensive use of GetValueEstimate() which was removed from the ml-agents API in version 0.11. Please put it back. Each time the project is upgraded I hack ML-Agents to add it back. It is insanely useful. I would not be able to train these models without it.
What is it used for? The CC has five models:
During Inference the GetValueEstimate() is used to decide if the WalkRun model is failing (character is falling over). Then it is used to decide if the "Getup" or "Tumbling" model should be use next (whichever has the higher value estimate). It is used for considering all transitions between all states. I tried doing this with physical measurements and heuristics, but GetValueEstimate works way better and it is so easy!
During training it is used to train transitions to other states. the "JumpAirbourne" model needs to be able to land in a position that will transition well to the "RunWalk" state. A very easy way to do this is to call RunWalkModel.GetValueEstimate(), and reward the JumpAribourne model based on how well the RunWalk model could continue.
The results are spectacular, but these would not be possible without GetValueEstimate()
Describe the solution you'd like Please put GetValueEstimate back. It is being generated as a side effect of training. It is easy to include it. Please export it from tensorflow and expose it in he API.
Please implement it so that the "policy" and "value_estimate" models can be exported to separate .nn files:
This is important for performance. Executing a model that contains the "value_estimate" takes almost twice as long as executing a model that contains only the "policy". Most of the time I want to query either the "value_estimate" or the "Policy" but rarely both together. It is a huge performance optimization to be able to separate these.
Describe alternatives you've considered The alternatives are very messy.
Very complicated training scenes running extra agents to simulate what the transition-to brain would be doing.
For inference the alternative is to fall back to pre-ML heuristic tricks to try to decide if an action is succeeding or failing. Isn't the point of ML to get away from these very limited techniques?
Train all actions into one giant ML model. This is not appealing because it would take a very long time to train and any changes would require re-training the entire thing and dding new actions would require retraining/re-tuning the whole thing and the reward function training scene would be incredibly complicated.
Another agent could be added to the training scene with a new value-estimator-model. The inputs would be the same observations and the output would be a single float that tries to predict the reward the agent would get. This feels silly given that this model is already being trained as a side effect of PPO. But this approach would be worth the effort if it becomes too hard to hack GetValueEstimate back into the project.