search

Unity-Technologies / ml-agents

The Unity Machine Learning Agents Toolkit (ML-Agents) is an open-source project that enables games and simulations to serve as environments for training intelligent agents using deep reinforcement learning and imitation learning.
https://unity.com/products/machine-learning-agents
Other
17.12k stars 4.15k forks source link

GAIL from state observations only #5869

Closed Paul-Hubert closed 1 year ago

Paul-Hubert commented 1 year ago

Description When making a humanoid physical ragdoll with joint movement with RL, many papers use GAIL to stylize and help accelerate training with motion datasets. However humanoid motion datasets do not include the actions that go with the states, i.e the actual joint torque to get to the next state. In this case, papers train GAIL on a transition of states [s(t),s(t+dt)], introduced with this paper : https://arxiv.org/abs/1807.06158

An example paper of what I am trying to implement : https://arxiv.org/abs/2104.02180

Solution If this is not already possible, it would be great to have the option to only input state transitions to GAIL.

Alternatives To encode an input state transition it may be possible just by changing the vector observation/stacked vectors to 2. However remains that the actions would be input to the gail network. If the actions are not set in the heuristics function with ActionBuffers while demonstrating (only set the state kinematically like with a walk animation), then will the actions still be passed to the discriminator ?

Another solution is to estimate the correct actions to produce, such as the joint target rotation, but these actions will not be correct as just reproducing the actions will not lead to the desired action (will just fall over and walk sideways).

Otherwise, the only solution would be to reimplement it all in python.

Anyway, I may be missing context on the specifics of GAIL and its implementation, and this actually might already be possible. In this case, how and where can I find info about it ?

Paul-Hubert commented 1 year ago

Just saw in the code that it is possible to use gail->use_actions : false to do this. It's actually written here : https://github.com/Unity-Technologies/ml-agents/blob/develop/docs/Training-Configuration-File.md#gail-intrinsic-reward But I couldn't find this file by searching in google or github.

github-actions[bot] commented 1 year ago

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.