search

Unity-Technologies / ml-agents

The Unity Machine Learning Agents Toolkit (ML-Agents) is an open-source project that enables games and simulations to serve as environments for training intelligent agents using deep reinforcement learning and imitation learning.
https://unity.com/products/machine-learning-agents
Other
16.93k stars 4.13k forks source link

BC: Should Cloning Loss be high or low during training? #2468

Closed tillmusshoff closed 4 years ago

tillmusshoff commented 5 years ago

Hi there,

the Cloning Loss drops from 130 to 70 and from 90 to 14 in another example after 4h of offline training. Both graphs are very smooth. What is the ideal value for the cloning loss? The description in the wiki doesn't really help me: "Losses/Cloning Loss (BC) - The mean magnitude of the behavioral cloning loss. Corresponds to how well the model imitates the demonstration data."

Thanks in advance!

vincentpierre commented 5 years ago

Discrete : 

self.loss = tf.reduce_sum(-tf.log(self.action_probs + 1e-10) * self.action_oh

Continuous : 

self.loss = tf.reduce_sum(tf.squared_difference(self.clipped_true_action, self.sample_action))

In both continuous and discrete, the value of the loss will depend on the batch size and on the size of the action space. The ideal value is as low as possible !

chriselion commented 4 years ago

Thank you for the discussion. We are closing this issue due to inactivity. Feel free to reopen it if you’d like to continue the discussion.

github-actions[bot] commented 3 years ago

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.