Trinkle23897
commented
4 years ago In supervised learning, the meanings of epoch and step_per_epoch (# of data samples/batch_size) are clear, but in reinforcement learning, it seems vague since we don't have a fixed dataset. Some algorithms are trained per episode, e.g., REINFORCE and some are trained per step, e.g., DQN. As a result, I am quite confused about the exact meanings of epoch and step_per_epoch here. Would you please better explain the connections between these concepts?
I think @danagi 's definition in #108 is quite clear: https://github.com/thu-ml/tianshou/pull/108#issuecomment-653433518
A global step or an algorithmic step, consists of collect step and update step. In other words,
an epoch with n steps that has m collect steps and t update steps, would collect total n*m frames
and update network total n*t times.But currently, the step means a policy network update. You can refer to https://tianshou.readthedocs.io/en/latest/api/tianshou.trainer.html#tianshou.trainer.offpolicy_trainer
Besides, I am also interested in how you evaluate algorithms. Do you evaluate them every few learning steps or something else? I see the number of test episodes is 100 as one of the arguments but what is the evaluation frequency?
Currently, it evaluates the algorithms when it reaches the given reward threshold in the training phase. In other words, if the collector collects some episodes by the current policy in the training phase, and it finds that the mean reward of these episodes is above the threshold (this is a prior), it will test this policy using 100 episodes to see if it is really above the reward threshold.
familyld
In supervised learning, the meanings of epoch and step_per_epoch (# of data samples/batch_size) are clear, but in reinforcement learning, it seems vague since we don't have a fixed dataset. Some algorithms are trained per episode, e.g., REINFORCE and some are trained per step, e.g., DQN. As a result, I am quite confused about the exact meanings of epoch and step_per_epoch here. Would you please better explain the connections between these concepts?
Besides, I am also interested in how you evaluate algorithms. Do you evaluate them every few learning steps or something else? I see the number of test episodes is 100 as one of the arguments but what is the evaluation frequency?
Thank you.