Eval stats collector - Githubissues

prabhatnagarajan commented 3 years ago

We currently collect basic performance statistics as well as agent statistics. This PR allows environment statistics to be collected. This gives a lot of flexibility. For example, if you have a binary success task such as grasping, your environment could give a "success" statistic, in addition to the reward. If you have a finance domain, your environment could track "profit" separate from the reward function.

The environment optionally implements two functions: get_statistics and clear_statistics. If these functions are not implemented, then the evaluator works as it did before (i.e. this PR is backwards compatible).

get_statistics: Returns the environment statistics. Defaults to returning an empty list if the environment doesn't implement it.
clear_statistics: Clears the statistics. This is called before evaluation so that get_statistics only returns statistics collected during evaluation. By default does nothing if clear_statistics isn't implemented by the environment.

prabhatnagarajan commented 3 years ago

/test