search

HumanCompatibleAI / evaluating-rewards

Library to compare and evaluate reward functions
https://arxiv.org/abs/2006.13900
Apache License 2.0
61 stars 7 forks source link

Explicitly specify discount rate for reward models #22

Closed AdamGleave closed 4 years ago

AdamGleave commented 4 years ago

Previously, the discount rate of potential shaping in reward models was specified on an ad-hoc basis, typically in the constructor arguments for relevant reward models. This was problematic for two reasons:

  1. Since there was no standard interface, experiments that depend on discount rate could not ensure it was set unifiormly amongst reward models.
  2. Furthermore, any model registered to be accessible with serialize.load_reward had a fixed discount rate.

This PR addresses this issue by adding a discount property and set_discount function to RewardModel, and adding a discount parameter to load_reward. It is backward compatible: the default implementations are no-ops, suitable for any reward model that does not include explicit shaping.

If I were to rearchitect this from scratch, I'd probably favour adding discount to the constructor arguments of RewardModel. This would not be too hard to do with the current code. Apart from changing the RewardModel classes, it would also require changing evaluating_rewards.serialize, and would break the API compatibility with imitation for serialization. Given this I'll punt this for some possibly-future rearchitect, e.g. if we ever merge this with imitation or switch to TF2/PyTorch.

codecov[bot] commented 4 years ago

Codecov Report

Merging #22 into master will not change coverage. The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff            @@
##            master       #22   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           11        11           
  Lines          540       552   +12     
=========================================
+ Hits           540       552   +12
Impacted Files Coverage Δ
tests/test_canonical_sample.py 100.00% <100.00%> (ø)
tests/test_comparisons.py 100.00% <100.00%> (ø)
tests/test_rewards.py 100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 796a545...5de690c. Read the comment docs.