initial research

[yamatokataoka]

• collect information for Replication of Deep Reinforcement Learning from Human Preferences (Christiano et al, 2017).

[yamatokataoka]

• Learning from Human Preferences
• Deep reinforcement learning from human preferences

[yamatokataoka]

implementations

• https://github.com/mrahtz/learning-from-human-preferences
• https://github.com/HumanCompatibleAI/learning-from-human-preferences
• https://github.com/nottombrown/rl-teacher

[yamatokataoka]

final training code would look like this using rl-replicas

class SamplerWithHumanPrefs():
    def __init__(self):

    def sample(self, env: Env, steps: int, reward_predictor, human_pref_collector):

        for :
            human_pref_collector.push(observations)
            prefs = human_pref_collector.collect()
            reward_predictor.train(prefs)
            reward = reward_predictor.reward(observations)

[yamatokataoka]

When do we collect preferences?

• At the beginning of training the reward predictor is pre-trained
• labels are fed in at a rate decaying inversely with the number of timesteps