Rationale for KL decay

[Rolv-Arild]

Hi, just a small question about the choice of making the KL term decay in importance.

In the paper you describe that high values of the KL divergence coefficient limit the policy, while low values make it forget useful stuff from the BC model. Thus you set it to decay gradually. I'm just wondering, can't this lead to having kind of the worst of both worlds, where it only hits a good range for a brief period? Did you try setting it to a constant in-between value instead of decaying?

[brandonhoughton]

The thinking is you get rotations at the beginning of training that encourage more general refinement later in training. Not sure about the ablation you proposed being run

[Rolv-Arild]

Did you try any other variations? Lower starting coefficient? Faster decay?