NeurIPS 2022 | Disentangling Transfer in Continual Reinforcement Learning

[chufanchen]

https://arxiv.org/abs/2209.13900

[chufanchen]

What enables transfer between RL tasks?

Use multi-head SAC(actor, critic, exploration policy) on Continual World.

• Critic is the most important role for FT.
• transfer independence of the components

Transfer in CL

• Fine-tuning
• Perfect memory
• EWC
• PackNet
• L2
• A-GEM
• MAS
• VCL

Exploration policy

• random
• uniform-previous
• preceding
• best-return

Data rehearsal

• Perfect memory
• Behavioral cloning

BC improves transfer in long-sequence scenario, but not in two-task scenario.

Regularize the critic

Regularize the critic deteriorates performance. The practical recommendation is to regularize only the actor.

[chufanchen]

Metrics

Average performance

Forward transfer

Forgetting

[chufanchen]

Method

ClonEx-SAC: behavioral cloning, improved exploration and SAC.

• behavioral cloning for the actor
• best-return exploration
• not use any CL regularization for the critic
• multiple output heads for both actor and critic