nagataka
commented
3 years ago - Bisimulation is an interesting concept. Especially, "Bisimulation Metrics" which softens the concept of state partitions and can be applicable to continuous state spaces sounds quite important.
- Not only the pixel-level disentanglement but the representation considers the downstream control tasks is the good point
- I like Figure-4 as it nicely show that their latent space push similar states into neighboring location
Summary
Link
Learning Invariant Representations for Reinforcement Learning without Reconstruction
Author/Institution
Amy Zhang, Rowan McAllister, Roberto Calandra, Yarin Gal, Sergey Levine UCB, FAIR, McGill, Oxford
What is this
One-sentence It proposes an approach of representation learning for RL to focus on task-relevant features while ignoring task-irrelevant ones based on the idea of bisimulation.
Full Appropriate representation could help RL agents to learn faster and also achieve other benefits such as improved generalization. In this work, the authors propose Deep Bisimulation for Control (DBC) to learn RL control and also representation at the same time. Bisimulation metric provides a measurement of the similarity of two states based on the reward and state transition dynamics. The idea of this work is that the l1 distance of two latent state representation should approximate the bisimulation metric of the two states. One nice thing of this approach is that since bisimulation metric will consider task-relevant information only, the constraint or regularization on latent state representation learning drives the representation to ignore task-irrelevant features. This is well demonstrated by CARLA tasks. Another thing to notice is that they used iterative update (policy, environment model and representation) in implementation. Sometimes engineering tricks are important to make your fancy idea really work.
Comparison with previous researches. What are the novelties/good points?
Compared with other representation learning approaches such as reconstruction-based or contrastive learning based approaches.
Key points
Bisimulation metric
How the author proved effectiveness of the proposal?
Any discussions?
It's actually related to our work Domain Adaptation In Reinforcement Learning Via Latent Unified State Representation . Our approach could be categorized into reconstruction-based state representation learning approaches. I agree representation learning could matter a lot for RL. Bisimulation is also interesting and have a good potential for further research. It could have more utilization in RL.
Their idea is simple but interesting. Their section 5 of proof could be a good plus. Their experiments are also valid.
What should I read next?
Learning continuous latent space models for representation learning Scalable methods for computing state similarity in deterministic Markov decision processes.