Awesome Decision Transformer

This is a collection of research papers for Decision Transformer (DT). And the repository will be continuously updated to track the frontier of DT.

Welcome to follow and star!

Table of Contents

• A Taxonomy of DT Algorithms
• Surveys

• Papers

  • Arxiv
  • ICML 2024 (New!!!)
  • ICLR 2024
  • NeurIPS 2023
  • CoRL 2023
  • IROS 2023
  • ICML 2023
  • ICRA 2023
  • ICLR 2023
  • NeurIPS 2022
  • CoRL 2022
  • ICML 2022
  • AAAI 2022
  • ICLR 2022
  • NeurIPS 2021
  • ICML 2021
• Contributing

Overview of Transformer

The Decision Transformer was proposed by “Decision Transformer: Reinforcement Learning via Sequence Modeling” by Chen L. et al. It casts (offline) Reinforcement Learning as a conditional-sequence modeling problem.

Specifically, DT model is a causal transformer model conditioned on the desired return, (past) states, and actions to generate future actions in an autoregressive manner.

Advantage

1. Bypass the need for bootstrapping for long term credit assignment
2. Avoid undesirable short-sighted behaviors due to the discounting future rewards.
3. Enjoy the transformer models widely used in language and vision, which are easy to scale and adapt to multi-modal data.

Surveys

• On Transforming Reinforcement Learning With Transformers: The Development Trajectory

  • Shengchao Hu, Li Shen, Ya Zhang, Yixin Chen, Dacheng Tao
  • Publisher: IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)

• Large sequence models for sequential decision-making: a survey

  • Muning Wen, Runji Lin, Hanjing Wang, Yaodong Yang, Ying Wen, Luo Mai, Jun Wang, Haifeng Zhang, Weinan Zhang
  • Publisher: Frontiers of Computer Science

• A Survey on Transformers in Reinforcement Learning

  • Wenzhe Li, Hao Luo, Zichuan Lin, Chongjie Zhang, Zongqing Lu, Deheng Ye
  • Publisher: Transactions on Machine Learning Research (TMLR)

• Transformers in Reinforcement Learning: A Survey

  • Pranav Agarwal, Aamer Abdul Rahman, Pierre-Luc St-Charles, Simon J.D. Prince, Samira Ebrahimi Kahou

Papers

format:
- [title](paper link) [links]
  - author1, author2, and author3...
  - publisher
  - key 
  - code 
  - experiment environment

Arxiv

• Context-Former: Stitching via Latent Conditioned Sequence Modeling

  • Ziqi Zhang, Jingzehua Xu, Zifeng Zhuang, Jinxin Liu, Donglin wang
  • Key: DT, Latent Conditioned Sequence Modeling
  • ExpEnv: D4RL

• Real-time Network Intrusion Detection via Decision Transformers

  • Jingdi Chen, Hanhan Zhou, Yongsheng Mei, Gina Adam, Nathaniel D. Bastian, Tian Lan
  • Key: DT, Network Intrusion Detection
  • ExpEnv: UNSW-NB15

• Prompt-Tuning Decision Transformer with Preference Ranking

  • Shengchao Hu, Li Shen, Ya Zhang, Dacheng Tao
  • Key: Prompt-Tuning
  • ExpEnv: D4RL

• Graph Decision Transformer

  • Shengchao Hu, Li Shen, Ya Zhang, Dacheng Tao
  • Key: graph transformer
  • ExpEnv: Atari

• Can Offline Reinforcement Learning Help Natural Language Understanding?

  • Ziqi Zhang, Yile Wang, Yue Zhang, Donglin Wang
  • Key: Language model
  • ExpEnv: MuJoco, Maze 2D

• SaFormer: A Conditional Sequence Modeling Approach to Offline Safe Reinforcement Learning

  • Qin Zhang, Linrui Zhang, Haoran Xu, Li Shen, Bowen Wang, Yongzhe Chang, Xueqian Wang, Bo Yuan, Dacheng Tao
  • Key: Offline Safe RL, DT
  • ExpEnv: D4RL

• Offline Pre-trained Multi-Agent Decision Transformer: One Big Sequence Model Tackles All SMAC Tasks

  • Linghui Meng, Muning Wen, Yaodong Yang, Chenyang Le, Xiyun Li, Weinan Zhang, Ying Wen, Haifeng Zhang, Jun Wang, Bo Xu
  • Key: Multi-Agent RL
  • Code: official
  • ExpEnv: SMAC

• Transfer learning with causal counterfactual reasoning in Decision Transformers

  • Ayman Boustati, Hana Chockler, Daniel C. McNamee
  • Key: Causal reasoning, Transfer Learning
  • ExpEnv: MINIGRID

• Pretraining for Language Conditioned Imitation with Transformers

  • Aaron L Putterman, Kevin Lu, Igor Mordatch, Pieter Abbeel
  • Key: Text-Conditioned Decision
  • ExpEnv: Text-Conditioned Frostbite (MultiModal Benchmark)

• An Offline Deep Reinforcement Learning for Maintenance Decision-Making

  • Hamed Khorasgani, Haiyan Wang, Chetan Gupta, Ahmed Farahat
  • Publisher: Annual Conference of the PHM Society 2021
  • Key: Offline Supervised RL, Remaining Useful Life Estimation
  • ExpEnv: NASA C-MAPSS

• A Sequence Modelling Approach to Question Answering in Text-Based Games

  • Gregory Furman, Edan Toledo, Jonathan Shock, Jan Buys
  • Publisher: Proceedings of the 3rd Wordplay: When Language Meets Games Workshop (Wordplay 2022)
  • Key: VQA
  • ExpEnv: QAIT

• Can Wikipedia Help Offline Reinforcement Learning?

  • Machel Reid, Yutaro Yamada, Shixiang Shane Gu
  • Key: VLN, Transfer Learning
  • Code: official
  • ExpEnv: MuJoco, D4RL, Atari

• Switch Trajectory Transformer with Distributional Value Approximation for Multi-Task Reinforcement Learning

  • Qinjie Lin, Han Liu, Biswa Sengupta
  • Key: Multi-Task RL, Sparse Reward
  • ExpEnv: MINIGRID

• Deep Transformer Q-Networks for Partially Observable Reinforcement Learning

  • Kevin Esslinger, Robert Platt, Christopher Amato
  • Key: POMDP, Transformer Q-Learning
  • ExpEnv: GV, Car Flag

• SimStu-Transformer: A Transformer-Based Approach to Simulating Student Behaviour

  • Zhaoxing Li, Lei Shi, Alexandra Cristea, Yunzhan Zhou, Chenghao Xiao, Ziqi Pan
  • Key: Intelligent Tutoring System

• Attention-Based Learning for Combinatorial Optimization

  • Carson Smith
  • Key: Combinatorial Optimization

ICML 2024

• HarmoDT: Harmony Multi-Task Decision Transformer for Offline Reinforcement Learning

  • Shengchao Hu, Ziqing Fan, Li Shen, Ya Zhang, Yanfeng Wang, Dacheng Tao
  • Publisher: ICML 2024
  • Key: Multi-task, DT
  • ExpEnv: MetaWorld

• Q-value Regularized Transformer for Offline Reinforcement Learning

  • Shengchao Hu, Ziqing Fan, Chaoqin Huang, Li Shen, Ya Zhang, Yanfeng Wang, Dacheng Tao
  • Publisher: ICML 2024
  • Key: Q-learning, DT
  • ExpEnv: D4RL

• Temporal Logic Specification-Conditioned Decision Transformer for Offline Safe Reinforcement Learning

  • Zijian Guo, Weichao Zhou, Wenchao Li
  • Publisher: ICML 2024
  • Key: Signal Temporal Logic (STL), DT
  • ExpEnv: DSRL

• Think Before You Act: Decision Transformers with Working Memory

  • Jikun Kang, Romain Laroche, Xingdi Yuan, Adam Trischler, Xue Liu, Jie Fu
  • Publisher: ICML 2024
  • Key: Working Memory, DT
  • ExpEnv: MetaWorld, Atari

• In-Context Decision Transformer: Reinforcement Learning via Hierarchical Chain-of-Thought

  • Sili Huang, Jifeng Hu, Hechang Chen, Lichao Sun, Bo Yang
  • Publisher: ICML 2024
  • Key: Hierarchical Structure, DT
  • ExpEnv: D4RL

• Rethinking Decision Transformer via Hierarchical Reinforcement Learning

  • Yi Ma, Jianye Hao, Hebin Liang, Chenjun Xiao,
  • Publisher: ICML 2024
  • Key: DT, Hierarchical Reinforcement Learning
  • ExpEnv: D4RL

ICLR 2024

• Learning Multi-Agent Communication from Graph Modeling Perspective

  • Shengchao Hu, Li Shen, Ya Zhang, Dacheng Tao
  • Publisher: ICLR 2024
  • Key: Communication, Sequence Modeling
  • ExpEnv: SMAC

• Decision ConvFormer: Local Filtering in MetaFormer is Sufficient for Decision Making

  • Jeonghye Kim, Suyoung Lee, Woojun Kim, Youngchul Sung
  • Key: MetaFormer, Decision ConvFormer
  • ExpEnv: D4RL, Atari

• When should we prefer Decision Transformers for Offline Reinforcement Learning?

  • Prajjwal Bhargava, Rohan Chitnis, Alborz Geramifard, Shagun Sodhani, Amy Zhang
  • Key: offline reinforcement learning, sequence modeling, reinforcement learning
  • ExpEnv: D4RL

• Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretraining

  • Licong Lin, Yu Bai, Song Mei
  • Key: transformers, in-context learning, reinforcement learning, learning theory
  • ExpEnv: [stochastic linear bandit]()

• Searching for High-Value Molecules Using Reinforcement Learning and Transformers

  • Raj Ghugare, Santiago Miret, Adriana Hugessen, Mariano Phielipp, Glen Berseth
  • Key: chemistry, reinforcement learning, language models
  • ExpEnv: [docking and pytdc tasks]()

NeurIPS 2023

• Structured state space models for in-context reinforcement learning

  • Chris Lu, Yannick Schroecker, Albert Gu, Emilio Parisotto, Jakob Foerster, Satinder Singh, Feryal Behbahani
  • Publisher: NeurIPS 2023
  • Key: state space model, in-context RL
  • ExpEnv: [t-maze]()

• Supervised Pretraining Can Learn In-Context Reinforcement Learning

  • Jonathan N. Lee, Annie Xie, Aldo Pacchiano, Yash Chandak, Chelsea Finn, Ofir Nachum, Emma Brunskill
  • Publisher: NeurIPS 2023
  • Key: Decision Pretrained Transformer, in-context learning
  • ExpEnv: [Dark Room]()

• Is Feedback All You Need? Leveraging Natural Language Feedback in Goal-Conditioned Reinforcement Learning

  • Sabrina McCallum, Max Taylor-Davies, Stefano V. Albrecht, Alessandro Suglia
  • Publisher: NeurIPS 2023 Workshop
  • Key: DT, language feedback
  • ExpEnv: BabyAI

• STEVE-1: A Generative Model for Text-to-Behavior in Minecraft

  • Shalev Lifshitz, Keiran Paster, Harris Chan, Jimmy Ba, Sheila McIlraith
  • Publisher: NeurIPS 2023
  • Key: instruction-tuned Video Pretraining
  • ExpEnv: [Minecraft]()

• Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selection

  • Yu Bai, Fan Chen, Huan Wang, Caiming Xiong, Song Mei
  • Publisher: NeurIPS 2023
  • Key: in-context learning, transformers, deep learning theory, learning theory
  • ExpEnv: [in-context regression problems]()

• Elastic Decision Transformer

  • Yueh-Hua Wu, Xiaolong Wang, Masashi Hamaya
  • Publisher: NeurIPS 2023
  • Key: Offline RL, stitch trajectory, Multi-Task
  • ExpEnv: D4RL

• Learning to Modulate pre-trained Models in RL

  • Thomas Schmied, Markus Hofmarcher, Fabian Paischer, Razvan Pascanu, Sepp Hochreiter
  • Publisher: NeurIPS 2023 (Poster)
  • Key: reinforcement learning, multi-task learning, continual learning, fine-tuning
  • ExpEnv: MetaWorld, DMControl

CoRL 2023

• Transformers are Adaptable Task Planners

  • Vidhi Jain, Yixin Lin, Eric Undersander, Yonatan Bisk, Akshara Rai
  • Publisher: CoRL 2023
  • Key: Task Planning, Prompt, Control, Generalization
  • Code: official
  • ExpEnv: Dishwasher Loading

• Q-Transformer

  • Yevgen Chebotar, Quan Vuong, Alex Irpan, Karol Hausman, Fei Xia, Yao Lu, Aviral Kumar, Tianhe Yu, Alexander Herzog, Karl Pertsch, Keerthana Gopalakrishnan, Julian Ibarz, Ofir Nachum, Sumedh Sontakke, Grecia Salazar, Huong T Tran, Jodilyn Peralta, Clayton Tan, Deeksha Manjunath, Jaspiar Singht, Brianna Zitkovich, Tomas Jackson, Kanishka Rao, Chelsea Finn, Sergey Levine
  • Publisher: CoRL 2023
  • Key: Reinforcement Learning, Offline RL, Transformers, Q-Learning, Robotic Manipulation
  • Code: Unofficial
  • ExpEnv: None

IROS 2023

• Hierarchical Decision Transformer

  • André Correia, Luís A. Alexandre
  • Publisher: IROS 2023
  • Key: Hierarchical Learning, Imitation Learning
  • ExpEnv: MuJoco, D4RL, RoboMimic, Maze 2D

• PACT: Perception-Action Causal Transformer for Autoregressive Robotics Pre-Training

  • Rogerio Bonatti, Sai Vemprala, Shuang Ma, Felipe Frujeri, Shuhang Chen, Ashish Kapoor
  • Publisher: IROS 2023
  • Key: Robotics, Pretrain, Multitask, Representation
  • ExpEnv: MuSHR car, Habitat

ICML 2023

• Constrained Decision Transformer for Offline Safe Reinforcement Learning

  • Zuxin Liu, Zijian Guo, Yihang Yao, Zhepeng Cen, Wenhao Yu, Tingnan Zhang, Ding Zhao
  • Publisher: ICML 2023
  • Key: Offline Safe RL, DT
  • ExpEnv: Bullet-Safety-Gym

• Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RL

  • Taku Yamagata, Ahmed Khalil, Raul Santos-Rodriguez
  • Publisher: ICML 2023
  • Key: Q-Learning
  • ExpEnv: D4RL

ICRA 2023

• LATTE: LAnguage Trajectory TransformEr
  • Arthur Bucker, Luis Figueredo, Sami Haddadin, Ashish Kapoor, Shuang Ma, Sai Vemprala, Rogerio Bonatti
  • Publisher: ICRA 2023
  • Key: MultiModal, Robotics
  • Code: official, official
  • ExpEnv: CoppeliaSim

ICLR 2023

• EDGI: Equivariant Diffusion for Planning with Embodied Agents

  • Johann Brehmer, Joey Bose, Pim de Haan, Taco Cohen
  • Publisher: ICLR 2023 Reincarnating RL workshop
  • Key: rich geometric structure, equivariant, conditional generative modeling, representation
  • ExpEnv: None

• Learning to Modulate pre-trained Models in RL

  • Thomas Schmied, Markus Hofmarcher, Fabian Paischer, Razvan Pascanu, Sepp Hochreiter
  • Publisher: ICLR 2023 Reincarnating RL workshop
  • Key: reinforcement learning, multi-task learning, continual learning, fine-tuning
  • ExpEnv: MetaWorld, DMControl

• DeFog: Decision Transformer under Random Frame Dropping

  • Kaizhe Hu, Ray Chen Zheng, Yang Gao, Huazhe Xu
  • Publisher: ICLR 2023
  • Key: Offline RL, POMDP, Frame-Dropping, Practical Application
  • Code: official
  • ExpEnv: MuJoco, D4RL, Atari

NeurIPS 2022

• When does return-conditioned supervised learning work for offline reinforcement learning?

  • David Brandfonbrener, Alberto Bietti, Jacob Buckman, Romain Laroche, Joan Bruna
  • Publisher: NeurIPS 2022
  • Key: Theoretical analysis
  • ExpEnv: MuJoco, D4RL

• You Can't Count on Luck: Why Decision Transformers Fail in Stochastic Environments

  • Keiran Paster, Sheila McIlraith, Jimmy Ba
  • Publisher: NeurIPS 2022
  • Key: Stochastic Environments
  • ExpEnv: Gambling, Connect Four, 2048

• Multi-Agent Reinforcement Learning is a Sequence Modeling Problem

  • Muning Wen, Jakub Grudzien Kuba, Runji Lin, Weinan Zhang, Ying Wen, Jun Wang, Yaodong Yang
  • Publisher: NeurIPS 2022
  • Key: Multi-Agent RL
  • ExpEnv: SMAC, MA MuJoco

• Bootstrapped Transformer for Offline Reinforcement Learning

  • Kerong Wang, Hanye Zhao, Xufang Luo, Kan Ren, Weinan Zhang, Dongsheng Li
  • Publisher: NeurIPS 2022
  • Key: Generation model
  • Code: official
  • ExpEnv: MuJoco, D4RL, Adroit

• Multi-Game Decision Transformers

  • Kuang-Huei Lee, Ofir Nachum, Mengjiao Yang, Lisa Lee, Daniel Freeman, Winnie Xu, Sergio Guadarrama, Ian Fischer, Eric Jang, Henryk Michalewski, Igor Mordatch
  • Publisher: NeurIPS 2022
  • Key: Multi-Task, Finetuning
  • Code: official
  • ExpEnv: Atari, REM

• Decision making as language generation

  • Roland Memisevic, Sunny Panchal, Mingu Lee
  • Publisher: NeurIPS 2022 Workshop FMDM
  • Key: Generation
  • ExpEnv: Traversals (Toy experiment)

CoRL 2022

• Offline Reinforcement Learning for Customizable Visual Navigation

  • Dhruv Shah, Arjun Bhorkar, Hrishit Leen, Ilya Kostrikov, Nicholas Rhinehart, Sergey Levine
  • Publisher: CoRL 2022 (Oral)
  • Key: Visual Navigation
  • ExpEnv: RECON

• Instruction-driven history-aware policies for robotic manipulations

  • Pierre-Louis Guhur, Shizhe Chen, Ricardo Garcia, Makarand Tapaswi, Ivan Laptev, Cordelia Schmid
  • Publisher: CoRL 2022 (Oral)
  • Key: Robotics, Language Instruction
  • Code: official
  • ExpEnv: RLBench

• Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation

  • Mohit Shridhar, Lucas Manuelli, Dieter Fox
  • Publisher: CoRL 2022
  • Key: Robotics, Language Grounding, Behavior Cloning
  • Code: official
  • ExpEnv: RLBench

ICML 2022

• Online Decision Transformer

  • Qinqing Zheng, Amy Zhang, Aditya Grover
  • Publisher: ICML 2022 (Oral)
  • Key: Online finetuning, Max-entropy, Exploration
  • Code: unofficial
  • ExpEnv: MuJoco, D4RL

• Prompting Decision Transformer for Few-Shot Policy Generalization

  • Mengdi Xu, Yikang Shen, Shun Zhang, Yuchen Lu, Ding Zhao, Joshua B. Tenenbaum, Chuang Gan
  • Publisher: ICML 2022 (Poster)
  • Key: Prompt, Few-shot, Generalization
  • Code: official (released soon)
  • ExpEnv: DMC

• Addressing Optimism Bias in Sequence Modeling for Reinforcement Learning

  • Adam R Villaflor, Zhe Huang, Swapnil Pande, John M Dolan, Jeff Schneider
  • Publisher: ICML 2022 (Poster)
  • Key: World model
  • Code: official (released soon)
  • ExpEnv: CARLA

• AnyMorph: Learning Transferable Polices By Inferring Agent Morphology

  • Brandon Trabucco, Mariano Phielipp, Glen Berseth
  • Publisher: ICML 2022 (Poster)
  • Key: Morphology, Transfer Learning, Zero Shot
  • ExpEnv: Modular-RL

AAAI 2022

• Dreaming with Transformers
  • Catherine Zeng, Jordan Docter, Alexander Amini, Igor Gilitschenski, Ramin Hasani, Daniela Rus
  • Publisher: AAAI 2022 (RLG Workshop)
  • Key: Dreamer, World Model
  • ExpEnv: Deepmind Lab, VISTA

ICLR 2022

• Learning Transferable Policies By Inferring Agent Morphology

  • Brandon Trabucco, Mariano Phielipp, Glen Berseth
  • Publisher: ICLR 2022 (GPL Workshop Poster)
  • Key: Morphology, Transfer Learning, Zero Shot
  • ExpEnv: Modular-RL

• Silver-Bullet-3D at ManiSkill 2021: Learning-from-Demonstrations and Heuristic Rule-based Methods for Object Manipulation

  • Yingwei Pan, Yehao Li, Yiheng Zhang, Qi Cai, Fuchen Long, Zhaofan Qiu, Ting Yao, Tao Mei
  • Publisher: ICLR 2022 (GPL Workshop Poster)
  • Key: Object Manipulation
  • Code: official
  • ExpEnv: ManiSkill

• Generalized Decision Transformer for Offline Hindsight Information Matching

  • Hiroki Furuta, Yutaka Matsuo, Shixiang Shane Gu
  • Publisher: ICLR 2021 (Spotlight)
  • Key: HIM, SMM
  • Code: official
  • ExpEnv: MuJoco, D4RL

NeurIPS 2021

• Decision Transformer: Reinforcement Learning via Sequence Modeling

  • Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, Igor Mordatch
  • Publisher: NeurIPS 2021 (Poster)
  • Key: Conditional sequence modeling
  • Code: official, DI-engine
  • ExpEnv: MuJoco, D4RL, Atari

• Offline Reinforcement Learning as One Big Sequence Modeling Problem

  • Michael Janner, Qiyang Li, Sergey Levine
  • Publisher: NeurIPS 2021 (Spotlight)
  • Key: Conditional sequence modeling, Discretization
  • Code: official
  • ExpEnv: MuJoco, D4RL

• TransDreamer: Reinforcement Learning with Transformer World Models

  • Chang Chen, Yi-Fu Wu, Jaesik Yoon, Sungjin Ahn
  • Publisher: NeurIPS 2021 (Deep RL Workshop)
  • Key: Dreamer, World Model
  • ExpEnv: Hidden Order Discovery, DMC, Atari

ICML 2021

• Reinforcement learning as one big sequence modeling problem
  • Michael Janner, Qiyang Li, Sergey Levine
  • Publisher: ICML workshop
  • Key: Conditional sequence modeling, Discretization
  • Code: official
  • ExpEnv: MuJoco, D4RL

Contributing

Our purpose is to make this repo even better. If you are interested in contributing, please refer to HERE for instructions in contribution.

License

Awesome Decision Transformer is released under the Apache 2.0 license.