Inspired by the long-term evolution (LTE) standard project in telecommunications, aiming to provide development components for and standards for advancing RL research and applications. Beyond delivering top-notch algorithm implementations, RLLTE also serves as a toolkit for developing algorithms.
Why RLLTE?
β οΈ Since the construction of RLLTE Hub requires massive computing power, we have to upload the training datasets and model weights gradually. Progress report can be found in Issue#30.
See the project structure below:
For more detailed descriptions of these modules, see API Documentation.
recommended
Open a terminal and install rllte with pip
:
conda create -n rllte python=3.8 # create an virtual environment
pip install rllte-core # basic installation
pip install rllte-core[envs] # for pre-defined environments
Open a terminal and clone the repository from GitHub with git
:
git clone https://github.com/RLE-Foundation/rllte.git
pip install -e . # basic installation
pip install -e .[envs] # for pre-defined environments
For more detailed installation instruction, see Getting Started.
RLLTE provides implementations for well-recognized RL algorithms and simple interface for building applications.
Suppose we want to use DrQ-v2 to solve a task of DeepMind Control Suite, and it suffices to write a train.py
like:
# import `env` and `agent` module
from rllte.env import make_dmc_env
from rllte.agent import DrQv2
if __name__ == "__main__":
device = "cuda:0"
# create env, `eval_env` is optional
env = make_dmc_env(env_id="cartpole_balance", device=device)
eval_env = make_dmc_env(env_id="cartpole_balance", device=device)
# create agent
agent = DrQv2(env=env, eval_env=eval_env, device=device, tag="drqv2_dmc_pixel")
# start training
agent.train(num_train_steps=500000, log_interval=1000)
Run train.py
and you will see the following output:
Similarly, if we want to train an agent on HUAWEI NPU, it suffices to replace cuda
with npu
:
device = "cuda:0" -> device = "npu:0"
Developers only need three steps to implement an RL algorithm with RLLTE. The following example illustrates how to write an Advantage Actor-Critic (A2C) agent to solve Atari games.
Firstly, select a prototype:
Secondly, select necessary modules to build the agent:
Thirdly, merge these modules and write an .update
function:
Finally, train the agent by
As shown in this example, only a few dozen lines of code are needed to create RL agents with RLLTE.
RLLTE allows developers to replace settled modules of implemented algorithms to make performance comparison and algorithm improvement, and both
built-in and custom modules are supported. Suppose we want to compare the effect of different encoders, it suffices to invoke the .set
function:
from rllte.xploit.encoder import EspeholtResidualEncoder
encoder = EspeholtResidualEncoder(...)
agent.set(encoder=encoder)
RLLTE is an extremely open framework that allows developers to try anything. For more detailed tutorials, see Tutorials.
Type | Algo. | Box | Dis. | M.B. | M.D. | M.P. | NPU | π° | π |
---|---|---|---|---|---|---|---|---|---|
On-Policy | A2C | βοΈ | βοΈ | βοΈ | βοΈ | βοΈ | βοΈ | βοΈ | β |
On-Policy | PPO | βοΈ | βοΈ | βοΈ | βοΈ | βοΈ | βοΈ | βοΈ | β |
On-Policy | DrAC | βοΈ | βοΈ | βοΈ | βοΈ | βοΈ | βοΈ | βοΈ | βοΈ |
On-Policy | DAAC | βοΈ | βοΈ | βοΈ | βοΈ | βοΈ | βοΈ | βοΈ | β |
On-Policy | DrDAAC | βοΈ | βοΈ | βοΈ | βοΈ | βοΈ | βοΈ | βοΈ | βοΈ |
On-Policy | PPG | βοΈ | βοΈ | βοΈ | β | βοΈ | βοΈ | βοΈ | β |
Off-Policy | DQN | βοΈ | β | β | β | βοΈ | βοΈ | βοΈ | β |
Off-Policy | DDPG | βοΈ | β | β | β | βοΈ | βοΈ | βοΈ | β |
Off-Policy | SAC | βοΈ | β | β | β | βοΈ | βοΈ | βοΈ | β |
Off-Policy | SAC-Discrete | β | βοΈ | β | β | βοΈ | βοΈ | βοΈ | β |
Off-Policy | TD3 | βοΈ | β | β | β | βοΈ | βοΈ | βοΈ | β |
Off-Policy | DrQ-v2 | βοΈ | β | β | β | β | βοΈ | βοΈ | βοΈ |
Distributed | IMPALA | βοΈ | βοΈ | β | β | βοΈ | β | β | β |
Dis., M.B., M.D.
:Discrete
,MultiBinary
, andMultiDiscrete
action space;M.P.
: Multi processing;- π: Developing;
- π°: Support intrinsic reward shaping;
- π: Support observation augmentation.
Type | Modules |
---|---|
Count-based | PseudoCounts, RND, E3B |
Curiosity-driven | ICM, GIRM, RIDE, Disagreement |
Memory-based | NGU |
Information theory-based | RE3, RISE, REVD |
See Tutorials: Use Intrinsic Reward and Observation Augmentation for usage examples.
Explore the ecosystem of RLLTE to facilitate your project:
Welcome to contribute to this project! Before you begin writing code, please read CONTRIBUTING.md for guide first.
To cite this project in publications:
@article{yuan2023rllte,
title={RLLTE: Long-Term Evolution Project of Reinforcement Learning},
author={Mingqi Yuan and Zequn Zhang and Yang Xu and Shihao Luo and Bo Li and Xin Jin and Wenjun Zeng},
year={2023},
journal={arXiv preprint arXiv:2309.16382}
}
This project is supported by The Hong Kong Polytechnic University, Eastern Institute for Advanced Study, and FLW-Foundation. EIAS HPC provides a GPU computing platform, and HUAWEI Ascend Community provides an NPU computing platform for our testing. Some code of this project is borrowed or inspired by several excellent projects, and we highly appreciate them. See ACKNOWLEDGMENT.md.