RLLib

(C++ Template Library to Predict, Control, Learn Behaviors, and Represent Learnable Knowledge using On/Off Policy Reinforcement Learning)

RLLib is a lightweight C++ template library that implements incremental, standard, and gradient temporal-difference learning algorithms in Reinforcement Learning. It is an optimized library for robotic applications and embedded devices that operates under fast duty cycles (e.g., < 30 ms). RLLib has been tested and evaluated on RoboCup 3D soccer simulation agents, physical NAO V4 humanoid robots, and Tiva C series launchpad microcontrollers to predict, control, learn behaviors, and represent learnable knowledge. The implementation of the RLLib library is inspired by the RLPark API, which is a library of temporal-difference learning algorithms written in Java.

Features

• Off-policy prediction algorithms:
  • GTD(λ)
  • GTD(λ)True
  • GQ(λ)
• Off-policy control algorithms:
  • Q(λ)
  • Greedy-GQ(λ)
  • Softmax-GQ(λ)
  • Off-PAC (can be used in on-policy setting)
• On-policy algorithms:
  • TD(λ)
  • TD(λ)AlphaBound
  • TD(λ)True
  • Sarsa(λ)
  • Sarsa(λ)AlphaBound
  • Sarsa(λ)True
  • Sarsa(λ)Expected
  • Actor-Critic (continuous actions, discrete actions, discounted reward settting, averaged reward settings, and so on)
• Supervised learning algorithms:
  • Adaline
  • IDBD
  • K1
  • SemiLinearIDBD
  • Autostep
• Policies: Random RandomX%Bias Greedy Epsilon-greedy Boltzmann Normal Softmax
• Dot product: An efficient implementation of the dot product for tile coding based feature representations (with culling traces).
• Benchmarking environments: Mountain Car Mountain Car 3D Swinging Pendulum Continuous Grid World Bicycle Cart Pole Acrobot Non-Markov Pole Balancing Helicopter
• Optimization: Optimized for very fast duty cycles (e.g., with culling traces, RLLib has been tested on the Robocup 3D simulator agent, and on the NAO V4 (cognition thread)).
• Usage: The algorithm usage is very much similar to RLPark, therefore, swift learning curve.
• Examples: There are a plethora of examples demonstrating on-policy and off-policy control experiments.
• Visualization: We provide a Qt4 based application to visualize benchmark problems.

Build

New: OpenAI Gym Binding

Open AI Gym is a toolkit for developing and comparing reinforcement learning algorithms. We have developed a bridge between Gym and RLLib to use all the functionalities provided by Gym, while writing the agents (on/off-policy) in RLLib. The directory, openai_gym, contains our bridge as well as RLLib agents that learn and control the classic control environments.

Extension

• Extension for Tiva C Series EK-TM4C123GXL LaunchPad, and Tiva C Series TM4C129 Connected LaunchPad microcontrollers.

• Tiva C series launchpad microcontrollers: https://github.com/samindaa/csc688

Demo

Usage

RLLib is a C++ template library. The header files are located in the include directly. You can simply include/add this directory from your projects, e.g., -I./include, to access the algorithms.

To access the control algorithms:

#include "ControlAlgorithm.h"

To access the predication algorithms:

#include "PredictorAlgorithm"

To access the supervised learning algorithms:

#include "SupervisedAlgorithm.h"

RLLib uses the namespace:

using namespace RLLib

Testing

RLLib provides a flexible testing framework. Follow these steps to quickly write a test case.

• To access the testing framework: #include "HeaderTest.h"

#include "HeaderTest.h"

RLLIB_TEST(YourTest)

class YourTest Test: public YourTestBase
{
  public:
    YourTestTest() {}

    virtual ~Test() {}
    void run();

  private:
    void testYourMethod();
};

void YourTestBase::testYourMethod() {/** Your test code */}

void YourTestBase::run() { testYourMethod(); }

• Add YourTest to the test/test.cfg file.
• You can use @YourTest to execute only YourTest. For example, if you need to execute only MountainCar test cases, use @MountainCarTest.

Test Configuration

We are using CMAKE >= 2.8.7 to build and run the test suite.

• mkdir build
• cd build; cmake ..
• make -j

Visualization

RLLib provides a QT5 based Reinforcement Learning problems and algorithms visualization tool named RLLibViz. Currently RLLibViz visualizes following problems and algorithms:

• On-policy:

  • SwingPendulum problem with continuous actions. We use AverageRewardActorCritic algorithm.

• Off-policy:

  • ContinuousGridworld and MountainCar problems with discrete actions. We use Off-PAC algorithm.

• In order to run the visualization tool, you need to have QT4.8 installed in your system.

• In order to install RLLibViz:

  • Change directory to visualization/RLLibViz
  • qmake RLLibViz.pro
  • make -j
  • ./RLLibViz

Documentation

• http://web.cs.miami.edu/home/saminda/rllib.html
• mloss.org

Operating Systems

• Ubuntu >= 11.04
• Windows (Visual Studio 2013)
• Mac OS X

TODO

• Variable action per state.
• Non-linear algorithms.
• Deep learning algorithms.

Publications

• Dynamic Role Assignment using General ValueFunctions
• Humanoid Robots and Spoken Dialog Systems for Brief Health Interventions

Contact

Saminda Abeyruwan, PhD (saminda@cs.miami.edu, samindaa@gmail.com)