Versatile modular neural locomotion control with fast learning

Code for M. Thor et al., Versatile modular neural locomotion control with fast learning, 2021, submitted to NMI.

Versatile modular neural locomotion control

Drawing inspiration from animal locomotion, we propose a simple yet versatile modular neural control structure with fast learning. The key advantages of our approach are that behavior-specific control modules can be added incrementally to obtain increasingly complex emergent locomotion behaviors, and that neural connections interfacing with existing modules can be quickly and automatically learned.

System Requirements

This code has been tested with the following hardware and software:

Intel® Core™ i9-9900K CPU @ 3.60GHz × 16
GeForce RTX 2080
Ubuntu 18.04.4 LTS
coppeliaSim V4.1.0
Vortex physics engine¹
Python 3.7.7
ROS melodic

¹ The Vortex physics engine requires a license (which is free for researchers). Alternatively, you can use the Newton physics engine, but for that, we cannot guarantee successful behaviors. Especially, the wall and pipe climbing behaviors does not work well with the Newton physics engine. We hypothesize that this is because the behavior was learned with the Vortex physics engine and because the behavior requires the higher complexity of that engine. To test the remaining behaviors an modified version of the advanced environment called Advanced_newton_env.ttt has been made.

Code overview

The following explains the content of the six main directories:

data: Contains data from running the simulation and PI^BB algorithm (both data used in the result section (in the storage directory) as well as job files used during learning). Additionally, it contains the learned weight sets (as .json files) for the base controller and eight behavior-specific modules presented in the paper. For a more detailed explaination see the README.md file in the data directory.
interfaces: Contains .lua files for interfacing with and setting up the simulation. It also contains the build_dir for cmake.
machine_learning: Contains all the code necessary for the PI^BB learning algorithm as well as bash scripts for running the simulation.
neural_controllers: Contains all the code nesseary for the locomotion controller (i.e., the CPG-RBF network).
simulations: Contains coppeliaSim simulation environments (including the MORF hexapod robot).
utils: Contains additional utilities needed by the controller implementation and simulation interface.

Install

Install time will take 15-30 minutes.

First, we need to set up the simulation (coppeliaSim):

Download coppeliaSim EDU from the downloads page (tested on V4.1.0 / Ubuntu 18.04).
Clone this repository to your local machine _(optional: set $FRAMEWORK_PATH to the path for the directory containing the cloned repository)_.
```
git clone https://github.com/MathiasThor/CPG-RBFN-framework.git
```
Extract the downloaded .zip file into a directory as many times as you need "simulation workers" (i.e., the number of simulations running in parallel. We used four workers in all our experiments).
Rename the extracted coppeliaSim directories as: VREP1, VREP2, VREP3, VREP4, etc. _(optional: set $VREP_WORKER_PATH to the path for the directory containing the workers)_.
In remoteApiConnections.txt in each of the VREP# directories, change portIndex1_port so that VREP1 has 19997, VREP2 has 19996, VREP3 has 19995, VREP4 has 19994, etc.

Copy libv_repExtRosInterface.so into each of the worker directories from the utils directory.

cp $FRAMEWORK_PATH/CPG-RBFN-framework/utils/libv_repExtRosInterface.so $VREP_WORKER_PATH/VREP1/

Install the required python libraries (matplotlib, jupyter, drawnow, and numpy).

cd $FRAMEWORK_PATH/CPG-RBFN-framework/

sudo apt install python3-pip

pip3 install -r requirements.txt

Install LIB GSL.
```
sudo apt-get install libgsl-dev
```

The neural controllers use ROS to communicate with coppeliaSim. Therefore, make sure that ros-xxx-desktop-full (tested on melodic) is installed (ROS install guide)

Run the controller with learned weights

Start an ROS core.
```
roscore
```

Start the simulation workers. In this example, we will only need a single worker.

cd $VREP_WORKER_PATH/VREP1/

./coppeliaSim.sh $FRAMEWORK_PATH/CPG-RBFN-framework/simulations/Advanced_env.ttt

Build the locomotion controller.

cd $FRAMEWORK_PATH/CPG-RBFN-framework/interfaces/morf/sim/build_dir

rm CMakeCache.txt

cmake .

make

Run with the learned weights presented in the paper for 400s.

cd $FRAMEWORK_PATH/CPG-RBFN-framework/machine_learning

./run_sim.sh -t 400

Select option 2 then 9 and finally 0.
It is expected that MORF will be able to traverse all the waypoints in the advanced environment (tested with the Vortex physics engine. See System Requirements¹).

Run learning algorithm

The following will show how to start learning the base controller. It is in this way possible to reproduce the quantitative results in the manuscript.

Start a ROS core.
```
roscore
```

Start the simulation workers. In this example, we will use four workers.

cd $VREP_WORKER_PATH/VREP1/

./coppeliaSim.sh $FRAMEWORK_PATH/CPG-RBFN-framework/simulations/MORF_base_behavior.ttt

cd $VREP_WORKER_PATH/VREP2/

./coppeliaSim.sh $FRAMEWORK_PATH/CPG-RBFN-framework/simulations/MORF_base_behavior.ttt

cd $VREP_WORKER_PATH/VREP3/

./coppeliaSim.sh $FRAMEWORK_PATH/CPG-RBFN-framework/simulations/MORF_base_behavior.ttt

cd $VREP_WORKER_PATH/VREP4/

./coppeliaSim.sh $FRAMEWORK_PATH/CPG-RBFN-framework/simulations/MORF_base_behavior.ttt

Build the locomotion controller.

cd $FRAMEWORK_PATH/CPG-RBFN-framework/interfaces/morf/sim/build_dir

rm CMakeCache.txt

cmake .

make

In $FRAMEWORK_PATH/CPG-RBFN-framework/machine_learning/RL_master.py set the following.
```
workers = 4
behaviour_selector = "walk"
```

Start the learning algorithm.

./RL_repeater.sh -t 1 -e indirect -r MORF

The program will now be running MORF (-r MORF) for 100 iterations one time (-t 1) using an indirect (-e indirect) encoding. Every 5th iteration will be shown visually in coppeliaSim, while all others will be blacked out for performance boost. It is expected that MORF will learn to walk within 20 iterations.

Note that learning the advanced modules requires the user to set the behavior active (i.e., = 1) in neutronController.cpp line 276, 281, 286, 291, or 296.

License

All software is available under the GPL-3 license.

MathiasThor / CPG-RBFN-framework

readme