🤖 ⚽ 🏋️ Robocup Gym

How it Works — Get Started — Training — Integration — Docs

This is a project for training low-level RL skills in the Robocup 3D simulation league. Integrated with stable-baselines, and providing Gymnasium-compliant environments, Robocup Gym allows you to easily train RL agents for robotic soccer-based tasks.

⬇️ Installation

1. Singularity/Apptainer

First install singularity (aka apptainer); on ubuntu you can follow the instructions here.

2. Container

Download the zip file here, extract it and put the apptainer folder in the robocup_gym directory.

3. Python Environment

conda create -n robocup python=3.11 

conda activate robocup

pip install -r requirements.txt

# Alternatively, install: pip install gymnasium wandb tensorboard tensorboardX numpy matplotlib pandas stable-baselines3 torch torchvision torchaudio onnx natsorted

[!NOTE] We also use wandb to log aspects of training. Run wandb login to log in before running any training.

[!IMPORTANT]
Also, Use ./run.sh from the repository root directory to run any files instead of python directly

4. Testing the configuration

To test if the configuration is properly set up, run the following:

./run.sh robocup_gym/test/test_configs/is_rcssserver_working.py

If you see [Success], RCSSServer Started Successfully, it is working.

./run.sh robocup_gym/test/test_configs/is_environment_working.py

If you see [Success] The environment seems to work, then everything should be working.

📜 Usage - Running Current Tasks

We have a few pre-made tasks already set up. If you want to use these, the process is quite simple. Simply run the following to start a PPO agent on a simple kick task. You can modify this file as you wish to alter the behaviour, change hyperparameters, and so on.

./run.sh robocup_gym/experiments/v0001_ppo.py

➕ Usage - Your Own Tasks

To use the code to run your own tasks, you need to do the following:

1. Create a task, i.e., a particular environment that has a reward function

For instance, create a file robocup_gym/envs/tasks/env_simple_kick.py with the following code:

from robocup_gym.rl.envs.base_env import BaseEnv
import numpy as np

class EnvSimpleKick(BaseEnv):
    def _get_reward(self, action: np.ndarray) -> float:
        return np.linalg.norm(np.array(self.python_agent_conn.player.real_ball_pos) - np.array(self.env_config.ball_start_pos))

2. Create a main, runner file

For instance, paste the following code in robocup_gym/experiments/hello_world.py

import os
import time
from robocup_gym.rl.envs.configs.default import DefaultConfig
from robocup_gym.rl.base_rl_loop import run_experiment
from robocup_gym.rl.envs.tasks.env_simple_kick import EnvSimpleKick

def main():
    os.system("killall -9 rcssserver3d")
    time.sleep(1)
    run_experiment('a000000', EnvSimpleKick, env_kwargs={'env_config':DefaultConfig})

if __name__ == '__main__':
    main()

3. Run the code

Run the following command:

./run.sh robocup_gym/experiments/hello_world.py

Then you should have a model training now!

4. Viewing your models during training

We use RoboViz to visualise policies. Please download and build that before continuing. You can run the visualiser ./roboviz.sh to view the models during training. Note, our monitor ports start at 20001, so you may need to run ./roboviz.sh --serverPort=20001 to actually see the training.

[!TIP] If you are training on a remote machine, you must set up an ssh tunnel (in a separate terminal) ssh <user>@<host> -NL 20001:localhost:20001

5. Evaluating your trained policy

To evaluate a trained policy, look at robocup_gym/rl/eval/eval.py. You can run it as follows.

./run.sh robocup_gym/rl/eval/eval.py <checkpoint_id>

where <checkpoint_id> is the first argument to run_experiment. You can run ls save_models to see which ID's are available. By default this file will evaluate the latest checkpoint.

Note that the evaluation environment exposes the visualisation on port 20500.

🤓 How it works

📖 Docs

🔎 Tips And Tricks

• Explanation
  • GoodConfigVelocitiesOrientationBall is the config that has:
  • Action Space = velocities
  • Observation Space = Most things, including (relative) ball pos and velocity, joint angles and velocities.
  • sleep_time_after_proc_starts = 20 This should be relatively high, as if it is too low the environments will block each other when spawning. There is not really a disadvantage to making it large, except that the construction of the environments (which happens once) will be delayed.
  • wait_steps = 20: This is how long to wait after completing the episode (e.g. to see how far the ball moved). A low value of 20 works well with the kick environments that reward velocity. Note, 50 steps corresponds to 1 second.
  • n_env_procs = 128: How many environments to run in parallel. More is better, but requires more compute. Rule of thumb:
  • 128 procs use around 45GB of RAM.
  • 256 procs use around 90GB of RAM.
  • n_steps = 64. This is basically how many environment steps (per process) PPO does between learning updates. So, n_steps * n_env_procs is how many total steps are done before learning. Therefore, if you double n_env_procs, you should halve n_steps (and so on).

📚 Other Projects Used & Inspiration

• Some of the Python server parsing code and the Player class's structure is from BahiaRT's Gym. This repository also provided inspiration for making our environment.
• Shoutout to the Magma Offenburg team for the great work in Robocup and RL.

👷 Contribution

We welcome community contributions. To contribute, please first install everything as follows:

git clone https://github.com/MichaelBeukman/RobocupGym3D
cd RobocupGym3D
pip install -r requirements.txt
pre-commit install

Make the change and submit a PR!

📜 Citation

Developed by the RAIL lab at the University of the Witwatersrand, Johannesburg, South Africa.

If you use Robocup Gym in your project, please cite it as:

@misc{beukman2024RobocupCode,
  author={Beukman, Michael and Ingram, Branden and Nangue Tasse, Geraud and Rosman, Benjamin and Ranchod, Pravesh},
  title        = {RobocupGym: A challenging continuous control benchmark in Robocup},
  year         = {2024},
  url          = {https://github.com/Michael-Beukman/RobocupGym_Private},
  note         = {GitHub repository}
}

@article{beukman2024Robocup,
  title={RobocupGym: A challenging continuous control benchmark in Robocup},
  author={Beukman, Michael and Ingram, Branden and Nangue Tasse, Geraud and Rosman, Benjamin and Ranchod, Pravesh},
  booktitle={Arxiv},
  year={2024},
}