search

mit-ll / spacegym-kspdg

Non-cooperative satellite operations challenge problems implemented in the Kerbal Space Program game engine
MIT License
52 stars 12 forks source link
autonomous-controller autonomous-vehicles differential-games kerbal-space-program non-cooperative-game openai-gym reinforcement-learning satellite-constellation space

readme

KSPDG: Kerbal Space Program Differential Games

Announcement: 2nd Annual Non-Cooperative Space Operations Challenge @ AIAA SciTech 2025

Join us at AIAA SciTech 2025 for a software design competition where participants develop autonomous agents for maneuvering satellites engaged in non-cooperative space operations.

Interested teams should sign up at https://www.surveymonkey.com/r/2MB8JDW, then they will be notified of upcoming meetings.

There is no cost to participate and membership and/or SciTech registration is not required.

KSPDG Overview

This library provides a suite of differential game (DG) challenge problems / environments set in the orbital domain built within the Kerbal Space Program (KSP) game engine. The intent of the KSPDG library is to provide benchmark challenge problems for evaluating autonomous control and AI algorithms applied to non-cooperative space operations.

The KSP differential game environments are implemented using the OpenAI Gym (or more specifically Gymnasium which is the drop-in-replacement, maintained fork of Gym) and PettingZoo standards. Non-GUI control of the KSP game engine is enabled by kRPC and control of physically distant spacecraft is enabled by PhysicsRangeExtender.

Design Principles

Environments / Challenge Scenarios

KSPDG is collection of orbital challenge problems defined within different scenario groups (i.e. "evaluation environments" to use nomenclature from reinforcement learning literature). As of Sep 2023, the challenge problems thus far implemented in KSPDG can be grouped into the following types of scenarios

In future development plans include scenarios with features like partial observability, multi-agent inspection problems with vision-based observation spaces, and head-to-head scenarios that leaverage the Luna Multiplayer mod that allow direct competition between two different AI players (current scenarios involve testing AI against pre-scipted bot adversaries)

Citation

DOI

@inproceedings{allen2023spacegym,
  title={SpaceGym: Discrete and Differential Games in Non-Cooperative Space Operations},
  author={Allen, Ross E and Rachlin, Yaron and Ruprecht, Jessica and Loughran, Sean and Varey, Jacob and Viggh, Herbert},
  booktitle={2023 IEEE Aerospace Conference},
  pages={1--12},
  year={2023},
  organization={IEEE}
}

Disclaimer

DISTRIBUTION STATEMENT A. Approved for public release. Distribution is unlimited.

This material is based upon work supported by the Under Secretary of Defense for Research and Engineering under Air Force Contract No. FA8702-15-D-0001. Any opinions, findings, conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the Under Secretary of Defense for Research and Engineering.

© 2024 Massachusetts Institute of Technology.

Subject to FAR52.227-11 Patent Rights - Ownership by the contractor (May 2014)

SPDX-License-Identifier: MIT

The software/firmware is provided to you on an As-Is basis

Delivered to the U.S. Government with Unlimited Rights, as defined in DFARS Part 252.227-7013 or 7014 (Feb 2014). Notwithstanding any copyright notice, U.S. Government rights in this work are defined by DFARS 252.227-7013 or DFARS 252.227-7014 as detailed above. Use of this work other than as specifically authorized by the U.S. Government may violate any copyrights that exist in this work.

Installation

The installation process includes several components:

:warning: Note These instructions have been written and verified on a macOS. They have been partially tested on Ubuntu 18.04 as well. Other operating systems should be similar with deviations on file and directory names

Install KSP & Making History Expansion

  1. Purchase and Download Kerbal Space Program and Making History expansion: https://store.privatedivision.com/game/buy-kerbal-space-program-ksp
    • Make sure to purchase Direct Download / DRM Free Private Division as the platform. Make sure you are buying KSP1 and not the recently released KSP2; none of this will work on KSP2!
    • Download the most recent version of KSP "On Final Approach" Portable (.zip). As of this writing and testing, the most recent version was v1.12.5
    • Download the most recent version of Making History expansion pack. As of this writing the most recent version was v1.12.1
  2. Unzip ksp-osx-1.12.5.zip to desired location; for simplicity, all instructions assume the unzipped KSP folder is placed on the Desktop
  3. Attempt to open the KSP game executable/app (e.g. KSP.app on Mac)

:warning: Troubleshooting

  • On a Mac, you might run into an error where you can't open the KSP app because the developer can't be verified.
  • To change these preferences on your Mac, choose Apple menu > System Preferences, click Security & Privacy
  • Under the General tab there should be a notification saying something like "KSP.app was blocked ..." if you've attempted to open the KSP app. Click the "Open Anyway" button next to the notification

:warning: Troubleshooting

  • On a Mac, after enabling KSP to be opened in Security and Privacy, you may encounter a bug where the game loading screen stalls indefinitely
  • The workaround is to move the KSP.app icon onto the desktop and then back into the KSP_osx directory. For some reason bash commands didn't seem to work to fix this bug. Had to manually open Finder, drag the KSP.app icon onto the Desktop, and then drag it back into the KSP_osx/ directory
  1. Unzip KSP-Making_History_Expansion-en-us-mac-1.12.1.zip
  2. Follow the instructions in the Instructions-xxx-xx-xx.txt file located in the unzipped Making History Expansion directory.

Instructions:

  1. Copy the two other files located in this folder (.command and .zip) to the folder where the KSP app is located
  2. Once you have copied the files, double click the .command file Thats it! Enjoy the Making History Expansion of Kerbal Space Program!
  1. Test installation by opening KSP (e.g. KSP.app on Mac). When main screen has loaded, select Start Game and you should see options for Play Missions and Mission Builder to confirm that the Making History Expansion was successfully installed

Install KSPDG Mission Save Files

For each differential game environment there are associated mission files created using the "Making History" expansion pack that serves to populate the KSP game engine with the necessary spacecraft in the appropriate orbits. There is also a number of mission save files for in-game software development testing purposes.

The save files are located in this repo under ksp_files/saves/missions and ksp_files/Missions; both sets are necessary to populate the differential game environments. These mission save files must be downloaded and manaully installed into the correct directory of the KSP game installation.

Copy the contents of ksp_files/saves/missions/ and ksp_files/Missions directory into your local installation of KSP. For example, on a Mac with this repo and KSP's install directories on the desktop this would look like:

git clone https://github.com/mit-ll/spacegym-kspdg.git
cd spacegym-kspdg
cp -r ksp_files/saves/missions/. ~/Desktop/KSP_osx/saves/missions
cp -r ksp_files/Missions/. ~/Desktop/KSP_osx/Missions

Install kRPC Server

kRPC is what allows external scripts and processes (such as python programs) to send commands to and control the KSP game engine

  1. Download latest version of kRPC from the GitHub link on the kRPC Getting Started Page. As of this writing, you should download krpc-0.5.2.zip. NOTE: make sure to download a full version, not just the python package; v0.5.2 is a full version but v0.5.3 is just the python package
  2. Unzip krpc-0.5.2/ folder to ~/Desktop/krpc-0.5.2/
  3. Create a new directory in KSP's GameData directory and move all of the krpc contents there 
    mkdir ~/Desktop/KSP_osx/GameData/kRPC
mv ~/Desktop/krpc-0.5.2/* ~/Desktop/KSP_osx/GameData/kRPC/

Install PhysicsRangeExtender

By default in KSP, high-fidelity physical simulation of spacecraft is only performed for spacecraft very near to the active spacecraft (e.g. only around 2km). PhysicsRangeExtender allows for better simulation (e.g. thusting maneuvers) of more distant spacecraft.

  1. Clone PhysicsRangeExtender (assumed to be cloned to Desktop in these instructions, but you can put it wherever you like since you will be copying things from the clone to GameData)
  2. Copy necessary subfolder from PhysicsRangeExtender to your KSP install's GameData folder 
    
# clone PhysicsRange Extender locally
cd ~/Desktop
git clone https://github.com/jrodrigv/PhysicsRangeExtender.git

copy the necessary game data for the mod into your KSP install

mkdir ~/Desktop/KSP_osx/GameData/PhysicsRangeExtender cp -r ~/Desktop/PhysicsRangeExtender/PhysicsRangeExtender/Distribution/GameData/PhysicsRangeExtender/* ~/Desktop/KSP_osx/GameData/PhysicsRangeExtender/


### Install `kspdg`

If you have not yet done so, clone this repository locally on your machine

```bash
git clone https://github.com/mit-ll/spacegym-kspdg.git

To install this package, run:

cd spacegym-kspdg
pip install -e .[adv_bots]  # installs juliacall allowing use of more advanced bots 
# if you plan to further develop the kspdg package, install the testing optional dependencies
# pip install -e .[full]

For development of this package, we recommend using the conda environment defined in environment.yml. To create and activate this environment, run:

cd spacegym-kspdg
conda env create -f environment.yml
conda activate kspdg

:warning: Troubleshooting

  • Note that the kspdg library depends upon astropy, which in turn depends upon pyerfa
  • FOR MAC USERS with M1 chipsets: as of this writing, pyerfa has not fully supported M1's arm64 architecture
  • This can lead to errors running kspdg such as 
    (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e'))
  • The workaround for Mac users with M1 chipsets is described here. For Python 3.9, the workaround entails cloning pyerfa locally, checking out a specific version, and installing in the conda environment 
    
# get pyerfa source code and switch to specific release of pyerfa
git clone --recursive https://github.com/liberfa/pyerfa/
cd pyerfa
git fetch origin
git checkout v2.0.0.1

install specific version of pyerfa in conda environment

conda activate kspdg pip install .

Install Advanced-Bots Julia Dependencies

For the [adv_bots] install, you need to also get Julia dependencies like iLQGames.jl. We've created a single python script to achieve this

conda activate kspdg
python install_julia_deps.py    # this may take a long time because julia may be installed here if not already present

Install Luna Multiplayer (LMP)

Future Work

Verify Installation

NOTE: Because the KSPDG environments are inexorably linked to the KSP game engine, many of the library's unit/integration test can only be run when a particular game mission file has been loaded and running. This means that verifying installation and testing during code development is a bit more involved than just a single pytest call

Serverless Tests: Quick test to run without KSP game engine running nor kRPC server connection

cd spacegym-kspdg
conda activate kspdg
pytest tests/serverless_tests/

KSP In-Game Tests: These tests require the KSP game engine to be running, the test-specific mission to be loaded, and a connection to the kRPC server

  1. Start KSP game application.
  2. Select Start Game > Play Missions > Community Created > pe1_i3 > Continue
  3. In kRPC dialog box click Add server. Select Show advanced settings and select Auto-accept new clients. Then select Start Server
  4. In a bash terminal: 
    
cd spacegym-kspdg
conda activate kspdg
pytest tests/ksp_ingame_tests/test_pe1_i3.py

for additional tests, load a different mission in KSP:

ESC > Quit to Main Menu > Exit to Main Menu > Play Missions > lbg1_i2 > Continue

pytest tests/ksp_ingame_tests/test_lbg1_i2.py

ESC > Quit to Main Menu > Exit to Main Menu > Play Missions > sb1_i5 > Continue

pytest tests/ksp_ingame_tests/test_sb1_i5.py

5. You should see the KSP game reset and focus on a vehicle that then performs several orientation and propulsive maneuvers. The pytest command should then indicate the number of passed tests.

> :warning: **Troubleshooting**
> If you are using a Mac with an arm64 architecture (e.g. M1 chipset) and recieve an error like `(mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e'))`, please refer to instructions in the [kspdg library installation section](#install-kspdg) about installing `pyerfa` from source.

------------

## Example: Hello KSPDG

Here we provide a "hello world" example of implementing a pursuit agent in the simple Pursuit-Evade environment. This agent simply points at the pursuer and burns full throttle (Do you think that can interecept even a non-maneuvering evader? Try it and find out!)

1. Start KSP game application. 
2. Select `Start Game` > `Play Missions` > `Community Created` > `pe1_i3` > `Continue`
3. In kRPC dialog box click `Add server`. Select `Show advanced settings` and select `Auto-accept new clients`. Then select `Start Server`

```python
from kspdg.pe1.e1_envs import PE1_E1_I3_Env

# instantiate and reset the environment to populate game
env = PE1_E1_I3_Env()
env.reset()

# Environment automatically orients pursuer toward target
# therefore a niave pusuit policy to to simply burn at full
# thrust in pursuer's body-forward direction.
# Do this until the episode 
# (Do you think it can intercept even a non-maneuvering evader??)
is_done = False
act = {
    "burn_vec": [1.0, 0, 0, 1.0], # burn vector in x, y, z, and duration [s]
    "vec_type": 0,  # burn vector as throttle values (if =1, burn vector represents thrust in [N])
    "ref_frame": 0  # burn_vec expressed in agent vessel's right-handed body frame. 
                    # i.e. forward throttle, right throttle, down throttle, 
                    # Can also use rhcbci (1) and rhntw (2) ref frames
}
while not is_done:
    obs, rew, is_done, info = env.step(act)

# printout info to see evaluation metrics of episode
print(info)

# close the environments to cleanup any processes
env.close()

See scripts/example_hello_kspdg.py for more details

Example: Agent-Environment Runner

Even though KSPDG environments are Gym (Gymnasium) environments, they require special handling because they must manage server connections to the KSP game engine, as well as handle the non-blocking nature of KSPDG environments (i.e. simulation time continues during and between calls to env.step()) which requires parallel processes for the environment interactions and agent's policy computation.

Therefore we have defined a Agent-Environment Runner class that helps manage these interactions. Users can define agents by extending the BaseAgentEnvRunner and execute their agents as follows

  1. Start KSP game application.
  2. Select Start Game > Play Missions > Community Created > pe1_i3 > Continue
  3. In kRPC dialog box click Add server. Select Show advanced settings and select Auto-accept new clients. Then select Start Server
  4. In a terminal, run python scripts/example_agent_runner.py

See scripts/example_agent_runner.py for more details

Some environments have obfuscated source code with more advanced bots. To here is an example of running a simple agent within such an environment (note that you load lbg1_i2 mission instead of pe1_i3)

  1. Start KSP game application.
  2. Select Start Game > Play Missions > Community Created > lbg1_i2 > Continue
  3. In kRPC dialog box click Add server. Select Show advanced settings and select Auto-accept new clients. Then select Start Server
  4. In a terminal, run python scripts/example_private_src_env_runner.py

See scripts/example_private_src_env_runner.py for more details

Example: Agent-Environment Evaluator for SciTech Challenge

This example walks through how to evaluate agents for scoring purpose in the AIAA SciTech 2024 Challenge. Due to the GUI-based interface of KSP that requires manual interaction, there is not a straight-forward process for hosting a centralized evaluation server upon which participants can submit agents. Instead we have adopted a decentralized process where agents are evaluated locally on particapants' own computers. Participants will then upload their evaluation results to a centralized scoreboard where they will be authenticated and ranked against other participants.

:warning: Honor System While we have taken some steps to prevent falsification of results, there is still potential for cheating. Therefore this competition works largely on the honor system.

If you think you are doing something inappropriate or unfair, you probably are.

We reserve the right to disqualify teams for unsporting behavior

The agent evaluation process uses a python scripts located in evaluation/evaluate.py which takes a command-line argument pointing to a Agent-Env runner configuration file. The configuration file contains several pieces of information, inlcuding:

See `evaluation/configs/example_eval_cfg.yaml`` for an example of the config file

Here is a basic example for running an agent-environment evaluation. As with other examples, you begin by:

  1. Start KSP game application.
  2. Select Start Game > Play Missions > Community Created > lbg1_i2 > Continue
  3. In kRPC dialog box click Add server. Select Show advanced settings and select Auto-accept new clients. Then select Start Server
conda activate kspdg # while it is not strictly necessary to use conda environments, it is encouraged for development and debugging purpose
cd evaluation # working directory is important due to relative path in cfg.yaml
python evaluate.py configs/example_eval_cfg.yaml

This should output to a file in the results/ subdirectory with a name like kspdg_results_20231018_125336.txt. That file has JSON-formatted results that look like (details will vary in your file)

{
    "agent_env_results": {
        "is_episode_done": true,
        "closest_lady_bandit_approach": 334.47956126016373,
        "closest_lady_bandit_approach_time": 90.81999999999907,
        "closest_bandit_guard_approach": 4.227775699613422,
        "minimum_lady_bandit_distance_speed_product": 15536.941529518028,
        "bandit_fuel_usage": 466.61279296875,
        "lady_fuel_usage": 0.0,
        "guard_fuel_usage": 468.96435546875,
        "weighted_score": 342942.1564060615
    },
    "user_id": "Evaluation Test",
    "user_key": "f2dcf0d1-f977-4e8b-9103-d03be0bffeec",
    "kspdg_version": "0.6.2",
    "agent_name": "Naive-Ned",
    "scenario_environment": "LBG1_LG3_I2_V1"
}
1239955493

This results file will then be sent to the authentication and scoreboard server for official ranking in the KSPDG Challenge.

INSTRUCTIONS FOR UPLOADING RESULTS TO COMPETITION SCOREBOARD WILL BE EMAILED TO PARTICIPANTS

Cautionary Notes

Here are some things you should NOT do as they will break/invalidate the proper functioning of the KSPDG environments

References

Throughout the documentation and code comments we refer to aerospace literature such as "Vallado Chp 3" for brevity. However this assumes anyone reading this code knows that "Vallado" is short hand for David Vallado's "Fundamentals of Astrodynamics and Applications", which is an unfair assumption to make. Here we list some of our short-hand references

Code Notation

Coordinate transforms are used throughout this code base. Without a rigorous notation it can become very difficult to determine the meaning of any particular variable. for example and ambiguous variable name like pursuer_pos seems to imply the position of a "pursuer" spacecraft, but position relative to what? Relative to a planet or relative to another spacecraft? Furthermore it says nothing of what coordinate frame that variable is expressed within. The pursuer position relative to the planet can be expressed in earth-centered inertial (ECI), perifocal, or even within another satellite's RSW frame (see Vallado, 3rd Edition, Sec 3.3 for frame descriptions). Each of these expressions lead to different numerical values for vector representations of the same physical vector

To make matters worse, kRPC---the crucial library that provides a python interface for controlling Kerbal Space Program---uses left-handed coordinate systems!! This will lead to much confusion if not pedantically handled.

To reduce the confusion we will strive to use the following notation for vector variables: w_x_y__z

Here are some abbreviations and acronyms used throughout the code