A Reinforcement-learning Agent learns how to play the classic game Snake

As part of a master project, a functional snake agent was developed with the help of Double Q-learning, PyTorch and a CNN.

The Blogpost (german) of this project can be found here.

Setup

1. Install Python, Pip, cuda and VirtualEnvironment
2. create a virtualenviroment with python3 -m venv /path/to/new/virtual/environment
3. activate the enviroment (unix) source <venv>/bin/activate (windows) C:\> <venv>\Scripts\activate.bat you should see a (venv) in front of your command prompt
4. install dependencys withpip install -r requirements.txt

Running

To start the learning process for the Snake,

python main.py

This starts the double Q-learning and logs the training data to the checkpoints folder.

You can also train an existing model by specifying a checkpoint in the main.py class. e.g current_checkpoint = Path(''checkpoints/timestamp/name_of_the_checkpoint.chkpt')

GPU will automatically be used if available.

To evaluate a trained Snake-model use,

python replay.py

This starts a window in which the Snake-agent "plays" a trained model. No training will take place. To evaluate a specific model, simply change the save_dir in replay.py. E.g checkpoint = Path('checkpoints/timestamp/name_of_the_checkpoint.chkpt')

Pre-Trained

A pretrained model can be found in the src/pretrained_checkpoint folder. The model was trained for approx. 24 hours. A Google Cloud VM instance was used as the training hardware:

• n1-highmem-2
• Tesla K 80 GPU

The pre-trained model achieved the following scores:

• max. Score: 28

• avg. Score: 12

  

Project Structure

• agent: communicates with game and neural, calculates training values, stores experiences and controlls learning
• game: the snake game that the agent will use
• main: used to start the training process
• neural: This class defines the neural network
• replay: used to let the agent play games with a pre trained Net (No learning will be done)
• metrics: Define a MetricLogger that helps track training/evaluation performance.

Ressourcen/Quellen

• Deep Learning Flappy-Bird

• Snake-Ai-Pytorch

• Screenshot Methode

  • DDQN
  • Exploitation vs Exploration
  • Mad Mario
  • Double Q-Learning
  • Q-Learning
  • epsilon-greedy-Prinzip
  • Loss
  • Q-values
  • Reward
  • deque
  • Reinforcement-Learning
  • PyGame
  • ReLU
  • Dense-Layer
  • Convolutional-Layer

• Start Videoreihe zum Thema

• Paper hs-albsig

• Paper towardsdatascience

• Paper Openreview.net siehe PDF Ordner

• Paper researchgate.net

• Paper geeksforgeeks