Deep Reinforcement Learning for Financial Portfolio Optimisation

paper

TD3-Based Ensemble Reinforcement Learning for Financial Portfolio Optimisation. Created for masters dissertation, Department of Artificial Intelligence, Faculty of Information & Communication Technology University of Malta. Nigel Cuschieri, supervised by Vince Vella and Josef Bajada.

Installation instructions

Our project was created on Ubuntu 20, using Anaconda. PIP and conda requirements are available in both .txt and .yml format. You may choose one as you deem fit. These are:

• requirements.txt - pip install --user --requirement requirements.txt
• tensor_keras_portfolio.yml - conda env create -f tensor_keras_portfolio.yml

Datasets

Datasets are in H5 format, stored in utils/datasets.

• NYSE(N) dataset, which may be found here found here
• SP500 custom dataset based on S&P500, with real stock data gathered using Yahoo Finance. New datasets can be created using the jupyter notebooks found in utils/.!

Settings

Main configuration is done in config/stock.json.

Training

Pre Trained models

The trained models used in our study are saved within the solution. The training results are saved in _results/nysen and results/SP500. The weights are stored in _weights/nysen and weights/SP500.

Training process

• Open cmd in the folder, and connect to the environment (_conda activate tensor_kerasportfolio).
• Execute the stock_trading.py python file. Note that this will overwrite the corresponding training results and weights. Parameters include:
  • Debug (-d = True)
  • Framework (-f = DDPG)
  • Predictor Type (-p = LSTM)
  • Window Length (-w = 3)
  • Technical Indicators (-t = False)

Testing

Multiple Jupyter notebooks are included in our project to allow for testing and evaluation of our trained models. These include:

• Finding Threshold Value for Training TD3 models.ipynb - Used to study the hypothetical rewards given to OLPS algorithms to justify the creation of a Value Function Threshold.
• On-line Portfolio Selection.ipynb - Tests on solely OLPS algorithms.
• Test DDPG and TD3.ipynb - Tests to compare models with the two different frameworks.
• TD3 vs TD3-rmr-pred.ipynb - Tests to compare performance of enhanced state format.
• Test DDPG and TD3-InDepth.ipynb - In depth analysis of performance is done with additional visualisations allowing us to see decisions made by agent.
• Test Ensemble.ipynb - Tests to compare performance of the ensemble model with OLPS algorithms and previous models.
• Training Results.ipynb - Visualisations of training results.

References

• A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem (Jiang et. al.)
• Continuous control with deep reinforcement learning (Hunt et. al.)
• Addressing Function Approximation Error in Actor-Critic Methods (Fujimoto et. al.), Code on Github
• The code is inspired by Using Reinforcement Learning for Portfolio Optimization
• The environment is inspired by wassname/rl-portfolio-management
• DDPG implementation is inspired by Deep Deterministic Policy Gradients in TensorFlow
• OLPS algorithms borrowed and extended from universal-portfolios