search

sadighian / crypto-rl

Deep Reinforcement Learning toolkit: record and replay cryptocurrency limit order book data & train a DDQN agent
850 stars 235 forks source link
arctic bitcoin bitfinex coinbase coinbase-pro cryptocurrencies deep-reinforcement-learning dqn hft limit-order-book mongodb multiprocessing multithreading order-book orderbook recorder tick-data tickstore trading trading-algorithms

readme

Deep Reinforcement Learning Toolkit for Cryptocurrencies

Table of contents:

  1. Purpose
  2. Scope
  3. Dependencies
  4. Project structure
  5. Design patterns
  6. Getting started
  7. Citing this project
  8. Appendix

1. Purpose

The purpose of this application is to provide a toolkit to:

High_Level_Overview

2. Scope

3. Dependencies

See requirements.txt

Note: to run and train the DQN Agent (./agent/dqn.py) tensorflow and Keras-RL need to be installed manually and are not listed in the requirements.txt in order to keep this project compatible with other open sourced reinforcement learning platforms (e.g., OpenAI Baselines).

Pip install the following:

git+https://github.com/manahl/arctic.git

Keras==2.2.4
Keras-Applications==1.0.7
Keras-Preprocessing==1.0.9
keras-rl==0.4.2

tensorboard==1.13.1
tensorflow-estimator==1.13.0
tensorflow-gpu==1.13.1

4. Project Structure

The key elements in this project and brief descriptions.

crypto-rl/
    agent/
                ...reinforcement learning algorithm implementations
    data_recorder/
                ...tools to connect, download, and retrieve limit order book data
    gym_trading/
                ...extended openai.gym environment to observe limit order book data
    indicators/
                ...technical indicators implemented to be O(1) time complexity
    design-patterns/
                ...visual diagrams module architecture
    venv/
                ...virtual environment for local deployments
    experiment.py          # Entry point for running reinforcement learning experiments
    recorder.py            # Entry point to start recording limit order book data
    configurations.py      # Constants used throughout this project
    requirements.txt       # List of project dependencies
    setup.py               # Run the command `python3 setup.py install` to 
                           #    install the extended gym environment i.e., gym_trading.py

5. Design Patterns

Refer to each individual module for design pattern specifications:

Sample snapshot of Limit Order Book levels: plot_lob_levels

Sample snapshot of Order Arrival flow metrics: plot_order_arrivals

6. Getting Started

Install the project on your machine:

# Clone the project from github
git clone https://github.com/sadighian/crypto-rl.git
cd crypto-rl

# Install a virtual environment for the project's dependencies
python3 -m venv ./venv

# Turn on the virtual environment
source venv/bin/activate

# Install keras-rl dependencies
pip3 install Keras==2.2.4 Keras-Applications==1.0.7 Keras-Preprocessing==1.0.9 keras-rl==0.4.2
 tensorboard==1.13.1 tensorflow-estimator==1.13.0 tensorflow-gpu==1.13.1

# Install database
pip3 install git+https://github.com/manahl/arctic.git

# Install the project
pip3 install -e .

6.1 Record limit order book data from exchanges

Step 1: Go to the configurations.py and define the crypto currencies which you would like to subscribe and record.

Note: basket list format is as follows [(Coinbase_Instrument_Name, Bitfinex_Instrument_Name), ...]

SNAPSHOT_RATE = 5  # I.e., every 5 seconds
BASKET = [('BTC-USD', 'tBTCUSD'),
         ('ETH-USD', 'tETHUSD'),
         ('LTC-USD', 'tLTCUSD'),
         ('BCH-USD', 'tBCHUSD'),
         ('ETC-USD', 'tETCUSD')]
RECORD_DATA = True

Step 2: Open a CLI/terminal and execute the command to start recording full limit order book and trade data.

 python3 recorder.py

6.2 Replay recorded data to export stationary feature set

Step 1: Ensure that you have data in your database.

Check with MongoDB shell or Compass. If you do not have data, see refer to the section above 6.1 Record limit order book data from exchanges.

Step 2: Run a historial data simulation to take snapshots of the limit order book(s) and export their stationary features to a compressed csv.

To do this, you can leverage the test cases in data_recorder/tests/ or write your own logic. When using the test case methods, make sure to change the query parameters to match what you've actually recorded and is in your database.

Example to export features to a compressed csv:

python3 data_recorder/tests/test_extract_features.py

6.3 Train an agent

Step 1: Ensure you have data in the data_recorder/database/data_exports/ folder. This is where the agent loads data from. If you do not have data exported into that folder, see refer to the section above 6.2 Replay recorded data to export stationary feature set.

Step 2: Open a CLI/terminal and start learning/training the agent. 

python3 experiment.py --window_size=50 --weights=False --fitting_file=...

Refer to experiment.py to see all the keyword arguments.

7. Citing this project

Please remember to cite this repository if used in your research:

    @misc{Crypto-RL,
        author = {Jonathan Sadighian},
        title = {Deep Reinforcement Learning Toolkit for Cryptocurrencies},
        year = {2019},
        publisher = {GitHub},
        journal = {GitHub repository},
        howpublished = {\url{https://github.com/sadighian/crypto-rl}},
    }

8. Appendix

8.1 Branches

There are multiple branches of this project, each with a different implementation pattern for persisting data:

8.2 Assumptions

8.3 Change Log