PRUDEX-Compass

PRUDEX-Compass is designed for systematic evaluation of FinRL methods with 6 axes and 16 measures.

Install

To install the dependency of PRUDEX-Compass, run the command:

pip install -r requirements.txt

Creating PRUDEX-Compass Using the Python Script

You can use the create_compass.py python script to generate a compass and specify how it is filled for each FinRL method in a JSON file:

$ python Compass/generate/compass/create_compass.py --h
usage: create_compass.py [-h] [--template TEMPLATE] [--output OUTPUT] [--data DATA]

CLEVA-Compass Generator.

optional arguments:
  -h, --help           show this help message and exit
  --template TEMPLATE  Tikz template file. (default: Compass/generate/compass/blank.tex)
  --output OUTPUT      Tikz filled output file. (default: Compass/generate/compass/filled.tex)
  --data DATA          Entries as JSON file. (default: Compass/generate/compass/data.json)

For this purpose we provide the blank file blank.tex.

Example Usage

The default reads the template file from blank.tex and writes the filled output file into filled.tex with the data specified via --data :

$ python Compass/generate/compass/create_compass.py--data Compass/generate/compass/data.json

A example CLEVA-Compass, generated with Compass/generate/compass/data.json.

JSON Data Format

The JSON file specifies a list of entries, where each element defines a color, label, inner_level, and outer_level. The latter two specify the attributes visualized in the compass.

color: Can be one of ["magenta", "green", "blue", "orange", "cyan", "brown"].

label: A label describing the compass entry.

inner_level: Specifies the inner compass level attributes. Attribute values must be between 1 and 100.

outer_level: Specifies the outer compass level attributes. Attribute values must boolean (true/false).

Here a json file is given as an example:

{"entries": [{
       "color": "orange",
       "label": "A2C",
       "inner_level": {
          "Proftability": 29,
          "Risk_Control": 31,
          "University": 33,
          "Diversity": 43,
          "Reliability": 42,
          "Explainability": 50
        },
       "outer_level": {
          "alpha_decay": false,
          "profit": true,
          "extreme_market": false,
          "risk_adjusted": true,
          "risk": false,
          "time_scale": false,
          "assert_type": false,
          "country": true,
          "rolling_window": false,
          "correlation": false,
          "entropy": false,
          "t_SNE": false,
          "rank_order": false,
          "variability": false,
          "profile": false,
          "equity_curve": false  }}]}

and a more complicated example can be found in Compass/generate/compass/data.json

The resulting file filled.tex can then be included into any LaTeX document, e.g.:

\begin{figure}
    \input{filled.tex}
    \caption{PRUDEX-Compass}
\end{figure}

More details about configuration could be found here.

PRIDE-Star

• The PRIDE-Star is used to evaluate profitability,risk-control and diversity

  (a) A2C	(b) PPO	(c) SAC	(d) SARL
  (e) IMIT	(f) DeepTrader	(g) EIIE	(h) AlphaMix+

And here is the file structure for PRIDE-Star

-- PRIDE-Star 
    |-- A2C.tex
    |-- Alphamix+.tex
    |-- DeepTrader.tex
    |-- PPO.tex
    |-- SAC.tex
    |-- SARL.tex
    |-- blank.tex

Here we provide a blank tex that you can play with, the blank tex does not have any color block but the hexagon and the outer ring, while the rest of tex generate the subpicture corresponding to the shown above. You can also manipulate the color and the value for different algorithms to generate graphs.

Performance Profile

• The performance profile reports FinRL methods' score distribution of all runs across the differentfinancial markets that are statistically unbiased and more robust to outliers.
  

The key is to generate a dictionary whose key is the name of algorithms and the value is 2d array which represents different seeds and different task, then with the dictionary naming overall_dict, we can simpily use the code

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import scipy.stats
from rliable import library as rly
from rliable import metrics
from rliable import plot_utils
import seaborn as sns
sns.set_style("white")
import matplotlib.patches as mpatches
import collections
import os
from Compass.distribution.distribution import make_distribution_plot
colors = ['moccasin','aquamarine','#dbc2ec','orchid','lightskyblue','pink','orange']
xlabels = ['A2C','PPO','SAC','SARL','DeepTrader',"AlphaMix+"]
color_idxs = [0, 1,2,3,4,5,6]
ATARI_100K_COLOR_DICT = dict(zip(xlabels, [colors[idx] for idx in color_idxs]))
from scipy.stats.stats import find_repeats
xlabel=r'total return score $(\tau)$',
dict=tt_dict_crypto
algorithms = ['A2C','PPO','SAC','SARL','DeepTrader',"AlphaMix+"]
make_distribution_plot(dict,algorithms,2000,xlabel,"./distribution",ATARI_100K_COLOR_DICT)

to generate the distribution. Notice that we only use one metrics (total return in the example) to demonstrate the graph.

For more precise information, please refer to here.

Rank Distribution

• The rank distribution plot is a bar plot, where the i-th column in the rank distribution shows the probability that a given method is assigned rank i in the corresponding metrics.