search

produvia / kryptos

Kryptos AI is a virtual investment assistant that manages your cryptocurrency portfolio
http://twitter.com/kryptos_ai
MIT License
48 stars 8 forks source link

Prevent Backtest Overfitting #85

Open slavakurilyak opened 6 years ago

slavakurilyak commented 6 years ago

Goals

As a developer, I want to prevent backtest overfitting, so that I can find more True Positives and eliminate False Positives.

Consider

Inspiration

Backtest overfitting is an industry-wide problem (Financial Times, 2014).

screen shot 2018-07-04 at 4 11 34 pm

Source: The 7 Reasons Most Machine Learning Funds Fail (Presentation Slides), 2018

True Positive Technologies, lead by Marcos Lopez de Prado, developed multiple solutions to the backtest overfitting problem, including:

  1. Probability of Backtest Overfitting (PBO): It computes the probability that the best in-sample strategy will underperform out-of-sample a randomly chosen alternative.
  2. Deflated Sharpe Ratio (DSR): It deflates the predicted performance of a strategy by controlling for the number of trials involved in a discovery. In the example above, the “buy on Tuesday and sell on Friday” strategy would be discarded because of the large number of combinations tried.
  3. Optimal Trading Rule (OTR): This empirical procedure determines the optimal trading rule on a large amount of synthetic data generated by the distribution that characterizes the observed data.
  4. CPCV: It derives the full distribution of the strategy’s performance (rather than a single path) under thousands of alternative scenarios.
slavakurilyak commented 6 years ago

@bukosabino Let's use pypbo, the python library, to prevent backtest overfitting