Consider ways to improve fitness

[jameschch]

Recent research points to inadequacy of sharpe (alone) as a fitness measure. Consider weighted fitness using tail ratio, benchmark alpha, kurtosis etc.

[worthy7]

IMO I think this should be left open as part of the users algorithm. It entirely depends on what they are trying to achieve, which means they should also decide how it is scored. We can of course include some defaults. For example, someone looking to build a long term strategy may want to measure both max drawdown and also Sharpe. But someone else may not care about risk so much and just wants to measure dairy profits. We could perhaps create an IOptimizedAlgorithm interface which people apply to their algo, and then they fill in the FitnessEvaluation themselves.

[jameschch]

The default fitness is based on sharpe ratio. There is another fitness measure provided based on annual returns. There is also a configuredfitness class that allows you to optimize based on any other statistic field from the main output.

I'm not convinced that a fitness measure is a concern of an algorithm: I believe at the very least a benchmark is going to be appropriate to a family of resemblances.

If you wish to define your own algorithm interface pattern for integrating with the config then feel free. You could also define your own benchmark and configure the optimizer towards this. However I'm not aware of strong supporting evidence that points to the urgent inadequacy of Sharpe ratio. This is one of the project's main areas of research interest. If you are aware of any pertinent research papers, please post here.

[jameschch]

This is the paper that prompted this issue:

https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2745220

[jameschch]

There has been some interest in a blended fitness, so for instance 50% win rate and 50% alpha. I suspect the blend ratio would just be another hyperparameter liable to overfitting.

[worthy7]

Fair enough, it seems you've done a bit more research than I.

[jameschch]

Well, this is just the tip if the iceberg. Please go ahead and try some options with the ConfiguredFitness. If there is a missing statistical measure it would be valuable to Lean in general.

[worthy7]

You're most likely correct, I only started using QC about 2 weeks ago. Still learning, but eager! I hope I can contribute to this project at some point. I loved my Machine Learning course back in university, but I feel first some performance optimizations might be a good place to start though.

[worthy7]

I mentioned this on the forum, but I'll mention it here too. How about just net profit %? I know it sounds simple, but when this is done using walk forward, then that somewhat takes care of the issues that thing's like "max drawdown" are supposed to deal with. It's literally just raw money return... Unless I'm not understanding the Walk Forward behavior properly... ( I read the second half of the paper btw, I see the point, but I'll need to read it again to understand it properly)

[worthy7]

Read it all again, in detail. Someone also summarized it online: https://optimal.quora.com/Trading-algorithms-Comparing-Backtest-and-Out-of-Sample-Performance-using-Machine-Learning

Why don't we: 1) Somehow restrict the Sharpe to only the last year of data. (By either, suggesting it to the user, or overwriting the dates set in their algorithm (somehow).) 2) Use the tail ratio as the default fitness function... It seems to be much more correlated.

Better still, make the default fitness a combined result of (0.65lastYearSharpe + 0.7tailRatio)

[worthy7]

Tbh, the paper DOES seem to suggest a fairly strong issue with Sharpe, especially since everyone is backtesting as far as they humanly can. Rather than depth, people should be going for breadth (across more markets/pairs). 10 different pairs over the last 1 year is better than 1 pair over 10 years, and, arguably, it could be better than 10 pairs over 10 years.

[jameschch]

These is already a fitness based on return provided in the project. The flaws of this measure are what gave rise to Sharpe ratio.

On 26 April 2017 07:06:41 BST, Worthy7 notifications@github.com wrote:

I mentioned this on the forum, but I'll mention it here too. How about just net profit %? I know it sounds simple, but when this is done using walk forward, then that somewhat takes care of the issues that thing's like "max drawdown" are supposed to deal with. It's literally just raw money return... Unless I'm not understanding the Walk Forward behavior properly... ( I read the second half of the paper btw, I see the point, but I'll need to read it again to understand it properly)

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHubhttps://github.com/jameschch/LeanOptimization/issues/4#issuecomment-297247515, or mute the threadhttps://github.com/notifications/unsubscribe-auth/APvtWIxI5ZzP-131fQayxNp4OYnlMrQNks5rzt7xgaJpZM4LqU0V.

[worthy7]

Yea, I realize that now. Ignore that comment :-)

[jameschch]

I like the idea of a blended fitness. However I think without some theoretical basis, the blend % will just be another overfit parameter. You could delegate this to a machine intelligence and just feed it all statistics.

[worthy7]

What do you mean by blend %? you mean the 0.65 & 0.7 figures? They are from the paper.

[jameschch]

In that case this would be a good candidate for fitness. Let me look closer at the paper.

[worthy7]

Just to clarify a little bit (I'm looking to work on the code soon, so rereading the article a bit):

• Basically, for any kind of backtesting it suggests that Tail-Ratio is a better predictor of REAL WORLD Sharpe. That's interesting.

• If we are going to use Sharpe, it suggests that the latest year of backtest data to be used and nothing more. Sharpe has 'recency'.

So basically, I want to implement the tailratio statistic first. Perhaps it should even be a part of Lean. At the very minimum, using tailratio for optimizing seems to be a better indicator of how the algorithm would perform in the real world, if performance is based on Sharpe.

I think my idea for blending can be ignored for now, It's situational in that it requires the use of only the last year of data. We would somehow have to recognize and check for this and it seems it everything can just be improved by using tailratio instead.

[jameschch]

Sounds like an excellent idea to try to get tail ratio integrated into lean.

My understanding is that there is a recency bias for Sharpe but I imagine you will find a recency bias for many measures due to capturing the current regime.

[worthy7]

Yea but think about it, the tail-ratio should be pretty robust. Key word being "Daily" and not something "Over time" I have no idea how to get a Daily Profits statistic out of lean. Any ideas?