backtest

[gibo]

I would like to contribute to this project and have read through the readme in detail.

I have noticed you speak about a fatal flaw in the backtest, what is it? I can work on this and submit a PR.

[robertmartin8]

Hi,

In a sense, this project is "designed" to be flawed, or at least I have implemented the simplest possible solution fully aware of the problems. However, I'm inclined to leave it as it stands for anyone who wants to use this project as a starting block.

Robert

[gibo]

understood.

do you have any hint of what the problem is? This is my first go at ML applied to finance.

[robertmartin8]

Try to think about what a backtest really should be doing. It is meant to see how a strategy would have performed in the past – it should tell you what your performance would have been if you had this code 5 years ago.

However, if you look at backtesting.py, you'll see that there are some parts of the methodology that you could not have reproduced 5 years ago, meaning that the backtest is flawed.

As a tangible hint, the flaw is introduced in the first half of the backtesting file. Let me know if you'd like more guidance, but I really think it would be very useful to mull over this problem.

Robert

[gibo]

that's enough for me to get going.. and agree i would prefer to solve the problem on my own.

[gibo]

Ok, after numerous train rides mulling it over I think I have the problem.

Presume the issue is that you are training the model on future data, yet making predictions for some stocks in the past?

So seems the solution would be to:

• randomly select the year you'd like to predict
• ensure the training test split is only run on years prior

(I can delete my comment if you want)

[robertmartin8]

@gibo I've dropped you an email (found on your github profile), if that's ok.

[gibo]

.