[Feature request] screenshot of backtest

[marcfon]

Would be great to have a screenshot right after a backtest strategy has run on a pair.

Some strategies display additional (performance) information on the chart which is very useful in later analysis of the data.

[timelyart]

I don't think a screenshot is the way to go. If you tell me which data you are interested in I can add them to the data.

[marcfon]

If you take a look at the QFL strategy which we are testing in #39 then you'll see that there's a pretty extensive stats table. It would great to have all this additional data captured as part of the backtest output. And I absolutely agree that if this can be done in json rather than a screenshot it would be much better. But could this be done in a generic and reusable way so it's not specific to only this one strategy? Also I think the stats table is "painted" on the canvas an not part of the dom tree.

[timelyart]

Ah, I see what you mean. I don't think Kairos is set up for this. Usually, you would want to test more markets rather than less, and it would be a pain to add data manually for each market. Can you share your thoughts on which data is important to you and why?

[marcfon]

Yes, collecting this additional data would especially be useful when testing many markets.

The key stats which I'm looking at right now in the additional stats table are Daily P/L, mean/max number of bars in deals and the safety order stats.

This data helps to evaluate the performance of a strategy. For instance a strategy could have a great net profit but if it's stuck for 4 months in a single deal that would eliminate it from my list of candidates.

[timelyart]

The strength of Kairos is that it can aggregate values over multiple markets and/or time frames and can say something about the performance of strategies at a higher abstraction level. Are you picking strategies per market on the basis of how well strategies have performed in the past for that specific market?

TV provides an average number of bars (in trade), as well as the Sharpe ratio, profit factor and buy&hold returns. Obviously, this isn't quite the data you are looking for but I am reluctant to add code that requires manual editing to be meaningful. I mean, I don't see the added value of browsing through screenshots when you can browse through a watchlist on TV to see and discern the same data.

[marcfon]

Are you picking strategies per market on the basis of how well strategies have performed in the past for that specific market?

Yes, this is one of the scenarios. Backtesting different strategy settings on a single pair can reveal the most optimal settings.

TV provides an average number of bars (in trade), as well as the Sharpe ratio, profit factor and buy&hold returns. Obviously, this isn't quite the data you are looking for but I am reluctant to add code that requires manual editing to be meaningful. I mean, I don't see the added value of browsing through screenshots when you can browse through a watchlist on TV to see and discern the same data.

What do you exactly mean with manual editing?

Browsing through screenshots makes sense from the point of view that recreating the strategy in TV with the exact settings from the backtest to see the additional stats takes time and effort. Having the screenshot would be much faster.

[timelyart]

Are you picking strategies per market on the basis of how well strategies have performed in the past for that specific market?

Yes, this is one of the scenarios. Backtesting different strategy settings on a single pair can reveal the most optimal settings.

Ah. I am of the opposite opinion. A strategy should work regardless of the market or time frame.

TV provides an average number of bars (in trade), as well as the Sharpe ratio, profit factor and buy&hold returns. Obviously, this isn't quite the data you are looking for but I am reluctant to add code that requires manual editing to be meaningful. I mean, I don't see the added value of browsing through screenshots when you can browse through a watchlist on TV to see and discern the same data.

What do you exactly mean with manual editing?

I assumed you wanted to incorporate data from the screenshot into a spreadsheet together with the other data.

Browsing through screenshots makes sense from the point of view that recreating the strategy in TV with the exact settings from the backtest to see the additional stats takes time and effort. Having the screenshot would be much faster.

The screenshot wouldn't capture the settings though; just the chart. However, the exact settings are already saved in the resulting dataset.

[marcfon]

Ah. I am of the opposite opinion. A strategy should work regardless of the market or time frame.

I agree with that. But the goal here is to find the most optimal settings for that strategy.

In my scenario where I'm backtesting a QFL strategy a lot of settings work regardless of the market or time frame. It's just that some QFL settings just perform better than others.

Or if we think about a grid trading strategy. Figuring out the best settings for the grid through backtesting is also an realistic use case, right?

I assumed you wanted to incorporate data from the screenshot into a spreadsheet together with the other data.

Yes, that would potentially be the case. This additional data could be used to evaluate which strategy settings are most desirable.

The screenshot wouldn't capture the settings though; just the chart. However, the exact settings are already saved in the resulting dataset.

The screenshot would contain the stats table right? (I've seen error screenshots that had the stats table in them.) And the stats table does contain the most important strategy settings.

Ideally you would want to link a screenshot to a specific set of backtest setting. And I think that's where the challenge lies. For instance if the screenshot filename would have an unique id that linked it to that backtest.

[timelyart]

Ah. I am of the opposite opinion. A strategy should work regardless of the market or time frame.

I agree with that. But the goal here is to find the most optimal settings for that strategy.

In my scenario where I'm backtesting a QFL strategy a lot of settings work regardless of the market or time frame. It's just that some QFL settings just perform better than others.

Or if we think about a grid trading strategy. Figuring out the best settings for the grid through backtesting is also an realistic use case, right?

Wouldn't you want to find the optimal setting generally speaking (i.e. over all time frames and all markets)?

I assumed you wanted to incorporate data from the screenshot into a spreadsheet together with the other data.

Yes, that would potentially be the case. This additional data could be used to evaluate which strategy settings are most desirable.

The screenshot wouldn't capture the settings though; just the chart. However, the exact settings are already saved in the resulting dataset.

The screenshot would contain the stats table right? (I've seen error screenshots that had the stats table in them.) And the stats table does contain the most important strategy settings. Ah yes. It would contain the stats table.

Ideally you would want to link a screenshot to a specific set of backtest setting. And I think that's where the challenge lies. For instance if the screenshot filename would have an unique id that linked it to that backtest. Kairos is meant for bulk testing strategies (i.e. multiple time frames and markets). For that purpose, screenshots are just a terrible way to capture statistics.

If you go down that route, you might as well set up the strategy manually and browse through the watchlist yourself since you would need to verify the screenshot anyway. It would save you the trouble of setting up Kairos and downloading thousands of screenshots (e.g. testing with 300 markets on 8 timeframes would result in 2400 screenshots per input set).

[marcfon]

Wouldn't you want to find the optimal setting generally speaking (i.e. over all time frames and all markets)?

Doesn't this depend on the strategy that you're following? I think for instance grid and QFL are prime candidates to be optimised for a specific time frame and market

If you go down that route, you might as well set up the strategy manually and browse through the watchlist yourself since you would need to verify the screenshot anyway. It would save you the trouble of setting up Kairos and downloading thousands of screenshots (e.g. testing with 300 markets on 8 timeframes would result in 2400 screenshots per input set).

What I have in mind is that you would let Kairos run the backtest in the background. Based on the initial net profit results create a shortlist of pairs that performed well and then dig deeper into those pairs based on the additional stats to come out with a handful of pairs.

I wouldn't test 300 markets on 8 timeframes. More like is that I would test a single pair with different parameters. Which fair enough might still generate a lot of screenshots.

If we go back to the start, my initial ideas was that the screenshot functionality is already in Kairos. So if people want a screenshot of their backtest -for whatever reason- it might be a useful addition.

But if you feel that this isn't no worries. It's not a show stopper for me.

[marcfon]

I know you're not a fan of doing manual post-processing on the data but stick with me for a moment. There might be a more generic solution rather than creating a screenshots.

What if Kairos had the option to export the list of trades per tested pair (as csv)? This would open up the option to do any (automated) post-process one might desire.

[marcfon]

@timelyart one more thought about this... This might actually be very well combined with #43. Just by adding tab in the google sheet with all the trades for a specific backtest. This would extremely helpful in my scenario.

[timelyart]

Yeah - that came across my mind too.

[marcfon]

Yeah - that came across my mind too.

I think there's an option to instantly copy all the TV trades to the clipboard. That might make things easier to implement.

[marcfon]

I've just checked and in order to export the trades from TV using their built in export feature you need at least the premium plan ($600/y). As not everyone might have that reading the values from trades table would be more accessible.