V1 vs V2 DSP alpha comparison

[alexveden]

Started working on V1/V2 results comparison issue.

Single member comparison

First of all, I've decided to compare pure results with exactly the same DSP alpha params in both versions.

I've found that even equal opt params produces completely different equity lines:

note: both swarms uses only single member, so ranking / rebalancing issues are excluded.

ES ContFut series are not equal

I've found that the ES ContFut EXOs series are slightly different, but at least in the period of 2011-2013 the series are almost the same, but that is not reflected in the equity lines (i.e. they are still diverging):

The reason of divergence in the different rules sets

I'm 100% sure that the divergence of the equities is the result of different signals values => exposures. I've checked this hypothesis and found that only 50% of market exposure values been equal to each other at the same date.

But if the price data is identical

But if the price data is identical both versions produces identical results.

Summary

I think that the problem is more fundamental than it seemed before. I think that the digital filter data is very sensitive to the even minor aberrations in the data and produces completely different set of signals.

For example, take a look at DSP rule_index == 1:

elif rule_index == 1:
            entry_rule = filt.shift(1) < filt
            exit_rule = filt.shift(1) > filt

            return entry_rule, exit_rule

where filt variable is a digital filter, for example due to data difference in ES cont fut, 'filt' value changed from 1123.11111115 (in V1 code) to 1123.11111117 (in V2 code). For the human it's a very minor change which should be ignored, but for the machine this is significant and as the result strategy code produces different set of entry/exit rules => different equity line.

I propose to ask @dmveden to add some rounding of the filt for the DSP signals, probably it could solve the problem.

@nikolas-joyce I think util we get the equal results of the DSP single member alphas in the both framework versions, the ranking algorithm porting should be paused. So I will wait your decision about what to do next.

p.s. Here are notebooks for comparison (v2) https://10.0.1.2:8889/notebooks/alphas/Alpha%20DSP%20BandPass%20-%20brute%20force%20optimizer%20V1%20comparison.ipynb (v1) https://10.0.1.2:8888/notebooks/strategies/Strategy_DSP_BandPass_StrategyModule_new-v2-comparison.ipynb

[dmveden]

tl;dr You have to align these time series and recalculate their cumulative equities

I have investigated differences between BP filters.

These are original BP filters in a log scale. As you can see they are really different. I supposed that, we get that picture because of different size of series.

So I joined these two price series into one df and i have removed nan's from it. Now we have an equally sized time series

BPs from aligned series. Price movement pattern is very different...

Is that because of difference in equities values? Let's find out

Price movement pattern is perfectly perfectly the same

Correlation of series above

Ok, let's add a random noise and look at the results. BP filters are from price series below

So I can say differences in prices are not affecting on resulting BP filter with that parameters.

V1 and V2 are starting from different dates

Therefore, we have to align these time series and recalculate their cumulative equities

Before

After

Now let's check the BP filters convergence

Zoomed in

[dmveden]

https://10.0.1.2:8889/notebooks/alphas/Alpha%20DSP%20BandPass%20-%20brute%20force%20optimizer%20V1%20comparison-Copy1.ipynb#Filters-comparison

[alexveden]

tl;dr You have to align these time series and recalculate their cumulative equities V1 and V2 are starting from different dates

As I clearly understand that historical datapoint added before history start can change filter values upside-down? If so that is very bad sign...

[dmveden]

[dmveden]

Do you see that exponential growth at the beginning? You can't get the same result from DSP filters on series with different length. When Starting point is same(look at very first picture from post above) then filter outcome is converges

[alexveden]

Well, I have tested DSP signals based on shifted data sets. I think that DSP alpha is not eligible for production use, because it's so unstable. I know, it have passed alpha sanity test, but this test doesn't simulate conditions when starting point of the history changes, probably I have to add this checks sometime.

Even 3-days shift at the history beginning

Even 3-days shift at the history beginning, changes up to 500 datapoint signals, this is ridiculous.

[nikolas-joyce]

I already make extensive use of the DSP in the production. Is there some other alternative?

Sent from my iPhone

On Sep 6, 2017, at 8:11 PM, alexveden notifications@github.com<mailto:notifications@github.com> wrote:

Well, I have tested DSP signals based on shifted data sets. I think that DSP alpha is not eligible for production use, because it's so unstable. I know, it have passed alpha sanity test, but this test doesn't simulate conditions when starting point of the history changes, probably I have to add this checks sometime.

Even 3-days shift at the history beginning

Even 3-days shift at the history beginning, changes up to 500 datapoint signals, this is ridiculous.

[image]https://user-images.githubusercontent.com/18488560/30144322-8979ae7c-939b-11e7-8302-7ad407b78688.png

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHubhttps://github.com/trendmanagement/Tmqr-framework-2/issues/39#issuecomment-327671564, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ARobLGctINTZlJGncyB6Z8JXJTfCGLz5ks5sf17sgaJpZM4PN101.

[dmveden]

Again, you can't get the same result from DSP filters on series with different startging point, because of nature of this indicator. In the begining there is always some kind of exponational growth(see my pervious post).

If you took some data from the END, indicator values does not change

[alexveden]

The problem with this alpha not only at the beginning, this case shows that deleting just 10 day at the middle of the history changes signals upside-down:

I mean simple data hole can change entire equity of the campaign after data backfill for example, this is not acceptable behavior of the alpha.

[dmveden]

well, same data hole(-s) can screw up alphas which based on sophisticated methods, like ML and so on.. Prove me if i'm wrong. Do you suggest to dump all these widely used alphas and work around MAs and tech. indicators or? Do you see here a missed opportunities to gain an edge because of poor datafeed and data-management?

[alexveden]

well, same data hole(-s) can screw up alphas which based on sophisticated methods, like ML and so on..

Don't get me wrong, but I don't think that ML will change long signals to short because of data hole. It's acceptable to get discrepancy in couple of trades or so, but when 450+ data-point signals of 500 have been reversed that is beyond good and evil.

Do you see here a missed opportunities to gain an edge because of poor datafeed and data-management?

In case of DSP alpha there is no edge, just some kind of random combination of numbers, which has been fitted to the curve by the genetic algorithms (i.e. optunity). I bet, that poor performance of this alpha on the V2 WFO backtest, is showing that it has poor stability.

[nikolas-joyce]

We actually had ok performance in v2 DSPs. the issue was the difficulty in finding acceptable results that where counter to the prevailing trend. In production in v1 they DSPs have been quite good. Based on the description DSP may not be a good match with the wfo genetic algo if the need for stability is so important. The question becomes what alphas do we have that are more stable? The fundamental issue of weak counter trend performance in the wfo is also visible in the ma cross alpha in v2. Any thoughts?

Sent from my iPhone

On Sep 8, 2017, at 5:20 AM, alexveden notifications@github.com<mailto:notifications@github.com> wrote:

well, same data hole(-s) can screw up alphas which based on sophisticated methods, like ML and so on.. Don't get me wrong, but I don't think that ML will change long signals to short because of data hole. It's acceptable to get discrepancy in couple of trades or so, but when 450+ data-point signals of 500 have been reversed that is beyond good and evil.

Do you see here a missed opportunities to gain an edge because of poor datafeed and data-management? In case of DSP alpha there is no edge, just some kind of random combination of numbers, which has been fitted to the curve by the genetic algorithms (i.e. optunity). I bet, that poor performance of this alpha on the V2 WFO backtest, is showing that it has poor stability.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHubhttps://github.com/trendmanagement/Tmqr-framework-2/issues/39#issuecomment-328088391, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ARobLD9Y6ij3kiBK-RjVF7do2XJ-EzONks5sgTEAgaJpZM4PN101.

[dmveden]

In case of DSP alpha there is no edge, just some kind of random combination of numbers, which has been fitted to the curve by the genetic algorithms (i.e. optunity).

Do you even know how it works, before saying "just some kind of random combination of numbers"? I guess not..

which has been fitted to the curve by the genetic algorithms (i.e. optunity). 1) The parameters were optimized in the first half of the available data. 2) Particle swarm is NOT a genetic optimization algo, since it not using a generations/hall-of-fame/crossovers/and other stuff.(http://deap.readthedocs.io/en/master/examples/pso_basic.html)

So in optunity workflow, there is 1 WFO step(optimize on first half -- evaluate on second).

[alexveden]

We actually had ok performance in v2 DSPs. the issue was the difficulty in finding acceptable results that where counter to the prevailing trend. In production in v1 they DSPs have been quite good.

Well I think this is more fundamental problem, and not related to DSPs. I had been suffering from it for years. From my point only way to tackle that is to use different alphas set, non correlated. In the mid 2012 I've done the research which analyzes the strategy equity and applies on/off filter to it. The bottom line was that these filters couldn't capture trends reversals or minify drawdowns of systems in long-run. Finally risk-reward was lower comparing to "do nothing".

I believe that only way to avoid trend changes is to add some alpha rules based on knowledge of the asset, which could predict market state. I.e. create alphas which include global trend filters, at least some time whey they are out of the regime, these alphas should be turned off.

I don't believe in alphas which can be effective on many assets and market regimes, but this is my market philosophy.

Based on the description DSP may not be a good match with the wfo genetic algo if the need for stability is so important. The question becomes what alphas do we have that are more stable?

Technically saying even simple CrossMA alpha is more stable than DSP, because of signals flipping effect. I've meet this effect before, for some alphas like renko or point&figure, which are calculated from the beginning of the history. But I didn't expect that signals will be reverted. Probably the coin is more stable in some sense, at least you know that it's random and probability is 0.5.

The fundamental issue of weak counter trend performance in the wfo is also visible in the ma cross alpha in v2. Any thoughts?

My recipe is:

1. Use alphas which know the asset regimes
2. Use diversified set of alphas (i.e. campaigns)
3. Single alpha performance is not meaningful, only the final result makes sense.

[alexveden]

Do you even know how it works, before saying "just some kind of random combination of numbers"? I guess not..

I've meant that if signals are flipping after minor data changes the results of the filter becomes random.

Particle swarm is NOT a genetic optimization algo, since it not using a generations/hall-of-fame/crossovers/and other stuff.

That is not true, Partical Swarm it's just strategy of GOpt, all of this stuff just hidden inside DEAP:

[nikolas-joyce]

Can we use a higher time frame to define the dominate trend or what do you suggest?

[nikolas-joyce]

or perhaps run the optimization/genetic on separate up and down trend areas based on a static definition like above or below a 60 day MA?

[alexveden]

or perhaps run the optimization/genetic on separate up and down trend areas based on a static definition like above or below a 60 day MA?

y, at least that is better then nothing.

From my point of view, we need to analyze underlying series, but not EXOs. In underlying case beside the price regime filtering, we can create models which predicts volatility regimes. I think ML could be good solutions for this task.