Weird behaviour wit max_sharpe and efficient_risk methods

[Nyquis]

I have been loading two datasets :

tickers = ['VBMFX', 'VTSMX']
select_val = 'Adj Close'
#tickers = []'AGG', 'VTSMX']

df_complete = web.DataReader(tickers ,data_source='yahoo', start='2000-01-01', end="2020-01-16")[select_val]

I then calculate the variables required by efficient frontier:

mu = expected_returns.ema_historical_return(df_complete)
S = CovarianceShrinkage(df_complete).ledoit_wolf()

ef = EfficientFrontier(mu, S,
                       #gamma=1,
                       )

# weights = ef.max_sharpe(risk_free_rate=0.005)
eff_risk = ef.efficient_risk(target_risk=0.1, risk_free_rate=0.005)

ef.portfolio_performance(verbose=True)

As a result I get:

Expected annual return: 7.6%
Annual volatility: 3.8%
Sharpe Ratio: 1.46
(0.0759353668927616, 0.03834329016364384, 1.4588045693011014)

It does not matter what risk I indicate it does not change fiven values. I have though no issue with efficient_target. Could you take a look why it seems to get stuck without adjusting to a target volatility. Manually I can tune it to 10% but the algorithm should be able to do so as well.

[schneiderfelipe]

Could you show us the calculated weights? I suspect neither of your tickers has volatility larger than 3.8%.

Since the volatility is simply a quadratic function (in the current approximation) and the weights sum to one, all possible portfolio volatilities are limited both below and above by the smallest and largest eigenvalues of the covariance matrix (or something like that, that's math from the top of my head right now).

The point is that there are simply impossible scenarios given the data, so not PyPortfolioOpt's fault. We would need to see the weights (the covariance matrix would be nice too).

[Nyquis]

Over the period of 20 years they both should have quite a high volatility. I mean due to several stress events such as the 2008 and 2000 crash events.

The obtained weights are: array([0.84500175, 0.15499825])

VBMFX over this span has a annualized volatility of 3.9% VTSMX over this span has a annualized volatility of 19.1%

Therefore it should be possible to optimize them for at least between 3.9 and 19.1 % volatility respectively.

When overriding the method of efficient_risk and changing the following line:

target_constraint = {
            "type": "ineq",
            "fun": lambda w: target_risk - np.sqrt(objective_functions.volatility(w, self.cov_matrix)),
        }

To "eq" seems to produce a result. I however have difficulties to judge what the implication to changing tis relation would be

target_constraint = {
            "type": "eq",
            "fun": lambda w: target_risk - np.sqrt(objective_functions.volatility(w, self.cov_matrix)),
        }

[Nyquis]

I had forgotten the covariance matrix:

Symbols VBMFX VTSMX Symbols
VBMFX 0.001666 -0.002234 VTSMX -0.002234 0.036580

[robertmartin8]

@Nyquis

Thanks for raising the issue. As @schneiderfelipe correctly pointed out, the variances are too low. Observe the main diagonal of your covariance matrix (these are the variances). VTSMX has the higher variance of the two, at 0.0366. Hence no allocation (without leverage) can increase the portfolio variance beyond 0.0366.

I should clarify that when I say variance here, I am referring to the variance implied by the Ledoit-Wolf covariance matrix (since you used Ledoit Wolf covariance). If, as you say, annualised volatility is really 19%, then this should be reflected in the diagonals of the sample covariance matrix. Why don't you try risk_models.sample_cov and report back?

Best, Robert

[Nyquis]

Hello @robertmartin8 and @schneiderfelipe I hope I didn't come across as uncouth.

The matrix I got is :

Symbols VBMFX   VTSMX
Symbols     
VBMFX   0.001589    -0.002244
VTSMX   -0.002244   0.036664

However as mentionned before https://www.google.com/search?q=vtsmx&oq=VTSMX&aqs=chrome.0.0l8.1395j1j7&sourceid=chrome&ie=UTF-8 shows guite a lot of volatility due to it being FTSE All-World type of ETF.

On the schwab website you can see the mean volatility over a three year period is at 14.43 % : https://www.schwab.wallst.com/Prospect/Research/mutualfunds/risk.asp?symbol=VTSMX

Which is in line with what I had found, due to several more market crashes it makes sense that over 20 years I would get higher volatility.

PS Equivalent index funds that follow the same index funds:

Symbols VAGE.DE VWRL.AS
Symbols     
VAGE.DE 0.000174    -0.000125
VWRL.AS -0.000125   0.021018

I can specify and obtain as expected the hoped for volatility

Expected annual return: 10.6%
Annual volatility: 10.0%
Sharpe Ratio: 0.86
(0.10576374758903338, 0.10000000010868033, 0.8576374749582505)

I also obtain meaningfull weights:

{'VAGE.DE': 0.30897, 'VWRL.AS': 0.69103}

Thank you very much for having taken time to answer my questions

[robertmartin8]

@Nyquis

Certainly not uncouth, just trying to get to the bottom of the issue. Let me play around with some of the data. I suspect one of the things causing confusion is the square root between volatility and variance.

For example, Schwab quotes the volatility as 0.1443. Hence the variance should be the square of that, i.e 0.021. This is not far from the value in the covariance matrix. But I'm not yet sure if this is just a coincidence. I'll get back to you shortly.

[Nyquis]

Thank you if you wish me to test some things I would love to do so. I tried to recreate the volatility matrix I get by using the sklearn implementation of Ledoit Wolf. I however seem to get very different results so I think I might be lacking the understanding of the correct way the stockprices are processed.

Best regards, Nyquis

[robertmartin8]

@Nyquis @schneiderfelipe

What is happening is that efficient_risk is returning the same as max_sharpe. This is because behind the scenes, efficient_risk actually optimises for sharpe ratio given a volatility constraint. But the optimiser is not actually able to properly deal with the constraint (inequalities are harder than equalities, see KKT conditions etc), so it seems to ignore it and return max_sharpe.

I propose the following changes:

1. Change it to an equality constraint – as you noted, this fixes the issue (I reproduced it on my machine)

2. Modify the constraint function. Currently, it is:

"fun": lambda w: target_risk - np.sqrt(objective_functions.volatility(w, self.cov_matrix)),

It is computationally better to use target_risk**2 - objective_functions.volatility(...).

3. Add an explicit warning if the optimisation fails (i.e by checking whether the resulting portfolio volatility is equal to the target).

@schneiderfelipe what do you think?

@Nyquis thanks for raising this issue and for your patience!

[schneiderfelipe]

I propose the following changes:

1. Change it to an equality constraint – as you noted, this fixes the issue (I reproduced it on my machine)

I think it's a nice idea. Also, the data given by @Nyquis could be assembled as an interesting test.

2. Modify the constraint function. Currently, it is:

"fun": lambda w: target_risk
- np.sqrt(objective_functions.volatility(w, self.cov_matrix)),

It is computationally better to use target_risk**2 - objective_functions.volatility(...).

I agree. It might not be a bottleneck, but squaring is definitely cheaper (almost fivefold):

In [1]: a = 1000

In [2]: %timeit a**2
229 ns ± 1.62 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

In [3]: import numpy as np

In [4]: %timeit np.sqrt(a)
1.06 µs ± 4.11 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

3. Add an explicit warning if the optimisation fails (i.e by checking whether the resulting portfolio volatility is equal to the target).

This might be a good general behaviour for all optimisers since it's hard to be really transparent about the optimisation process. But if so, I think this warning should be tested as well.

[robertmartin8]

@schneiderfelipe

I have pushed a fix in a new branch. In particular:

• Checking the result
• Testing the warning:

def test_efficient_risk_error():
    ef = setup_efficient_frontier()
    ef.min_volatility()
    min_possible_vol = ef.portfolio_performance()[1]
    with pytest.raises(ValueError):
        # This volatility is too low
        ef.efficient_risk(min_possible_vol - 0.01)

Let me know if you have any thoughts, if not I will hope to merge it soon.

[Nyquis]

Thank you very much. Btw I adjusted the function with basinhopping instead of using directly the minimize fonction. I don't know if several minima might exist in the case of several tickers. However if you think they may, might I recommend to consider using it ?

Best regards, Nyquis

[robertmartin8]

@Nyquis

Hmmm I don't think that is necessary because we are dealing with convex functions. Unless you add funny objective functions, SLSQP should be fine.

But I'm afraid I can't help you too much here – optimisation is a very complex topic and the cost landscape depends heavily on individual problems. My apologies.