sector constraints don't work properly for market-neutral efficient risk optimization

[bass-dee]

Hi there,

sector constraints don't work properly for market-neutral efficient risk optimization, despite setting negative lower boundaries! Setting weight bounds globally (-1,1) also doesn't help. I tried different solver, but same there.

I want stocks being able to go short/long and strategies & funds just long in the portfolio (because they might be long/short already). If market_neutral=True everything is either long or short and sectors get overwritten, if False, everything just long. Any ideas how to fix it?

mu = expected_returns.ema_historical_return(df)
S = risk_models.risk_matrix(df, method="ledoit_wolf")
ef = EfficientFrontier(mu, S, solver='CVXOPT', verbose=True)

      sector_mapper={'sp500_etf': 'fund',
       'AAPL': 'stocks',
       '1001413': 'strategy'
}

        sector_lower = {"strategy": 0.0,
                       "fund": 0.0,
                        "stocks": -1.0
                       }  
        sector_upper = {
                    "strategy": 1.0,
                       "fund": 1.0, 
                        "stocks": 1.0
        }
        ef.add_sector_constraints(sector_mapper, sector_lower,sector_upper)
        raw_weights=ef.efficient_risk(0.107, market_neutral=True)

Windows Version 10.0.19042 Build 19042, Python 3.6.7, PyPortfolioOpt 1.2.6

[phschiele]

I think using sector constraints together with market-neutral portfolios is not supported at the moment: https://github.com/robertmartin8/PyPortfolioOpt/blob/6158b1e22819d174bae8885fc74eed772890bb55/pypfopt/base_optimizer.py#L313-L316

[robertmartin8]

I still haven't convinced myself that market-neutral optimisation with sector constraints "works", but from PyPortfolioOpt's perspective you can definitely do it.

sector_mapper = {
    "MSFT": "Tech",
    "AMZN": "Consumer Discretionary",
    "KO": "Consumer Staples",
    "MA": "Financial Services",
    "COST": "Consumer Staples",
    "LUV": "Aerospace",
    "XOM": "Energy",
    "PFE": "Healthcare",
    "JPM": "Financial Services",
    "UNH": "Healthcare",
    "ACN": "Misc",
    "DIS": "Media",
    "GILD": "Healthcare",
    "F": "Auto",
    "TSLA": "Auto"
}

sector_lower = {
    "Consumer Staples": 0.1, # at least 10% to staples
    "Tech": 0.05 # at least 5% to tech
    # For all other sectors, it will be assumed there is no lower bound
}

sector_upper = {
    "Tech": 0.2,
    "Aerospace":0.1,
    "Energy": 0.1,
    "Auto":0.15

ef = EfficientFrontier(mu, S, weight_bounds=(-1, 1))
ef.add_sector_constraints(sector_mapper, sector_lower, sector_upper)
ef.efficient_risk(target_volatility=0.15, market_neutral=True)
weights = ef.clean_weights()

for sector in list(set().union(sector_upper, sector_lower)):
    sector_sum = 0
    for t, v in weights.items():
        if sector_mapper[t] == sector:
            sector_sum += v

    print(sector, sector_sum)
    assert sector_sum <= sector_upper.get(sector, 1) + 1e-5
    assert sector_sum >= sector_lower.get(sector, -1) - 1e-5

Output:

Tech 0.05
Auto 0.15
Aerospace -0.0148
Consumer Staples 0.1
Energy -0.46745

As you can see, the output is between the provided bounds (this is using the data/code in the Binder cookbooks).

So I guess the behaviour you're seeing might just be the true output of the optimiser? As I said – I'm not sure whether this optimisation procedure produces the intended results, hence the warning in the docs.

But it doesn't seem that there's anything wrong with it? So maybe the warning should be removed. Any thoughts on this @phschiele?

[phschiele]

@robertmartin8 I agree that from a technical point of view it would still work. In the market-neutral case e.g. having a positive lower bound of 30% for a sector means one would be net long this sector (and eventually another sector needs to be net short). Perhaps it's a bit unintuitive to what this 30% refers to in a market-neutral portfolio, especially if there is no constraint on gross leverage. Since it's quite easy to end up with an infeasible problem or an unintuitive solution, the warning might still be helpful.

I also think that in the original post @bass-dee might be looking for single asset bounds, rather than sector constraints. It is stated that funds/strategies should only go long, but with a sector constraint, this is only ensured in aggregate.

[bass-dee]

Actually I found a total different solution to my problem. Not only, because of the shorting-issues but also, because the long/short market neutral portfolio produces x-times of leverage (in my case 15x), which I am not sure, I want to be exposed to in reality, even if it is market neutral (like overnight holding cost etc.).

It might be already a bit of topic, but I created synthetic inverse Close-prices as input of the stocks, I want to go short and then ran a long-only optimization. However, if I add long and short prices to the portfolio, I have to correct the final allocation for each stock separately at the end. e.g. go 2 x 'MSFT' long - 1 x 'MSFT' short= 1x 'MSFT' long. And you get quite a lot of assets, so the optimizer might fail, coz he can't find an optimal solution.

This function I found on github initially for simulating leverage. I just use -1 for shorting:

def sim_leverage(proxy, leverage=-1, expense_ratio = 0.0, initial_value=1000.0): pct_change = proxy.pct_change(1) pct_change = (pct_change - expense_ratio / 252) * leverage sim = (1 + pct_change).cumprod() * initial_value sim[0] = initial_value return sim

Maybe this workaround helps.

[robertmartin8]

Will leave in the warning for the time being