SolverError: Solver 'OSQP' failed when using max_sharpe

[jujbates]

I am running into an error: SolverError: Solver 'OSQP' failed. Try another solver, or solve with verbose=True for more information.

I import the dataset. yahoo_sp500_adj_close_interim.csv.zip

df = pd.read_csv('yahoo_sp500_adj_close_interim.csv', index_col=['date'])

I am trying to run the dataset over two different risk models and then see the max sharpe ratio but when I take the max sharpe I end up with the error.

I was thinking its because the dataset is to wide because when I reduce it to 300 columns it works and 301 it breaks.

I checked the results with min_volitility and that worked with all assets. I saw this check from #86 and #82 .

When I check

from pypfopt.expected_returns import mean_historical_return
from pypfopt.risk_models import CovarianceShrinkage, sample_cov
from pypfopt.efficient_frontier import EfficientFrontier

mu = mean_historical_return(df)

S = sample_cov(df)
ef = EfficientFrontier(mu, S)
weights = ef.max_sharpe()

S = CovarianceShrinkage(df).ledoit_wolf()
ef = EfficientFrontier(mu, S)
weights = ef.max_sharpe()

S = sample_cov(df)
ef = EfficientFrontier(mu, S)
weights = ef.min_volitility()

Does anyone have an idea what this issue is? Or point my to a resource that might help me fix the issue.

[robertmartin8]

How many rows do you have in the dataset? Portfolio optimisation normally runs into issues when you don't have enough data relative to the number of assets (ill conditioned cov matrix).

[jujbates]

I have a year of data (252) and planned on increasing once I got the optimizer up and running. I updated the data to 10 years (2516) and now I am getting a different error.

OptimizationError: Please check your objectives/constraints or use a different solver.

Below is my new large dataset used with the same covariance functions.

yahoo_sp500_adj_close_interim_2.csv.zip

[robertmartin8]

@jujbates

I've tried running this locally. I can reproduce the failure for max_sharpe, but it works fine for min_volatility. I suspect the solver is having difficulty optimising for Sharpe (a comparatively difficult optimisation problem) for such a large set of assets. I was able to get it to work by using a different solver:

df = pd.read_csv("yahoo_sp500_adj_close_interim.csv", parse_dates=["date"], index_col="date")
mu = mean_historical_return(df)
S = sample_cov(df)
ef = EfficientFrontier(mu, S, solver="SCS")
weights = ef.max_sharpe()

I would add a note of caution, however. PyPortfolioOpt is designed for asset allocation rather than asset selection. In general, I would try to narrow down which stocks you are interested in first (<100) before optimising, rather than using portfolio optimisation to select assets. Just me $0.02.

Hope this helps!

Robert