scipy.sparse.linalg.eigen.arpack.arpack.ArpackNoConvergence: ARPACK error -1: No convergence

[dev590t]

Describe the bug Get scipy.sparse.linalg.eigen.arpack.arpack.ArpackNoConvergence: ARPACK error -1: No convergence for some input data

Code sample

    df = pd.read_csv(file, parse_dates = True,date_parser=pd.Timestamp, index_col="date")
    df.dropna(how='any',inplace=True,axis=1)
    mu = expected_returns.mean_historical_return(df)
    S = risk_models.semicovariance(df)
    ef = EfficientFrontier(mu, S)
    weights = ef.max_sharpe()

get

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "PyPortfolioOpt/pypfoptUserGuide.py", line 42, in <module>
    f()
  File "PyPortfolioOpt/pypfoptUserGuide.py", line 36, in f
    weights = ef.max_sharpe()
  File "/home/workDir/PyPortfolioOpt/__pypackages__/3.8/lib/pypfopt/efficient_frontier/efficient_frontier.py", line 288, in max_sharpe
    self._solve_cvxpy_opt_problem()
  File "/home/workDir/PyPortfolioOpt/__pypackages__/3.8/lib/pypfopt/base_optimizer.py", line 292, in _solve_cvxpy_opt_problem
    self._opt.solve(
  File "/home/workDir/PyPortfolioOpt/__pypackages__/3.8/lib/cvxpy/problems/problem.py", line 473, in solve
    return solve_func(self, *args, **kwargs)
  File "/home/workDir/PyPortfolioOpt/__pypackages__/3.8/lib/cvxpy/problems/problem.py", line 966, in _solve
    data, solving_chain, inverse_data = self.get_problem_data(
  File "/home/workDir/PyPortfolioOpt/__pypackages__/3.8/lib/cvxpy/problems/problem.py", line 581, in get_problem_data
    solving_chain = self._construct_chain(
  File "/home/workDir/PyPortfolioOpt/__pypackages__/3.8/lib/cvxpy/problems/problem.py", line 809, in _construct_chain
    return construct_solving_chain(self, candidate_solvers, gp=gp,
  File "/home/workDir/PyPortfolioOpt/__pypackages__/3.8/lib/cvxpy/reductions/solvers/solving_chain.py", line 155, in construct_solving_chain
    reductions = _reductions_for_problem_class(problem, candidates, gp)
  File "/home/workDir/PyPortfolioOpt/__pypackages__/3.8/lib/cvxpy/reductions/solvers/solving_chain.py", line 83, in _reductions_for_problem_class
    if not gp and not problem.is_dcp():
  File "/home/workDir/PyPortfolioOpt/__pypackages__/3.8/lib/cvxpy/utilities/performance_utils.py", line 73, in _compute_once
    result = func(self, *args, **kwargs)
  File "/home/workDir/PyPortfolioOpt/__pypackages__/3.8/lib/cvxpy/problems/problem.py", line 241, in is_dcp
    return all(
  File "/home/workDir/PyPortfolioOpt/__pypackages__/3.8/lib/cvxpy/problems/problem.py", line 242, in <genexpr>
    expr.is_dcp(dpp) for expr in self.constraints + [self.objective])
  File "/home/workDir/PyPortfolioOpt/__pypackages__/3.8/lib/cvxpy/problems/objective.py", line 153, in is_dcp
    return self.args[0].is_convex()
  File "/home/workDir/PyPortfolioOpt/__pypackages__/3.8/lib/cvxpy/utilities/performance_utils.py", line 73, in _compute_once
    result = func(self, *args, **kwargs)
  File "/home/workDir/PyPortfolioOpt/__pypackages__/3.8/lib/cvxpy/atoms/atom.py", line 176, in is_convex
    elif self.is_atom_convex():
  File "/home/workDir/PyPortfolioOpt/__pypackages__/3.8/lib/cvxpy/atoms/quad_form.py", line 66, in is_atom_convex
    return P.is_constant() and P.is_psd()
  File "/home/workDir/PyPortfolioOpt/__pypackages__/3.8/lib/cvxpy/utilities/performance_utils.py", line 73, in _compute_once
    result = func(self, *args, **kwargs)
  File "/home/workDir/PyPortfolioOpt/__pypackages__/3.8/lib/cvxpy/expressions/constants/constant.py", line 216, in is_psd
    self._psd_test = eig_util.is_psd_within_tol(self.value, EIGVAL_TOL)
  File "/home/workDir/PyPortfolioOpt/__pypackages__/3.8/lib/cvxpy/utilities/eigvals.py", line 58, in is_psd_within_tol
    ev = SA_eigsh(-temp)
  File "/home/workDir/PyPortfolioOpt/__pypackages__/3.8/lib/cvxpy/utilities/eigvals.py", line 36, in SA_eigsh
    return sparla.eigsh(A, k=1, sigma=sigma, which='SA', return_eigenvectors=False)
  File "/home/workDir/PyPortfolioOpt/__pypackages__/3.8/lib/scipy/sparse/linalg/eigen/arpack/arpack.py", line 1690, in eigsh
    params.iterate()
  File "/home/workDir/PyPortfolioOpt/__pypackages__/3.8/lib/scipy/sparse/linalg/eigen/arpack/arpack.py", line 570, in iterate
    self._raise_no_convergence()
  File "/home/workDir/PyPortfolioOpt/__pypackages__/3.8/lib/scipy/sparse/linalg/eigen/arpack/arpack.py", line 376, in _raise_no_convergence
    raise ArpackNoConvergence(msg % (num_iter, k_ok, self.k), ev, vec)
scipy.sparse.linalg.eigen.arpack.arpack.ArpackNoConvergence: ARPACK error -1: No convergence (1721 iterations, 0/1 eigenvectors converged)

version pyportfolioopt 1.4.1 cvxpy 1.1.18

Additional context According https://github.com/cvxpy/cvxpy/issues/1424, it is a problem from cvxpy. The version 1.1.18 of cvxpy should contain a fix https://github.com/cvxpy/cvxpy/issues/1532. But v1.1.18 doesn't solve my problem. I have try to use psd_wrap to bypass the issue as describe in https://github.com/cvxpy/cvxpy/issues/1424#issuecomment-865967780, but that isn't run for me. I'm not a cvxpy programmer, maybe I don't use in right way. I have change PyPortfolioOpt/pypfopt/base_optimizer.py, line 276:

-                self._opt = cp.Problem(cp.Minimize(self._objective), self._constraints)
+                self._opt = cp.Problem(cp.Minimize(psd_wrap(self._objective)), self._constraints)

[robertmartin8]

What does your price dataframe look like? Are there any infinities, or data issues?

[dev590t]

I have decrease the resolution of input data to bypass the problem.

According cvxpy/cvxpy#1424, even if the input matrix is correct, cvxpy can raise this error. PyPortfolio maybe crash because this bug from cvxpy. That can limit usage of PyPortfolio in some case. Maybe that could be interested Pyportfolio add a psd_wrap protection against it

Currently, decrease the resolution is the only solution possible. This couldn't very practice if user want on higher resolution data.

[robertmartin8]

Thanks for sharing your solution! I'll see if I can catch the error or do a check. In theory, PyPortfolioOpt makes sure that the matrix is positive semidefinite first.

[dev590t]

@robertmartin8 I don't exclude the possibility that exist some issue in my data. I haven't check it before I submit the issue. It is also possible that the problem is from my side. But if PyPortfolioOpt do already a check, it could be nice to have a warning for user that the input data has some problem, user can understand why that doesn't run, and solve the issue immediately. And thanks for this very useful library.