Migrate backend to cvxpy

[robertmartin8]

In general, PyPortfolioOpt receives positive feedback from users, professional and retail alike. The main plus is an intuitive interface that is quite easy to get up and running with.

However, I receive repeated (constructive) criticism on one area in particular – the use the scipy optimiser on the backend. This is a generic nonlinear constrained optimiser – so in theory it should be able to make progress on any combination of objective functions and constraint. The truth is that it doesn't – optimisation is generally a difficult computational problem, and it is easy for the optimiser to get stuck in local minima (see #75 for a reproducible example).

The decision to use a scipy backend, in some ways, was "lazy development" – I hadn't started my undergrad when I first built PyPortfolioOpt and didn't have the technical ability to properly implement convex optimisation "by hand". However, I think I am now intellectually capable of rebuilding the backend properly.

cvxpy is a modelling language for convex optimisation problems – essentially, if you frame your convex opt problem in their language, they provide strong guarantees about the resulting solution. I want to shift the majority of the optimisation from scipy to cvxpy. Scipy will still remain as an alternative if people want to use nonconvex objective functions, but the standard efficient frontier stuff should use cvxpy.

I anticipate that this will result in breaking changes, though I will try to minimise this. I currently like the end-user API of PyPortfolioOpt, even if it was heavily inspired by the scipy optimiser. Note: this will not affect HRP, Black-Litterman, etc.

Happy to discuss this further and receive feedback, since it is quite a major undertaking.

[robertmartin8]

@schneiderfelipe Let me know what you think. I've been playing around with cvxpy and I think most of the Markowitz stuff can be easily translated while preserving the same frontend API for specifying bounds.

[robertmartin8]

The discrete allocation routine (currently using PuLP) can easily be replaced – the syntax of PuLP is very similar to that of cvxpy. This might solve issues like #76 and #63.

[schneiderfelipe]

@schneiderfelipe Let me know what you think. I've been playing around with cvxpy and I think most of the Markowitz stuff can be easily translated while preserving the same frontend API for specifying bounds.

I think any extra backend is good. I've never used cvxpy, but I've read it's quite powerful. Should the code be replaced, or isn't it better to keep more than one concurrent implementations? This way, one might change/compare things in the future.

[robertmartin8]

@schneiderfelipe Are you thinking of something like ef = EfficientFrontier(backend="scipy")?

[schneiderfelipe]

@schneiderfelipe Are you thinking of something like ef = EfficientFrontier(backend="scipy")?

Exactly. Methods and functions could become private (like _efficient_frontier_max_sharpe_scipy(...)) and adhere to a standard calling signature, such that when something is called (ef.max_sharpe()), it delegates the work (an if and simple call/return instructions). It also makes it easy for testing specific implementations separately.

[robertmartin8]

@schneiderfelipe

I've started building v1.0.0 – it is actually proving very difficult from an architecture perspective to support split backends for everything. The issue is that all of the objective functions are contained in a different module, so I'd have to refactor all the objective functions to have scipy/cvxpy options.

At the same time, I have been wowed cvxpy – it's syntax is extremely intuitive and clean. For example, you can literally add objective functions. Let's say I start with a baseline objective which is to minimise the portfolio variance. This is straightforward:

w = cp.Variable(n_assets)  # weights to optimise
variance = cp.quad_form(w, cov_matrix)
objective = cp.Minimize(variance)
opt = cp.Problem(objective, constraints)
opt.solve()

Now let's say I also want "L2 regularisation" (which I have currently implemented with the gamma parameter). Currently, with scipy, I have had to redefine every objective function to include the gamma term. However, in cvxpy I can just add this objective:

L2_reg_obj = cp.Minimize(cp.norm(w))
new_objective = objective + L2_reg_obj  # or use +=
opt = cp.Problem(new_objective, constraints)

This allows for true modularisation, so within the objective_functions module I can now add advanced (convex) objectives like minimise transaction costs etc, then users can pick and choose them to construct their own complex objectives.

This is less of an issue for constraints since you can also += constraints in scipy, but the cvxpy syntax is significantly better:

constraints = [{"type": "eq", "fun": lambda x: np.sum(x) - 1}]
# VERSUS
constraints = [cp.sum(x) == 1]

This isn't to say that scipy is completely useless. If a user adds a nonconvex objective or nonlinear constraint, cvxpy will fail – though part of the beauty of the cvxpy type system is that it will fail definitively (if the problem is not solvable using convex opt, cvxpy figures that out and tells you).

Therefore I am inclined to add a nonconvex_optimization(objective, constraints, solver="SLSQP") method that relies on the scipy backend (probably will be quite a thin wrapper). In the case of cvxpy optimisation failure, I will catch the error and return to the user something like:

OptimizationError:  this problem cannot be solved by convex optimisation. 

Please ensure you have convex objectives and linear constraints, 
or try nonconvex_optimization (see the caveats in the documentation)

I apologise for the deluge of information – it is very helpful for me to write down my thinking regarding the architecture in a public place. As always, let me know which bits you agree/disagree with :)

[schneiderfelipe]

@robertmartin8 I didn't use cvxpy before, but it now looks like it's much more robust than scipy (for the applications in PyPortfolioOpt, at least). I believe it is the right decision to switch entirely (or almost entirely) to cvxpy :rocket:. And thanks for introducing me to this tool!

[robertmartin8]

I've now migrated the for "core" functions to cvxpy (max sharpe, min vol, efficient risk/return), with custom constraints and additional objectives (e.g L2 reg).

The remaining TODOs for the migration are:

• Add generic convex_optimize method for custom convex objectives/constraints
• Add generic nonconvex_optimize as a fallback (using scipy.optimize).
• Settle CVaR optimisation – either implement something or remove completely.
• Look for any other breaking changes in the pypfopt API (e.g in HRP). Hopefully this will be the last time I have to make major breaking changes