The-SS commented 6 months ago

I'm not sure if this is an issue with cvxpygen or just a discussion on the pros/cons of the different solvers.

I noticed that the solve times when solving the problem with C code via the python wrapper varied significantly depending on the solver and sometimes the problem. Sometimes, this resulted in a slow-down instead of a speed-up. So, I thought I'd document the results here. I used the MPC.ipynb as a benchmark. I have some additional observations on other problem instances at the end.

Changes to MPC.ipynb:

Added a SOLVER variable which I set to cp.ECOS, cp.OSQP, ...
val = problem.solve() changed to val = problem.solve(solver=SOLVER)
cpg.generate_code(problem, code_dir='MPC_code') changed to cpg.generate_code(problem, code_dir='MPC_code', solver=SOLVER)
val = prob.solve(eps_abs=1e-3, eps_rel=1e-3, max_iter=4000, polish=False) changed to val = prob.solve()

I am using cvxpygen 0.3.2, cvxpy 1.4.1, and Python 3.10.13. The solver versions are listed below.

ECOS 2.0.12

CVXPY
Solve time: 9.618 ms
Objective function value: 134.722750

CVXPYgen
Solve time: 3.264 ms
Objective function value: 134.722740

OSQP 0.6.3

CVXPY
Solve time: 8.780 ms
Objective function value: 134.722750

CVXPYgen
Solve time: 0.828 ms
Objective function value: 134.703097

CLARABEL 0.6.0

CVXPY
Solve time: 9.794 ms
Objective function value: 134.722750

CVXPYgen
Solve time: 19.171 ms
Objective function value: 134.722745

Note: I tried passing verbose=False to prob.solve(method='CPG') but it didn't change the results much

SCS 3.2.4.post1

CVXPY
Solve time: 6.914 ms
Objective function value: 134.722750

CVXPYgen
Solve time: 1.139 ms
Objective function value: 134.722684

Other observations

I have also tried other problems such as a simple single integrator MPC with a Gaussian chance constraint (reformulated into a deterministic problem) and the distributionally robust portfolio optimization problem in eq. (27) from "Data-driven distributionally robust optimization using the Wasserstein metric: performance guarantees and tractable reformulations".

I have noticed that calling the C code generated using Clarabel is slower than solving the problem with Clarabel directly with cvxpy (as demonstrated above). This sometimes results in an order of magnitude slow-down when using the C code with the Python wrapper.
I have noticed mixed performance with SCS where on some problems calling the C code is faster but for other problems it is slower.
The performance of ECOS is good, but currently, per Issue #45 , it is unusable for any problems involving repeated solves since the results are wrong for all but the initial solution.

PTNobel commented 6 months ago

I'm really surprised by the Clarabel numbers. Pinging @phschiele @goulart-paul to see if they have any ideas what is going on there.

maxschaller commented 6 months ago

We're currently interfacing with Clarabel by compiling its C++ interface and then using the standard cvxpygen Python-C++ interface. This might incur more overhead than just using Clarabel's Python interface with cvxpy. Generating all Rust code (and then using Rust bindings for Python) is something we have in our backlog.

goulart-paul commented 6 months ago

Are you certain that the problems are being solved in the same form when solved in the two different ways? It seems odd that the optimal values differ slightly in any given solver when comparing the two cases.

Since it's an MPC problem, are the equality constraints maybe getting eliminated in one case but not the other? That could explain both the slight difference in optimal value and the degradation of solve times.

I think there is a bit of extra data copying that happens when passing through our C interface (or maybe just the C++ one), but not enough to explain a factor-of-2 solve time difference.

Maybe post the verbose output from one or more of the solvers so we can check there is nothing strange going on in either problem dimensions or settings?

PTNobel commented 6 months ago

@maxschaller do you support quadratic objectives in cone programs?

maxschaller commented 6 months ago

Yes.

The-SS commented 6 months ago

@goulart-paul below are some verbose outputs. I've included two successive calls for the compiled code as well (there's an interesting observation where OSQP is not consistent when repeatedly solving the problem).

Clarabel

-------------------------------------------------------------------------------
                                Numerical solver                               
-------------------------------------------------------------------------------
(CVXPY) Apr 22 11:04:10 AM: Invoking solver CLARABEL  to obtain a solution.
-------------------------------------------------------------
           Clarabel.rs v0.6.0  -  Clever Acronym              

                   (c) Paul Goulart                          
                University of Oxford, 2022                   
-------------------------------------------------------------

problem:
  variables     = 222
  constraints   = 252
  nnz(P)        = 96
  nnz(A)        = 1338
  cones (total) = 2
    :        Zero = 1,  numel = 162
    : Nonnegative = 1,  numel = 90

settings:
  linear algebra: direct / qdldl, precision: 64 bit
  max iter = 200, time limit = Inf,  max step = 0.990
  tol_feas = 1.0e-8, tol_gap_abs = 1.0e-8, tol_gap_rel = 1.0e-8,
  static reg : on, ϵ1 = 1.0e-8, ϵ2 = 4.9e-32
  dynamic reg: on, ϵ = 1.0e-13, δ = 2.0e-7
  iter refine: on, reltol = 1.0e-13, abstol = 1.0e-12,
               max iter = 10, stop ratio = 5.0
  equilibrate: on, min_scale = 1.0e-4, max_scale = 1.0e4
               max iter = 10

iter    pcost        dcost       gap       pres      dres      k/t        μ       step      
---------------------------------------------------------------------------------------------
  0  +1.3642e+02  +8.5492e+01  5.96e-01  3.85e-01  8.24e-02  1.00e+00  5.99e+00   ------   
  1  +1.3270e+02  +1.1556e+02  1.48e-01  1.67e-01  2.87e-02  9.93e-01  3.60e+00  5.67e-01  
  2  +1.3487e+02  +1.2958e+02  4.09e-02  5.04e-02  7.67e-03  5.90e-01  1.47e+00  7.22e-01  
  3  +1.3492e+02  +1.3369e+02  9.17e-03  6.14e-03  9.18e-04  9.70e-02  2.57e-01  8.89e-01  
  4  +1.3478e+02  +1.3449e+02  2.14e-03  2.49e-04  3.78e-05  1.04e-02  3.06e-02  9.73e-01  
  5  +1.3473e+02  +1.3469e+02  2.78e-04  5.60e-06  8.58e-07  9.58e-04  2.98e-03  9.90e-01  
  6  +1.3472e+02  +1.3472e+02  2.00e-05  5.90e-08  9.04e-09  5.86e-05  1.88e-04  9.90e-01  
  7  +1.3472e+02  +1.3472e+02  6.39e-07  6.01e-10  9.23e-11  1.78e-06  5.76e-06  9.90e-01  
  8  +1.3472e+02  +1.3472e+02  7.14e-09  6.02e-12  9.24e-13  1.99e-08  6.41e-08  9.90e-01  
---------------------------------------------------------------------------------------------
Terminated with status = Solved
solve time = 936.54µs
-------------------------------------------------------------------------------
                                    Summary                                    
-------------------------------------------------------------------------------
(CVXPY) Apr 22 11:04:10 AM: Problem status: optimal
(CVXPY) Apr 22 11:04:10 AM: Optimal value: 1.347e+02
(CVXPY) Apr 22 11:04:10 AM: Compilation took 7.584e-03 seconds
(CVXPY) Apr 22 11:04:10 AM: Solver (including time spent in interface) took 1.182e-03 seconds

CVXPY
Solve time: 9.535 ms
Objective function value: 134.722745

-------------------------------------------------------------
           Clarabel.rs v0.6.0  -  Clever Acronym              

                   (c) Paul Goulart                          
                University of Oxford, 2022                   
-------------------------------------------------------------

problem:
  variables     = 222
  constraints   = 252
  nnz(P)        = 96
  nnz(A)        = 1338
  cones (total) = 2
    :        Zero = 1,  numel = 162
    : Nonnegative = 1,  numel = 90

settings:
  linear algebra: direct / qdldl, precision: 64 bit
  max iter = 50, time limit = 1000000.0,  max step = 0.990
  tol_feas = 1.0e-8, tol_gap_abs = 1.0e-8, tol_gap_rel = 1.0e-8,
  static reg : on, ϵ1 = 1.0e-8, ϵ2 = 4.9e-32
  dynamic reg: on, ϵ = 1.0e-13, δ = 2.0e-7
  iter refine: on, reltol = 1.0e-13, abstol = 1.0e-12,
               max iter = 10, stop ratio = 5.0
  equilibrate: on, min_scale = 1.0e-4, max_scale = 1.0e4
               max iter = 10

iter    pcost        dcost       gap       pres      dres      k/t        μ       step      
---------------------------------------------------------------------------------------------
  0  +1.3642e+02  +8.5492e+01  5.96e-01  3.85e-01  8.24e-02  1.00e+00  5.99e+00   ------   
  1  +1.3270e+02  +1.1556e+02  1.48e-01  1.67e-01  2.87e-02  9.93e-01  3.60e+00  5.67e-01  
  2  +1.3487e+02  +1.2958e+02  4.09e-02  5.04e-02  7.67e-03  5.90e-01  1.47e+00  7.22e-01  
  3  +1.3492e+02  +1.3369e+02  9.17e-03  6.14e-03  9.18e-04  9.70e-02  2.57e-01  8.89e-01  
  4  +1.3478e+02  +1.3449e+02  2.14e-03  2.49e-04  3.78e-05  1.04e-02  3.06e-02  9.73e-01  
  5  +1.3473e+02  +1.3469e+02  2.78e-04  5.60e-06  8.58e-07  9.58e-04  2.98e-03  9.90e-01  
  6  +1.3472e+02  +1.3472e+02  2.00e-05  5.90e-08  9.04e-09  5.86e-05  1.88e-04  9.90e-01  
  7  +1.3472e+02  +1.3472e+02  6.39e-07  6.01e-10  9.23e-11  1.78e-06  5.76e-06  9.90e-01  
  8  +1.3472e+02  +1.3472e+02  7.14e-09  6.02e-12  9.24e-13  1.99e-08  6.41e-08  9.90e-01  
---------------------------------------------------------------------------------------------
Terminated with status = Solved
solve time = 16.06696ms

CVXPYgen
Solve time: 16.792 ms
Objective function value: 134.722745

-------------------------------------------------------------
           Clarabel.rs v0.6.0  -  Clever Acronym              

                   (c) Paul Goulart                          
                University of Oxford, 2022                   
-------------------------------------------------------------

problem:
  variables     = 222
  constraints   = 252
  nnz(P)        = 96
  nnz(A)        = 1338
  cones (total) = 2
    :        Zero = 1,  numel = 162
    : Nonnegative = 1,  numel = 90

settings:
  linear algebra: direct / qdldl, precision: 64 bit
  max iter = 50, time limit = 1000000.0,  max step = 0.990
  tol_feas = 1.0e-8, tol_gap_abs = 1.0e-8, tol_gap_rel = 1.0e-8,
  static reg : on, ϵ1 = 1.0e-8, ϵ2 = 4.9e-32
  dynamic reg: on, ϵ = 1.0e-13, δ = 2.0e-7
  iter refine: on, reltol = 1.0e-13, abstol = 1.0e-12,
               max iter = 10, stop ratio = 5.0
  equilibrate: on, min_scale = 1.0e-4, max_scale = 1.0e4
               max iter = 10

iter    pcost        dcost       gap       pres      dres      k/t        μ       step      
---------------------------------------------------------------------------------------------
  0  +1.3642e+02  +8.5492e+01  5.96e-01  3.85e-01  8.24e-02  1.00e+00  5.99e+00   ------   
  1  +1.3270e+02  +1.1556e+02  1.48e-01  1.67e-01  2.87e-02  9.93e-01  3.60e+00  5.67e-01  
  2  +1.3487e+02  +1.2958e+02  4.09e-02  5.04e-02  7.67e-03  5.90e-01  1.47e+00  7.22e-01  
  3  +1.3492e+02  +1.3369e+02  9.17e-03  6.14e-03  9.18e-04  9.70e-02  2.57e-01  8.89e-01  
  4  +1.3478e+02  +1.3449e+02  2.14e-03  2.49e-04  3.78e-05  1.04e-02  3.06e-02  9.73e-01  
  5  +1.3473e+02  +1.3469e+02  2.78e-04  5.60e-06  8.58e-07  9.58e-04  2.98e-03  9.90e-01  
  6  +1.3472e+02  +1.3472e+02  2.00e-05  5.90e-08  9.04e-09  5.86e-05  1.88e-04  9.90e-01  
  7  +1.3472e+02  +1.3472e+02  6.39e-07  6.01e-10  9.23e-11  1.78e-06  5.76e-06  9.90e-01  
  8  +1.3472e+02  +1.3472e+02  7.14e-09  6.02e-12  9.24e-13  1.99e-08  6.41e-08  9.90e-01  
---------------------------------------------------------------------------------------------
Terminated with status = Solved
solve time = 14.615041ms

CVXPYgen
Solve time: 14.749 ms
Objective function value: 134.722745

OSQP

-------------------------------------------------------------------------------
                                Numerical solver                               
-------------------------------------------------------------------------------
(CVXPY) Apr 22 11:06:59 AM: Invoking solver OSQP  to obtain a solution.
-----------------------------------------------------------------
           OSQP v0.6.3  -  Operator Splitting QP Solver
              (c) Bartolomeo Stellato,  Goran Banjac
        University of Oxford  -  Stanford University 2021
-----------------------------------------------------------------
problem:  variables n = 222, constraints m = 252
          nnz(P) + nnz(A) = 1434
settings: linear system solver = qdldl,
          eps_abs = 1.0e-05, eps_rel = 1.0e-05,
          eps_prim_inf = 1.0e-04, eps_dual_inf = 1.0e-04,
          rho = 1.00e-01 (adaptive),
          sigma = 1.00e-06, alpha = 1.60, max_iter = 10000
          check_termination: on (interval 25),
          scaling: on, scaled_termination: off
          warm start: on, polish: on, time_limit: off

iter   objective    pri res    dua res    rho        time
   1   0.0000e+00   2.00e+00   2.00e+02   1.00e-01   1.84e-04s
 100   1.3472e+02   1.85e-06   5.01e-06   6.37e-01   7.96e-04s

status:               solved
solution polish:      unsuccessful
number of iterations: 100
optimal objective:    134.7228
run time:             9.17e-04s
optimal rho estimate: 5.20e-01

-------------------------------------------------------------------------------
                                    Summary                                    
-------------------------------------------------------------------------------
(CVXPY) Apr 22 11:06:59 AM: Problem status: optimal
(CVXPY) Apr 22 11:06:59 AM: Optimal value: 1.347e+02
(CVXPY) Apr 22 11:06:59 AM: Compilation took 6.840e-03 seconds
(CVXPY) Apr 22 11:06:59 AM: Solver (including time spent in interface) took 1.151e-03 seconds

CVXPY
Solve time: 8.698 ms
Objective function value: 134.722750

CVXPYgen
Solve time: 0.842 ms
Objective function value: 134.703097

CVXPYgen
Solve time: 0.285 ms
Objective function value: 134.718937

Note 1: It seems that while the objective function values different between successive calls, are consistent when I rerun the script; i.e. upon rerunning the script, the first CVXPYgen call results in Objective function value: 134.703097 while the second call returns 134.718937.

Note 2: prob.solve(method='CPG', verbose=True) fails with the following error:

Traceback (most recent call last):
  File "/.../MPC_code/cpg_solver.py", line 34, in cpg_solve
    eval(f'cpg_module.set_solver_{standard_settings_names.get(key, key)}(value)')
  File "<string>", line 1, in <module>
AttributeError: module 'MPC_code.cpg_module' has no attribute 'set_solver_verbose'. Did you mean: 'set_solver_rho'?

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/.../cvxpygen_mpc.py", line 100, in <module>
    val = prob.solve(method='CPG', verbose=True)
  File "/.../python3.10/site-packages/cvxpy/problems/problem.py", line 503, in solve
    return solve_func(self, *args, **kwargs)
  File "/.../MPC_code/cpg_solver.py", line 36, in cpg_solve
    raise AttributeError(f'Solver setting "{key}" not available.')
AttributeError: Solver setting "verbose" not available.

SCS

-------------------------------------------------------------------------------
                                Numerical solver                               
-------------------------------------------------------------------------------
(CVXPY) Apr 22 11:16:31 AM: Invoking solver SCS  to obtain a solution.
------------------------------------------------------------------
           SCS v3.2.4 - Splitting Conic Solver
    (c) Brendan O'Donoghue, Stanford University, 2012
------------------------------------------------------------------
problem:  variables n: 222, constraints m: 252
cones:    z: primal zero / dual free vars: 162
      l: linear vars: 90
settings: eps_abs: 1.0e-05, eps_rel: 1.0e-05, eps_infeas: 1.0e-07
      alpha: 1.50, scale: 1.00e-01, adaptive_scale: 1
      max_iters: 100000, normalize: 1, rho_x: 1.00e-06
      acceleration_lookback: 10, acceleration_interval: 10
lin-sys:  sparse-direct-amd-qdldl
      nnz(A): 1338, nnz(P): 96
------------------------------------------------------------------
 iter | pri res | dua res |   gap   |   obj   |  scale  | time (s)
------------------------------------------------------------------
     0| 4.00e+00  4.00e-01  5.91e+01  1.40e+02  1.00e-01  5.10e-04 
   125| 1.28e-07  1.50e-07  1.56e-07  1.35e+02  8.91e-01  1.65e-03 
------------------------------------------------------------------
status:  solved
timings: total: 1.65e-03s = setup: 4.15e-04s + solve: 1.24e-03s
     lin-sys: 9.60e-04s, cones: 2.70e-05s, accel: 1.49e-05s
------------------------------------------------------------------
objective = 134.722745
------------------------------------------------------------------
-------------------------------------------------------------------------------
                                    Summary                                    
-------------------------------------------------------------------------------
(CVXPY) Apr 22 11:16:31 AM: Problem status: optimal
(CVXPY) Apr 22 11:16:31 AM: Optimal value: 1.347e+02
(CVXPY) Apr 22 11:16:31 AM: Compilation took 8.307e-03 seconds
(CVXPY) Apr 22 11:16:31 AM: Solver (including time spent in interface) took 1.907e-03 seconds

CVXPY
Solve time: 11.178 ms
Objective function value: 134.722745

------------------------------------------------------------------
           SCS v3.2.0 - Splitting Conic Solver
    (c) Brendan O'Donoghue, Stanford University, 2012
------------------------------------------------------------------
problem:  variables n: 222, constraints m: 252
cones:    z: primal zero / dual free vars: 162
      l: linear vars: 90
settings: eps_abs: 1.0e-04, eps_rel: 1.0e-04, eps_infeas: 1.0e-07
      alpha: 1.50, scale: 1.00e-01, adaptive_scale: 1
      max_iters: 100000, normalize: 1, rho_x: 1.00e-06
      acceleration_lookback: 0, acceleration_interval: 0
lin-sys:  sparse-direct
      nnz(A): 1338, nnz(P): 96
WARN: aa_init returned NULL, no acceleration applied.
------------------------------------------------------------------
 iter | pri res | dua res |   gap   |   obj   |  scale  | time (s)
------------------------------------------------------------------
     0| 4.00e+00  4.00e-01  5.91e+01  1.40e+02  1.00e-01  7.38e-05 
   100| 4.40e-05  7.01e-07  4.88e-05  1.35e+02  1.00e-01  9.25e-04 
------------------------------------------------------------------
status:  solved
timings: total: 1.23e-03s = setup: 3.03e-04s + solve: 9.27e-04s
     lin-sys: 7.11e-04s, cones: 1.97e-05s, accel: 0.00e+00s
------------------------------------------------------------------
objective = 134.722708
------------------------------------------------------------------

CVXPYgen
Solve time: 1.417 ms
Objective function value: 134.722684

------------------------------------------------------------------
           SCS v3.2.0 - Splitting Conic Solver
    (c) Brendan O'Donoghue, Stanford University, 2012
------------------------------------------------------------------
problem:  variables n: 222, constraints m: 252
cones:    z: primal zero / dual free vars: 162
      l: linear vars: 90
settings: eps_abs: 1.0e-04, eps_rel: 1.0e-04, eps_infeas: 1.0e-07
      alpha: 1.50, scale: 1.00e-01, adaptive_scale: 1
      max_iters: 100000, normalize: 1, rho_x: 1.00e-06
      acceleration_lookback: 0, acceleration_interval: 0
lin-sys:  sparse-direct
      nnz(A): 1338, nnz(P): 96
WARN: aa_init returned NULL, no acceleration applied.
------------------------------------------------------------------
 iter | pri res | dua res |   gap   |   obj   |  scale  | time (s)
------------------------------------------------------------------
     0| 4.00e+00  4.00e-01  5.91e+01  1.40e+02  1.00e-01  6.70e-05 
   100| 4.40e-05  7.01e-07  4.88e-05  1.35e+02  1.00e-01  9.03e-04 
------------------------------------------------------------------
status:  solved
timings: total: 1.17e-03s = setup: 2.64e-04s + solve: 9.05e-04s
     lin-sys: 7.00e-04s, cones: 1.89e-05s, accel: 0.00e+00s
------------------------------------------------------------------
objective = 134.722708
------------------------------------------------------------------

CVXPYgen
Solve time: 1.307 ms
Objective function value: 134.722684

ECOS

-------------------------------------------------------------------------------
                                Numerical solver                               
-------------------------------------------------------------------------------
(CVXPY) Apr 22 11:17:17 AM: Invoking solver ECOS  to obtain a solution.

ECOS 2.0.10 - (C) embotech GmbH, Zurich Switzerland, 2012-15. Web: www.embotech.com/ECOS

It     pcost       dcost      gap   pres   dres    k/t    mu     step   sigma     IR    |   BT
 0  -1.541e-17  -3.019e+01  +3e+02  4e-01  1e-01  1e+00  3e+00    ---    ---    1  1  - |  -  - 
 1  -3.180e+00  -1.454e+01  +1e+02  2e-01  3e-02  5e-01  1e+00  0.5715  1e-01   1  1  2 |  0  0
 2  -9.467e-01  -1.527e+01  +1e+02  2e-01  3e-02  8e-01  1e+00  0.1001  8e-01   2  2  2 |  0  0
 3  +8.764e+00  +3.187e+00  +7e+01  9e-02  1e-02  5e-01  7e-01  0.6484  2e-01   2  2  2 |  0  0
 4  -3.410e+00  -2.423e+01  +6e+01  8e-01  3e-02  3e+00  7e-01  0.3171  8e-01   2  2  2 |  0  0
 5  +3.367e+01  +3.011e+01  +1e+01  1e-01  4e-03  4e-01  2e-01  0.7851  2e-02   2  2  2 |  0  0
 6  +3.328e+01  +3.023e+01  +1e+01  1e-01  4e-03  5e-01  1e-01  0.3932  6e-01   2  2  2 |  0  0
 7  +5.547e+01  +5.465e+01  +3e+00  4e-02  1e-03  3e-01  3e-02  0.9166  2e-01   2  2  2 |  0  0
 8  +5.456e+01  +5.361e+01  +2e+00  1e-01  2e-03  9e-01  2e-02  0.8153  4e-01   2  2  2 |  0  0
 9  +7.907e+01  +7.876e+01  +8e-01  5e-02  6e-04  4e-01  8e-03  0.8486  3e-01   2  2  2 |  0  0
10  +8.319e+01  +8.293e+01  +6e-01  4e-02  5e-04  4e-01  7e-03  0.2930  4e-01   2  2  2 |  0  0
11  +1.110e+02  +1.110e+02  +1e-01  1e-02  1e-04  2e-01  1e-03  0.8838  8e-02   2  2  2 |  0  0
12  +1.173e+02  +1.174e+02  +6e-02  1e-02  1e-04  2e-01  7e-04  0.6858  3e-01   2  2  2 |  0  0
13  +1.271e+02  +1.271e+02  +2e-02  5e-03  4e-05  1e-01  3e-04  0.6970  1e-01   2  2  2 |  0  0
14  +1.282e+02  +1.282e+02  +2e-02  4e-03  4e-05  8e-02  2e-04  0.4104  6e-01   2  2  2 |  0  0
15  +1.328e+02  +1.328e+02  +4e-03  9e-04  9e-06  2e-02  5e-05  0.8582  1e-01   3  2  2 |  0  0
16  +1.339e+02  +1.339e+02  +2e-03  5e-04  4e-06  1e-02  2e-05  0.7662  3e-01   3  3  3 |  0  0
17  +1.344e+02  +1.344e+02  +7e-04  2e-04  2e-06  4e-03  8e-06  0.8780  3e-01   2  2  2 |  0  0
18  +1.346e+02  +1.346e+02  +3e-04  7e-05  7e-07  2e-03  4e-06  0.7965  3e-01   3  2  2 |  0  0
19  +1.346e+02  +1.346e+02  +2e-04  4e-05  4e-07  1e-03  2e-06  0.6744  3e-01   3  2  2 |  0  0
20  +1.347e+02  +1.347e+02  +1e-04  3e-05  3e-07  7e-04  1e-06  0.6376  5e-01   3  2  2 |  0  0
21  +1.347e+02  +1.347e+02  +3e-05  7e-06  7e-08  2e-04  4e-07  0.7643  6e-02   3  2  2 |  0  0
22  +1.347e+02  +1.347e+02  +3e-06  7e-07  6e-09  2e-05  3e-08  0.9500  4e-02   3  2  2 |  0  0
23  +1.347e+02  +1.347e+02  +4e-07  9e-08  9e-10  3e-06  5e-09  0.8649  1e-02   2  1  2 |  0  0
24  +1.347e+02  +1.347e+02  +6e-08  1e-08  1e-10  4e-07  6e-10  0.9113  5e-02   2  2  2 |  0  0
25  +1.347e+02  +1.347e+02  +1e-08  2e-09  2e-11  6e-08  1e-10  0.8434  3e-02   3  1  1 |  0  0

OPTIMAL (within feastol=2.4e-09, reltol=8.2e-11, abstol=1.1e-08).
Runtime: 0.002718 seconds.

-------------------------------------------------------------------------------
                                    Summary                                    
-------------------------------------------------------------------------------
(CVXPY) Apr 22 11:17:17 AM: Problem status: optimal
(CVXPY) Apr 22 11:17:17 AM: Optimal value: 1.347e+02
(CVXPY) Apr 22 11:17:17 AM: Compilation took 7.550e-03 seconds
(CVXPY) Apr 22 11:17:17 AM: Solver (including time spent in interface) took 2.762e-03 seconds

CVXPY
Solve time: 11.043 ms
Objective function value: 134.722746

CVXPYgen
Solve time: 2.624 ms
Objective function value: 134.722740

Note: I only ran the problem once with ECOS due to #45 (I am not using cvxpygen 0.3.3 yet).

Summary:

Solver	CVXPY solve time	CVXPY Obj val	CVXPYgen solve time (1)	CVXPYgen Obj val (1)	CVXPYgen solve time (2)	CVXPYgen Obj val (2)
Clarabel	9.535 ms	134.722745	16.792 ms	134.722745	14.749 ms	134.722745
OSQP	8.698 ms	134.722750	0.842 ms	134.703097	0.285 ms	134.718937
SCS	11.178 ms	134.722745	1.417 ms	134.722684	1.307 ms	134.722684
ECOS	11.043 ms	134.722746	2.624 ms	134.722740	N/A	N/A

Some observations

Clarabel was the only solver that resulted in the same obj val using cvxpy and the compiled code.
OSQP: as noted earlier, it seems that while the objective function values different between successive calls, they are consistent when I rerun the script; i.e. upon rerunning the script, the first CVXPYgen call results in Objective function value: 134.703097 while the second call returns 134.718937. Not sure if something similar to #45, but less obvious, is happening with OSQP. @maxschaller any thoughts on this?
Passing verbose=True to prob.solve(method='CPG') fails with some solvers.

goulart-paul commented 6 months ago

It seems very odd that the first Clarabel solve took 936.54µs, but then the two subsequent ones took ~15ms even though the problem appears to be identical (i.e. same number of nonzeros, cones and solution). Is it maybe not being compiled with "--release" in one case? That's the only thing I can think of that would make that much difference.

The most likely explanation for inconsistency in OSQP is that the solver behaviour is determined in part by the relative factor and solve times, and these can vary between solvers. You can change the settings to get deterministic behaviour though. See here.

maxschaller commented 6 months ago

Confirming both. We must explicitly set CMAKE_BUILD_TYPE to Release when including the Clarabel subdirectory in the generated code. I'll work on this later today.

We allow users to have matrices as parameters, meaning that OSQP codegen is used with a different embedded flag that also allows time based updates, which gives the observed differences on general purpose machines. This is not related to #45.

Yes, some solvers don't provide verbose in embedded mode. For OSQP, just add enable_settings=['verbose'] when calling generate_code.

maxschaller commented 6 months ago

@The-SS: I cannot reproduce your results. For the MPC example, I get a speed-up of 8-10x with cvxpygen compared to cvxpy, using Clarabel compiled in Debug mode. After switching to Release, the speed-up becomes 10-12x. I was using AppleClang in these experiments. Which OS and compiler are you using?

The-SS commented 6 months ago

@maxschaller I'm on MacOS Sonoma 14.1.1 and should be using AppleClang as well. Here's cvxpygen's output (I only edited the fulls paths it prints):

Generating code with CVXPYgen ...
CVXPYgen finished generating code.
Compiling python wrapper with CVXPYgen ... 
-- The C compiler identification is AppleClang 15.0.0.15000040
-- The CXX compiler identification is AppleClang 15.0.0.15000040
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Clarabel.cpp: `Eigen3` package found.
-- Configuring done (0.7s)
-- Generating done (0.0s)
-- Build files have been written to: /.../MPC_code/c/build
    Updating crates.io index
   Compiling proc-macro2 v1.0.81
   Compiling unicode-ident v1.0.12
   Compiling syn v1.0.109
   Compiling fnv v1.0.7
   Compiling strsim v0.10.0
   Compiling ident_case v1.0.1
   Compiling autocfg v1.2.0
   Compiling thiserror v1.0.59
   Compiling either v1.11.0
   Compiling once_cell v1.19.0
   Compiling cfg-if v1.0.0
   Compiling lazy_static v1.4.0
   Compiling itertools v0.11.0
   Compiling num-traits v0.2.18
   Compiling quote v1.0.36
   Compiling syn v2.0.60
   Compiling amd v0.2.2
   Compiling thiserror-impl v1.0.59
   Compiling enum_dispatch v0.3.13
   Compiling darling_core v0.14.4
   Compiling darling_macro v0.14.4
   Compiling darling v0.14.4
   Compiling derive_builder_core v0.11.2
   Compiling derive_builder_macro v0.11.2
   Compiling derive_builder v0.11.2
   Compiling clarabel v0.6.0 (/.../MPC_code/c/solver_code/Clarabel.rs)
warning: unused import: `adjoint::*`
  --> /.../MPC_code/c/solver_code/Clarabel.rs/src/algebra/mod.rs:24:9
   |
24 | pub use adjoint::*;
   |         ^^^^^^^^^^
   |
   = note: `#[warn(unused_imports)]` on by default

warning: unused import: `reshaped::*`
  --> /.../MPC_code/c/solver_code/Clarabel.rs/src/algebra/mod.rs:30:9
   |
30 | pub use reshaped::*;
   |         ^^^^^^^^^^^

warning: unused import: `symmetric::*`
  --> /.../MPC_code/c/solver_code/Clarabel.rs/src/algebra/mod.rs:32:9
   |
32 | pub use symmetric::*;
   |         ^^^^^^^^^^^^

warning: unused import: `vecmath::*`
  --> /.../MPC_code/c/solver_code/Clarabel.rs/src/algebra/mod.rs:33:9
   |
33 | pub use vecmath::*;
   |         ^^^^^^^^^^

warning: unused import: `utils::*`
 --> /.../MPC_code/c/solver_code/Clarabel.rs/src/algebra/csc/mod.rs:6:9
  |
6 | pub use utils::*;
  |         ^^^^^^^^

warning: unused import: `matrix_math::*`
 --> /.../examples/MPC_code/c/solver_code/Clarabel.rs/src/algebra/csc/mod.rs:8:9
  |
8 | pub use matrix_math::*;
  |         ^^^^^^^^^^^^^^

warning: unused import: `block_concatenate::*`
  --> /.../MPC_code/c/solver_code/Clarabel.rs/src/algebra/csc/mod.rs:10:9
   |
10 | pub use block_concatenate::*;
   |         ^^^^^^^^^^^^^^^^^^^^

warning: unused import: `info_print::*`
  --> /.../MPC_code/c/solver_code/Clarabel.rs/src/solver/implementations/default/mod.rs:21:9
   |
21 | pub use info_print::*;
   |         ^^^^^^^^^^^^^

warning: `clarabel` (lib) generated 8 warnings (run `cargo fix --lib -p clarabel` to apply 8 suggestions)
   Compiling clarabel_c v0.1.0 (/.../MPC_code/c/solver_code/rust_wrapper)
    Finished dev [unoptimized + debuginfo] target(s) in 5.95s
     Ignored package `cbindgen v0.24.5` is already installed, use --force to override
warning: be sure to add `/-/.cargo/bin` to your PATH to be able to run the installed binaries
[  0%] Built target libclarabel_c
[ 33%] Building C object CMakeFiles/cpg.dir/src/cpg_workspace.c.o
[ 66%] Building C object CMakeFiles/cpg.dir/src/cpg_solve.c.o
[100%] Linking C static library out/libcpg.a
[100%] Built target cpg
ld: warning: duplicate -rpath '/../anaconda3/envs/test/lib' ignored
ld: warning: object file (/.../MPC_code/c/build/out/libcpg.a[2](cpg_workspace.c.o)) was built for newer 'macOS' version (14.0) than being linked (11.1)
ld: warning: object file (/.../MPC_code/c/build/out/libcpg.a[3](cpg_solve.c.o)) was built for newer 'macOS' version (14.0) than being linked (11.1)
CVXPYgen finished compiling python wrapper.

Are any of the warnings causing this issue? Am I missing something?

Thanks!

maxschaller commented 6 months ago

Nothing overly suspicious in your output. Try pip install cvxpygen==0.3.4 in case the build type Debug versus Release makes a larger difference for you.

The-SS commented 6 months ago

That did help a lot. Here are the current solve times (solving with cvxpy three times then cvxpygen three times):

CVXPY
Solve time: 10.606 ms
Objective function value: 134.722745

CVXPY
Solve time: 1.757 ms
Objective function value: 134.722745

CVXPY
Solve time: 1.654 ms
Objective function value: 134.722745

CVXPYgen
Solve time: 1.227 ms
Objective function value: 134.722745

CVXPYgen
Solve time: 0.909 ms
Objective function value: 134.722745

CVXPYgen
Solve time: 0.874 ms
Objective function value: 134.722745

cvxpygen 0.3.4 helped resolve the slow-down. However, in terms of a speed-up, excluding cvxpy's initial solve, cvxpygen is faster than cvxpy maybe by a factor of 2, but not 8-10x faster. Was the 8-10x speed-up you mentioned against cvxpy's initial solve (as is the case in the mpc script)?

maxschaller commented 6 months ago

That's great. Yes, this is expected since cvxpy will cache the canonicalization after the first solve.

The-SS commented 6 months ago

Great! Just wanted to confirm. I think that resolves this issue. Thanks a lot.

cvxgrp / cvxpygen

Solve time using C code with python wrapper varies based on the chosen solver #46

ECOS 2.0.12

OSQP 0.6.3

CLARABEL 0.6.0

SCS 3.2.4.post1

Other observations

Clarabel

OSQP

SCS

ECOS

Summary:

Some observations