search

robotology / osqp-eigen

Simple Eigen-C++ wrapper for OSQP library
https://robotology.github.io/osqp-eigen/
BSD 3-Clause "New" or "Revised" License
395 stars 118 forks source link

Avoid non-deterministic QPUnitTests failure by setting `polish` to true in QPUnitTests #172

Closed traversaro closed 3 weeks ago

traversaro commented 3 weeks ago

This was a bit hidden by https://github.com/robotology/osqp-eigen/issues/161, but I recently noticed that while most time the test QPUnitTests the output was:

traversaro@IITBMP014LW012:~/robotology-superbuild/buildapt/src/OsqpEigen$ ctest -VV -R QPUnitTest
UpdateCTestConfiguration  from :/home/traversaro/robotology-superbuild/buildapt/src/OsqpEigen/DartConfiguration.tcl
Parse Config file:/home/traversaro/robotology-superbuild/buildapt/src/OsqpEigen/DartConfiguration.tcl
UpdateCTestConfiguration  from :/home/traversaro/robotology-superbuild/buildapt/src/OsqpEigen/DartConfiguration.tcl
Parse Config file:/home/traversaro/robotology-superbuild/buildapt/src/OsqpEigen/DartConfiguration.tcl
Test project /home/traversaro/robotology-superbuild/buildapt/src/OsqpEigen
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 2
    Start 2: QPUnitTests

2: Test command: /home/traversaro/robotology-superbuild/buildapt/src/OsqpEigen/bin/QPUnitTests
2: Working Directory: /home/traversaro/robotology-superbuild/buildapt/src/OsqpEigen/tests
2: Test timeout computed to be: 1500
2: Randomness seeded to: 3883189244
2: -----------------------------------------------------------------
2:            OSQP v0.6.3  -  Operator Splitting QP Solver
2:               (c) Bartolomeo Stellato,  Goran Banjac
2:         University of Oxford  -  Stanford University 2021
2: -----------------------------------------------------------------
2: problem:  variables n = 2, constraints m = 0
2:           nnz(P) + nnz(A) = 3
2: settings: linear system solver = qdldl,
2:           eps_abs = 1.0e-03, eps_rel = 1.0e-03,
2:           eps_prim_inf = 1.0e-04, eps_dual_inf = 1.0e-04,
2:           rho = 1.00e-01 (adaptive),
2:           sigma = 1.00e-06, alpha = 1.00, max_iter = 4000
2:           check_termination: on (interval 25),
2:           scaling: on, scaled_termination: off
2:           warm start: on, polish: off, time_limit: off
2:
2: iter   objective    pri res    dua res    rho        time
2:    1  -1.6875e+00   0.00e+00   3.75e-06   1.00e-01   3.98e-05s
2:   25  -1.6875e+00   0.00e+00   3.33e-16   1.00e-01   4.64e-05s
2:
2: status:               solved
2: number of iterations: 25
2: optimal objective:    -1.6875
2: run time:             4.95e-05s
2: optimal rho estimate: 1.00e-06
2:
2: [OsqpEigen::Data::setHessianMatrix] Please set the number of variables before add the hessian matrix.
2: -----------------------------------------------------------------
2:            OSQP v0.6.3  -  Operator Splitting QP Solver
2:               (c) Bartolomeo Stellato,  Goran Banjac
2:         University of Oxford  -  Stanford University 2021
2: -----------------------------------------------------------------
2: problem:  variables n = 2, constraints m = 3
2:           nnz(P) + nnz(A) = 7
2: settings: linear system solver = qdldl,
2:           eps_abs = 1.0e-03, eps_rel = 1.0e-03,
2:           eps_prim_inf = 1.0e-04, eps_dual_inf = 1.0e-04,
2:           rho = 1.00e-01 (adaptive),
2:           sigma = 1.00e-06, alpha = 1.00, max_iter = 4000
2:           check_termination: on (interval 25),
2:           scaling: on, scaled_termination: off
2:           warm start: on, polish: off, time_limit: off
2:
2: iter   objective    pri res    dua res    rho        time
2:    1  -4.9384e-03   1.00e+00   2.00e+02   1.00e-01   1.37e-05s
2:   50   1.8800e+00   1.91e-07   7.50e-07   1.38e+00   2.69e-05s
2:
2: status:               solved
2: number of iterations: 50
2: optimal objective:    1.8800
2: run time:             2.93e-05s
2: optimal rho estimate: 1.36e+00
2:
2: expectedSolution: 0.3
2: 0.7
2: solver.getSolution(): 0.3
2: 0.7
2: tolerance: 0.0001
2: expectedSolution-solver.getSolution(): -1.91114e-07
2:  1.90972e-07
2: ===============================================================================
2: All tests passed (14 assertions in 2 test cases)
2:
1/1 Test #2: QPUnitTests ......................   Passed    0.00 sec

in some rare case, the test was failing as the solution found was much less accurate:

    Test #2: QPUnitTests ......................***Failed    0.00 sec
Randomness seeded to: 3198543723
-----------------------------------------------------------------
           OSQP v0.6.3  -  Operator Splitting QP Solver
              (c) Bartolomeo Stellato,  Goran Banjac
        University of Oxford  -  Stanford University 2021
-----------------------------------------------------------------
problem:  variables n = 2, constraints m = 0
          nnz(P) + nnz(A) = 3
settings: linear system solver = qdldl,
          eps_abs = 1.0e-03, eps_rel = 1.0e-03,
          eps_prim_inf = 1.0e-04, eps_dual_inf = 1.0e-04,
          rho = 1.00e-01 (adaptive),
          sigma = 1.00e-06, alpha = 1.00, max_iter = 4000
          check_termination: on (interval 25),
          scaling: on, scaled_termination: off
          warm start: on, polish: off, time_limit: off

iter   objective    pri res    dua res    rho        time
   1  -1.6875e+00   0.00e+00   3.75e-06   1.00e-01   3.82e-05s
  25  -1.6875e+00   0.00e+00   3.33e-16   1.00e-01   4.52e-05s

status:               solved
number of iterations: 25
optimal objective:    -1.6875
run time:             4.83e-05s
optimal rho estimate: 1.00e-06

[OsqpEigen::Data::setHessianMatrix] Please set the number of variables before add the hessian matrix.
-----------------------------------------------------------------
           OSQP v0.6.3  -  Operator Splitting QP Solver
              (c) Bartolomeo Stellato,  Goran Banjac
        University of Oxford  -  Stanford University 2021
-----------------------------------------------------------------
problem:  variables n = 2, constraints m = 3
          nnz(P) + nnz(A) = 7
settings: linear system solver = qdldl,
          eps_abs = 1.0e-03, eps_rel = 1.0e-03,
          eps_prim_inf = 1.0e-04, eps_dual_inf = 1.0e-04,
          rho = 1.00e-01 (adaptive),
          sigma = 1.00e-06, alpha = 1.00, max_iter = 4000
          check_termination: on (interval 25),
          scaling: on, scaled_termination: off
          warm start: on, polish: off, time_limit: off

iter   objective    pri res    dua res    rho        time
   1  -4.9384e-03   1.00e+00   2.00e+02   1.00e-01   1.02e-04s
  50   1.8798e+00   1.23e-03   2.46e-05   1.00e-01   1.16e-04s

status:               solved
number of iterations: 50
optimal objective:    1.8798
run time:             1.19e-04s
optimal rho estimate: 1.38e+00

expectedSolution: 0.3
0.7
solver.getSolution(): 0.298771
0.701228
tolerance: 0.0001
expectedSolution-solver.getSolution(): 0.00122892
 -0.001228

QPUnitTests is a Catch2 v3.4.0 host application.
Run with -? for options

-------------------------------------------------------------------------------
QPProblem
-------------------------------------------------------------------------------
/home/traversaro/robotology-superbuild/src/OsqpEigen/tests/QPTest.cpp:46
...............................................................................

/home/traversaro/robotology-superbuild/src/OsqpEigen/tests/QPTest.cpp:99: FAILED:
  REQUIRE( solver.getSolution().isApprox(expectedSolution, tolerance) )
with expansion:
  false

===============================================================================
test cases:  2 |  1 passed | 1 failed
assertions: 14 | 13 passed | 1 failed

0% tests passed, 1 tests failed out of 1

Total Test time (real) =   2.51 sec

The following tests FAILED:
          2 - QPUnitTests (Failed)

To remove the non-determinism, I also tried to set solver.settings()->setAdaptiveRho(false); to remove the non-determinism (see https://github.com/google/osqp-cpp?tab=readme-ov-file#faq), but this only made the failure deterministic, with output:

traversaro@IITBMP014LW012:~/robotology-superbuild/buildapt/src/OsqpEigen$ ctest --output-on-failure -R QPUnitTests --repeat until-fail:100000
Test project /home/traversaro/robotology-superbuild/buildapt/src/OsqpEigen
    Start 2: QPUnitTests
    Test #2: QPUnitTests ......................***Failed    0.00 sec
Randomness seeded to: 3354483813
-----------------------------------------------------------------
           OSQP v0.6.3  -  Operator Splitting QP Solver
              (c) Bartolomeo Stellato,  Goran Banjac
        University of Oxford  -  Stanford University 2021
-----------------------------------------------------------------
problem:  variables n = 2, constraints m = 0
          nnz(P) + nnz(A) = 3
settings: linear system solver = qdldl,
          eps_abs = 1.0e-03, eps_rel = 1.0e-03,
          eps_prim_inf = 1.0e-04, eps_dual_inf = 1.0e-04,
          rho = 1.00e-01 (adaptive),
          sigma = 1.00e-06, alpha = 1.00, max_iter = 4000
          check_termination: on (interval 25),
          scaling: on, scaled_termination: off
          warm start: on, polish: off, time_limit: off

iter   objective    pri res    dua res    rho        time
   1  -1.6875e+00   0.00e+00   3.75e-06   1.00e-01   6.05e-05s
  25  -1.6875e+00   0.00e+00   3.33e-16   1.00e-01   6.79e-05s

status:               solved
number of iterations: 25
optimal objective:    -1.6875
run time:             7.12e-05s
optimal rho estimate: 1.00e-06

[OsqpEigen::Data::setHessianMatrix] Please set the number of variables before add the hessian matrix.
-----------------------------------------------------------------
           OSQP v0.6.3  -  Operator Splitting QP Solver
              (c) Bartolomeo Stellato,  Goran Banjac
        University of Oxford  -  Stanford University 2021
-----------------------------------------------------------------
problem:  variables n = 2, constraints m = 3
          nnz(P) + nnz(A) = 7
settings: linear system solver = qdldl,
          eps_abs = 1.0e-03, eps_rel = 1.0e-03,
          eps_prim_inf = 1.0e-04, eps_dual_inf = 1.0e-04,
          rho = 1.00e-01 ,
          sigma = 1.00e-06, alpha = 1.00, max_iter = 4000
          check_termination: on (interval 25),
          scaling: on, scaled_termination: off
          warm start: on, polish: off, time_limit: off

iter   objective    pri res    dua res    rho        time
   1  -4.9384e-03   1.00e+00   2.00e+02   1.00e-01   1.52e-05s
  50   1.8798e+00   1.23e-03   2.46e-05   1.00e-01   2.71e-05s

status:               solved
number of iterations: 50
optimal objective:    1.8798
run time:             2.96e-05s
optimal rho estimate: 1.38e+00

expectedSolution: 0.3
0.7
solver.getSolution(): 0.298771
0.701228
tolerance: 0.0001
expectedSolution-solver.getSolution(): 0.00122892
 -0.001228

QPUnitTests is a Catch2 v3.4.0 host application.
Run with -? for options

-------------------------------------------------------------------------------
QPProblem
-------------------------------------------------------------------------------
/home/traversaro/robotology-superbuild/src/OsqpEigen/tests/QPTest.cpp:46
...............................................................................

/home/traversaro/robotology-superbuild/src/OsqpEigen/tests/QPTest.cpp:99: FAILED:
  REQUIRE( solver.getSolution().isApprox(expectedSolution, tolerance) )
with expansion:
  false

===============================================================================
test cases:  2 |  1 passed | 1 failed
assertions: 14 | 13 passed | 1 failed

0% tests passed, 1 tests failed out of 1

Total Test time (real) =   0.00 sec

The following tests FAILED:
          2 - QPUnitTests (Failed)
Errors while running CTest

After playing a bit with the tolerances, I noticed that setting the polish option to true ensure that the solution is always provided with high accuracy, so in this PR I enabled the option to avoid the risk of non-deterministic failure of QPUnitTests.

traversaro commented 3 weeks ago

Please ignore the failing tests that are due to https://github.com/robotology/osqp-eigen/issues/161 and will be fixed in https://github.com/robotology/osqp-eigen/pull/171 .

GiulioRomualdi commented 3 weeks ago

Thank you!!! You fixed a really annoying bug in the test!

traversaro commented 3 weeks ago

Please ignore the failing tests that are due to #161 and will be fixed in #171 .

That really scared me when working on https://github.com/ami-iit/qpsolvers-eigen, as for some reason there the issue was much more frequent.