Failed to invert Hessian in example script

[Mathijssch]

First of all, thank you for your efforts in developing this toolbox and making it publicly available! Unfortunately, however, I am having some trouble when running example mpcmhe/mpcmhe_unicycle.m (I left everything on the default settings). Other examples do seem to work. I've tried examples/TCminmax.m (which also build Nash equilibrium solvers if I understand correctly) and examples/mpcmhe/mpc_unicycle_pursuit.m.

In particular, for mpcmhe_unicycle.m I get the error message

Error using mpcmhe_unicycle (line 226)
failed to invert hessian, regenerate code to compile with saved values for hessian

Platform information

Running MATLAB R2019b on Ubuntu 20.04. gcc is used as the C compiler for MEX:

>> mex -setup C
MEX configured to use 'gcc' for C language compilation.

Extra diagnostics

Detailed output

``` MATLAB >> mpcmhe_unicycle deleting object, clear mex, unloading library tmp_mm_uni_TS_20211130_115538_771330_classname; Recompilation needed cmex2equilibriumLatentCS: ... declaring sets for parameters and primal variables... Starting ipmPDeqlat_CS symbolic computations (largeNewtonMatrix)... # primal vars = 157, # equal constr = 100, # inequal constr = 44... getfg()...(0.14 sec) 1st derivates...(0.38 sec) 2nd derivatives...(0.70 sec) adding LFF...(0.01 sec) WW & b...(0.04 sec) adding WW...(0.12 sec) combined direction...(0.03 sec) new vectores...(0.07 sec) updatePrimalDual()...(0.02 sec) done ipmPDeqlat_CS symbolic computations (1.607 sec) class classFolder @tmp_mm_uni_TS_20211130_115608_715893_classname does not exist, creating it... done creating csparse object (2.207 sec) creating C code... computeScalarInstructions... computeScalarInstructions (454 expressions)... computeScalarInstructions: currently at expr 50/454... computeScalarInstructions: currently at expr 100/454... computeScalarInstructions: currently at expr 150/454... computeScalarInstructions: currently at expr 200/454... computeScalarInstructions: currently at expr 250/454... computeScalarInstructions: currently at expr 300/454... computeScalarInstructions: currently at expr 350/454... sparsifyLU(373): A size=[401,401] , nnz=1348, distinct nnz=286 loadCsparse: cannot open subscripts file "./@tmp_mm_uni_TS_20211130_115608_715893_classname/tmp_mm_uni_TS_20211130_115608_715893_classname_WW.subscripts" (fd=-1) using random values, 1197/1348 nonzero entries of A and A' do not match 1/ 401: pivot( 1, 1)= 1.40736 ( 0 rows) 1.6sec 101/ 401: pivot( 101, 101)= 1.39714 ( 2 rows) 1.6sec 201/ 401: pivot( 201, 201)= 1.4529 ( 2 rows) 1.7sec 301/ 401: pivot( 301, 301)= -11.6004 ( 5 rows) 1.8sec 401/ 401: pivot( 401, 401)= -0.312669 ( 0 rows) 2.0sec no changes in pivoting from matlab's lu, sparsifyLU(373): LU size=[401,401] , nnz=2569, # new instr=4405 (3323..7727) (3.07 sec) sparsify_mldivide_l1(383): LU size=[401,401] , nnz=2569, B size=[401] , nnz=401 sparsify_mldivide_l1(383): (I+L)\B size=[401] , nnz=401, # new instr= 200 (8173..8372) (0.04 sec) computeScalarInstructions: currently at expr 400/454... computeScalarInstructions: currently at expr 450/454... done computeScalarInstructions (4.999 sec) dependencyGroups... dependency matrix with 9879 instructions(gets,saves,copies,sets,copies 0.18 sec) determining groups (0.22 sec), ordering groups (0.23 sec), local variables (0.37 sec), memory locations ( unused-instructions reuse-within-group 0.44 sec) done dependencyGroups (0.460 sec) compile2C: 456 added vectorized Operations, 454 unique vectorized Operations compile2C: 11200 added instructions, 9879 unique instructions compile2C: sizeScratchbook=7077 double, nGroups=112, nSets=19, nGets=7, nCopies=6 write C code... done (0.773 sec) done creating C code (6.243 sec) createGateway: private class folder '././@tmp_mm_uni_TS_20211130_115608_715893_classname/private' does not exist, creating it ATTENTION: must add '.' or '/home/mathijs/Work/Research/code/hespanha/tenscalc/examples' to LD_LIBRARY_PATH (LD_LIBRARY_PATH=.;export LD_LIBRARY_PATH) ATTENTION: must add '.' or '/home/mathijs/Work/Research/code/hespanha/tenscalc/examples' to LD_LIBRARY_PATH (LD_LIBRARY_PATH=.;export LD_LIBRARY_PATH) optimization = -O0 -msse -msse2 -msse3 -msse4 -msse4.1... ././@tmp_mm_uni_TS_20211130_115608_715893_classname/tmp_mm_uni_TS_20211130_115608_715893_classname.c = 585.030kB, /home/mathijs/Work/Research/code/hespanha/tenscalc/examples/@tmp_mm_uni_TS_20211130_115608_715893_classname/private/tmp_mm_uni_TS_20211130_115608_715893_classname.so = 592.781kB (0.608 sec) ATTENTION: must add '.' or '/home/mathijs/Work/Research/code/hespanha/tenscalc/examples' to LD_LIBRARY_PATH (LD_LIBRARY_PATH=.;export LD_LIBRARY_PATH) createGateway: dynamicLibrady file 'tmp_mm_uni_TS_20211130_115608_715893_classname_load' NOT IN PATH Warning: Function dissect has the same name as a MATLAB builtin. We suggest you rename the function to avoid a potential name conflict. > In createGateway (line 969) In cmex2equilibriumLatentCS (line 596) In mpcmhe_unicycle (line 99) done creating & compiling gateways & library (1.86 sec) done cmex2equilibriumLatentCS (10.307 sec) cmex2equilibriumLatentCS: saving classname -> ./tmp_mm_uni+TS=20211130-115608-715893+classname... done (0.019 sec) ipmPDeq_CSsolver.c (coupledAlphas=1,skipAffine=1,delta=3,allowSave=0): 157 primal variables (12+45+100), 100 eq. constr., 44 ineq. constr. -> failed to invert hessian 1:status=0x4 ) cost= -7.58368e+03, 7.58368e+03, |eq|= 0.00e+00, ineq= 1.00e+00, dual= 1.00e+00, gap= 4.40e+01, last alpha= 1.01e+00, |grad|= 4.91e+03 (108.8us,108.80us/iter) t=9, J=-7583.68 computed in 1 iterations & 0.108802 ms Error using mpcmhe_unicycle (line 226) failed to invert hessian, regenerate code to compile with saved values for hessian ```

Do you have any insights in what may be the issue?

[hespanha]

Sorry for the slow response.

That particular script needs to be run twice: the first time it generates the code with random pivoting used to invert the hessian. For many problems that suffices, but not for this one and the code fails with the error you saw

 Error using mpcmhe_unicycle (line 226)
  failed to invert hessian, regenerate code to compile with saved values for hessian

The second time you ran the script, it should find a file with values for the hessian matrix that will greatly improve pivoting.

However, I had mistakenly introduced an error in the code that prevented tenscalc from finding that file. Because of that, no matter how many times you ran the code, it would always do random pivoting and fail exactly in the same way.

I fixed that so the first time you ran the script, you should see a message close to the start saying

  creating C code...   computeScalarInstructions...
   computeScalarInstructions (454 expressions)...
    computeScalarInstructions: currently at expr 50/454...
    computeScalarInstructions: currently at expr 100/454...
    computeScalarInstructions: currently at expr 150/454...
    computeScalarInstructions: currently at expr 200/454...
    computeScalarInstructions: currently at expr 250/454...
    computeScalarInstructions: currently at expr 300/454...
    computeScalarInstructions: currently at expr 350/454...
   sparsifyLU(373): A  size=[401,401] , nnz=1348, distinct nnz=286
     loadCsparse: cannot open subscripts file "./@tmp_mpcmhe_uni/tmp_mpcmhe_uni_WW.subscripts" (fd=-1)
     using random values, 1197/1348 nonzero entries of A and A' do not match
   ...

The code will like fail this time. However, the second and subsequent times you ran the script, the message changes to

   creating C code...   computeScalarInstructions...
   computeScalarInstructions (454 expressions)...
    computeScalarInstructions: currently at expr 50/454...
    computeScalarInstructions: currently at expr 100/454...
    computeScalarInstructions: currently at expr 150/454...
    computeScalarInstructions: currently at expr 200/454...
    computeScalarInstructions: currently at expr 250/454...
    computeScalarInstructions: currently at expr 300/454...
    computeScalarInstructions: currently at expr 350/454...
  sparsifyLU(373): A  size=[401,401] , nnz=1348, distinct nnz=286
     using values from "./@tmp_mpcmhe_uni/tmp_mpcmhe_uni_WW.values
   ...

This time the script should not fail. Please let me know how it goes. Joao

[hespanha]

Hi Mathijs, Sorry for the delay. I've just pushed a fix. Please see my detailed response in GitHub. Joao

On Tue, Nov 30, 2021 at 3:20 AM Mathijs Schuurmans @.***> wrote:

First of all, thank you for your efforts in developing this toolbox and making it publicly available! Unfortunately, however, I am having some trouble when running example mpcmhe/mpcmhe_unicycle.m (I left everything on the default settings). Other examples do seem to work. I've tried examples/TCminmax.m (which also build Nash equilibrium solvers if I understand correctly) and examples/mpcmhe/mpc_unicycle_pursuit.m.

In particular, for mpcmhe_unicycle.m I get the error message

Error using mpcmhe_unicycle (line 226)failed to invert hessian, regenerate code to compile with saved values for hessian

Platform information

Running MATLAB R2019b on Ubuntu 20.04. gcc is used as the C compiler for MEX:

mex -setup CMEX configured to use 'gcc' for C language compilation.

Extra diagnostics Detailed output

mpcmhe_unicycledeleting object, clear mex, unloading library tmp_mm_uni_TS_20211130_115538_771330_classname;Recompilation neededcmex2equilibriumLatentCS: ... declaring sets for parameters and primal variables... Starting ipmPDeqlat_CS symbolic computations (largeNewtonMatrix)...

primal vars = 157, # equal constr = 100, # inequal constr = 44...

getfg()...(0.14 sec) 1st derivates...(0.38 sec) 2nd derivatives...(0.70 sec) adding LFF...(0.01 sec) WW & b...(0.04 sec) adding WW...(0.12 sec) combined direction...(0.03 sec) new vectores...(0.07 sec) updatePrimalDual()...(0.02 sec) done ipmPDeqlat_CS symbolic computations (1.607 sec)class classFolder @tmp_mm_uni_TS_20211130_115608_715893_classname does not exist, creating it... done creating csparse object (2.207 sec) creating C code... computeScalarInstructions... computeScalarInstructions (454 expressions)... computeScalarInstructions: currently at expr 50/454... computeScalarInstructions: currently at expr 100/454... computeScalarInstructions: currently at expr 150/454... computeScalarInstructions: currently at expr 200/454... computeScalarInstructions: currently at expr 250/454... computeScalarInstructions: currently at expr 300/454... computeScalarInstructions: currently at expr 350/454... sparsifyLU(373): A size=[401,401] , nnz=1348, distinct nnz=286 loadCsparse: cannot open subscripts file @._mm_uni_TS_20211130_115608_715893_classname/tmp_mm_uni_TS_20211130_115608_715893_classname_WW.subscripts" (fd=-1) using random values, 1197/1348 nonzero entries of A and A' do not match 1/ 401: pivot( 1, 1)= 1.40736 ( 0 rows) 1.6sec 101/ 401: pivot( 101, 101)= 1.39714 ( 2 rows) 1.6sec 201/ 401: pivot( 201, 201)= 1.4529 ( 2 rows) 1.7sec 301/ 401: pivot( 301, 301)= -11.6004 ( 5 rows) 1.8sec 401/ 401: pivot( 401, 401)= -0.312669 ( 0 rows) 2.0secno changes in pivoting from matlab's lu, sparsifyLU(373): LU size=[401,401] , nnz=2569, # new instr=4405 (3323..7727) (3.07 sec) sparsify_mldivide_l1(383): LU size=[401,401] , nnz=2569, B size=[401] , nnz=401 sparsify_mldivide_l1(383): (I+L)\B size=[401] , nnz=401, # new instr= 200 (8173..8372) (0.04 sec) computeScalarInstructions: currently at expr 400/454... computeScalarInstructions: currently at expr 450/454... done computeScalarInstructions (4.999 sec) dependencyGroups... dependency matrix with 9879 instructions(gets,saves,copies,sets,copies 0.18 sec) determining groups (0.22 sec), ordering groups (0.23 sec), local variables (0.37 sec), memory locations ( unused-instructions reuse-within-group 0.44 sec) done dependencyGroups (0.460 sec) compile2C: 456 added vectorized Operations, 454 unique vectorized Operations compile2C: 11200 added instructions, 9879 unique instructions compile2C: sizeScratchbook=7077 double, nGroups=112, nSets=19, nGets=7, nCopies=6 write C code... done (0.773 sec) done creating C code (6.243 sec)createGateway: private class folder @._mm_uni_TS_20211130_115608_715893_classname/private' does not exist, creating it ATTENTION: must add '.' or '/home/mathijs/Work/Research/code/hespanha/tenscalc/examples' to LD_LIBRARY_PATH (LD_LIBRARY_PATH=.;export LD_LIBRARY_PATH) ATTENTION: must add '.' or '/home/mathijs/Work/Research/code/hespanha/tenscalc/examples' to LD_LIBRARY_PATH (LD_LIBRARY_PATH=.;export LD_LIBRARY_PATH) optimization = -O0 -msse -msse2 -msse3 -msse4 -msse4.1... .@._mm_uni_TS_20211130_115608_715893_classname/tmp_mm_uni_TS_20211130_115608_715893_classname.c = 585.030kB, @._mm_uni_TS_20211130_115608_715893_classname/private/tmp_mm_uni_TS_20211130_115608_715893_classname.so = 592.781kB (0.608 sec) ATTENTION: must add '.' or '/home/mathijs/Work/Research/code/hespanha/tenscalc/examples' to LD_LIBRARY_PATH (LD_LIBRARY_PATH=.;export LD_LIBRARY_PATH)createGateway: dynamicLibrady file 'tmp_mm_uni_TS_20211130_115608_715893_classname_load' NOT IN PATHWarning: Function dissect has the same name as a MATLAB builtin. We suggest you rename the function to avoid a potential name conflict. In createGateway (line 969) In cmex2equilibriumLatentCS (line 596) In mpcmhe_unicycle (line 99) done creating & compiling gateways & library (1.86 sec)done cmex2equilibriumLatentCS (10.307 sec)cmex2equilibriumLatentCS: saving classname -> ./tmp_mm_uni+TS=20211130-115608-715893+classname... done (0.019 sec)ipmPDeq_CSsolver.c (coupledAlphas=1,skipAffine=1,delta=3,allowSave=0): 157 primal variables (12+45+100), 100 eq. constr., 44 ineq. constr. -> failed to invert hessian 1:status=0x4 ) cost= -7.58368e+03, 7.58368e+03, |eq|= 0.00e+00, ineq= 1.00e+00, dual= 1.00e+00, gap= 4.40e+01, last alpha= 1.01e+00, |grad|= 4.91e+03 (108.8us,108.80us/iter) t=9, J=-7583.68 computed in 1 iterations & 0.108802 msError using mpcmhe_unicycle (line 226)failed to invert hessian, regenerate code to compile with saved values for hessian

Do you have any insights in what may be the issue?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/hespanha/tenscalc/issues/3, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC3N7NOJVYLBAX7W5BJDZC3UOSXPPANCNFSM5JBRELNQ .

--

Prof. Joao P. Hespanha Electrical & Computer Engineering Dept. University of California, Santa Barbara, CA 93106-9560 url: http://www.ece.ucsb.edu/~hespanha

[Mathijssch]

Yes, that fixed it. Thanks a lot!