search

ralna / CUTEst

The Constrained and Unconstrained Testing Environment with safe threads (CUTEst) for optimization software
Other
83 stars 18 forks source link

Segmentation Fault when solving problems with most recent version of Ipopt #15

Closed chrundle closed 4 years ago

chrundle commented 4 years ago

Issue Overview: I followed the directions in $CUTEST/src/ipopt/README.ipopt for getting Ipopt running with CUTEst but Ipopt with CUTEst results in a segmentation fault.

I recently installed Ipopt version 3.13.3 and have been unable to solve problems with Ipopt using CUTEst. In particular, I am experiencing a segmentation fault from an invalid memory reference. I did a quick edit of runcutest to run with Valgrind and have included the output below. The following three outputs are included below:

(1) Indication that Ipopt was able to pass the tests upon installation (2) Console output on attempt to solve BIGGSC4 with Ipopt (3) Console output on attempt to solve BIGGSC4 with Ipopt with Valgrind in runcutest

(1) Console output indicating Ipopt 3.13.3 has been successfully installed on machine:

Running unitTests...

Testing AMPL Solver Executable... no AMPL solver executable found, skipping test... Testing C++ Example... Test passed! Testing C Example... Test passed! Testing Fortran Example... Test passed! Skip testing Java Example (Java interface not build) Testing sIpopt Example parametric_cpp... Test passed! Testing sIpopt Example redhess_cpp... Test passed! make[1]: Leaving directory '/home/james/Programs/Ipopt/Ipopt/master/test'

##################################################

Installing Ipopt master

##################################################

Install completed. If executing any of the installed binaries results in an error that shared libraries cannot be found, you may need to

(2) Console output when attempting to solve BIGGSC4 with Ipopt:

:~$ runcutest -p ipopt -D BIGGSC4 sifdecoder -A pc.lnx.gfo -st BIGGSC4

Problem name: BIGGSC4

Double precision version will be formed

The objective function uses 1 nonlinear group

There are 7 linear inequality constraints

There are 4 variables bounded from below and above

File successfully decoded CUTEST: tools (double precision version) compiled successfully CUTEst: ipopt (double precision version) compiled successfully

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:

0 0x7fed07f1b2ed in ???

1 0x7fed07f1a503 in ???

2 0x7fed07b4df1f in ???

3 0x7fed0a5f71e8 in ???

4 0x55eac0e9dd22 in ???

5 0x55eac0e9e5f2 in ???

6 0x7fed07b30b96 in ???

7 0x55eac0e9d269 in ???

8 0xffffffffffffffff in ???

/home/james/Programs/CUTEST-Program/cutest/bin/runcutest: line 238: 32615 Segmentation fault (core dumped) ${EXEC}/run_${PACKAGE}

(3) Console output when solving BIGGSC4 with Ipopt (Valgrind in runcutest):

:~$ runcutest -p ipopt -D BIGGSC4 sifdecoder -A pc.lnx.gfo -st BIGGSC4

Problem name: BIGGSC4

Double precision version will be formed

The objective function uses 1 nonlinear group

There are 7 linear inequality constraints

There are 4 variables bounded from below and above

File successfully decoded CUTEST: tools (double precision version) compiled successfully CUTEst: ipopt (double precision version) compiled successfully ==32182== Memcheck, a memory error detector ==32182== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al. ==32182== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info ==32182== Command: /home/james/runipopt ==32182== ==32182== Use of uninitialised value of size 8 ==32182== at 0x126341E8: ipsolve (in /home/james/Programs/Ipopt/lib/libipopt.so.3.13.3) ==32182== by 0x109D22: MAIN (in /home/james/run_ipopt) ==32182== by 0x10A5F2: main (in /home/james/run_ipopt) ==32182== Uninitialised value was created by a stack allocation ==32182== at 0x10947F: MAIN (in /home/james/run_ipopt) ==32182==

This program contains Ipopt, a library for large-scale nonlinear optimization. Ipopt is released as open source code under the Eclipse Public License (EPL). For more information visit http://projects.coin-or.org/Ipopt

This is Ipopt version 3.13.3, running with linear solver mumps. NOTE: Other linear solvers might be more efficient (see Ipopt documentation).

Number of nonzeros in equality constraint Jacobian...: 0 Number of nonzeros in inequality constraint Jacobian.: 16 Number of nonzeros in Lagrangian Hessian.............: 6

==32182== Syscall param sched_setaffinity(mask) points to unaddressable byte(s) ==32182== at 0x15197839: syscall (syscall.S:38) ==32182== by 0x122FB0A8: kmp_affinity_determine_capable (z_Linux_util.cpp:186) ==32182== by 0x122A5571: __kmp_env_initialize(char const*) (kmp_settings.cpp:5783) ==32182== by 0x1228C717: kmp_do_serial_initialize (kmp_runtime.cpp:7003) ==32182== by 0x1228C717: kmp_do_middle_initialize (kmp_runtime.cpp:7146) ==32182== by 0x1228C717: kmp_middle_initialize (kmp_runtime.cpp:7255) ==32182== by 0x1226CD2D: omp_get_num_procs@@VERSION (kmp_ftn_entry.h:612) ==32182== by 0xBB5E8BD: mkl_serv_get_num_stripes (in /home/james/intel/compilers_and_libraries_2019.5.281/linux/mkl/lib/intel64_lin/libmkl_intel_thread.so) ==32182== by 0xBC244F1: mkl_blas_dgemm (in /home/james/intel/compilers_and_libraries_2019.5.281/linux/mkl/lib/intel64_lin/libmkl_intel_thread.so) ==32182== by 0xAF4DBA6: DGEMM (in /home/james/intel/compilers_and_libraries_2019.5.281/linux/mkl/lib/intel64_lin/libmkl_intel_lp64.so) ==32182== by 0x15B8BD66: dmumps237 (in /home/james/Programs/Ipopt/lib/libcoinmumps.so.2.1.0) ==32182== by 0x15B8EBC5: dmumps140 (in /home/james/Programs/Ipopt/lib/libcoinmumps.so.2.1.0) ==32182== by 0x15B79AC5: dmumps251 (in /home/james/Programs/Ipopt/lib/libcoinmumps.so.2.1.0) ==32182== by 0x15B3DB47: dmumps244 (in /home/james/Programs/Ipopt/lib/libcoinmumps.so.2.1.0) ==32182== Address 0x0 is not stack'd, malloc'd or (recently) free'd ==32182== Total number of variables............................: 4 variables with only lower bounds: 0 variables with lower and upper bounds: 4 variables with only upper bounds: 0 Total number of equality constraints.................: 0 Total number of inequality constraints...............: 7 inequality constraints with only lower bounds: 1 inequality constraints with lower and upper bounds: 6 inequality constraints with only upper bounds: 0

iter objective inf_pr inf_du lg(mu) ||d|| lg(rg) alpha_du alpha_pr ls 0 -1.9999960e-04 4.96e+00 3.67e-01 -1.0 0.00e+00 - 0.00e+00 0.00e+00 0 1 -1.4654948e-03 4.89e+00 1.20e+00 -1.0 1.42e+00 - 7.45e-03 1.58e-02h 1 2 -3.1187792e+00 0.00e+00 7.82e+01 -1.0 1.66e+00 - 2.97e-02 1.00e+00f 1 3 -3.2911450e+00 0.00e+00 2.11e+00 -1.0 2.31e-01 - 6.48e-01 1.00e+00f 1 4 -7.4033627e+00 0.00e+00 8.95e-01 -1.0 2.23e+00 - 2.02e-01 1.00e+00f 1 5 -9.3688176e+00 0.00e+00 7.61e-01 -1.0 7.61e-01 0.0 1.00e+00 1.00e+00f 1 6 -1.2634455e+01 0.00e+00 9.40e-01 -1.7 5.23e+00 -0.5 3.90e-01 2.36e-01f 1 7 -1.5042427e+01 0.00e+00 8.31e-01 -1.7 1.20e+01 -1.0 2.78e-01 5.61e-02f 1 8 -1.5089210e+01 0.00e+00 1.33e+00 -1.7 5.67e-02 - 4.60e-01 1.00e+00f 1 9 -1.7926460e+01 0.00e+00 5.64e-01 -1.7 1.91e+00 -0.5 6.65e-01 1.00e+00f 1 iter objective inf_pr inf_du lg(mu) ||d|| lg(rg) alpha_du alpha_pr ls 10 -1.9765668e+01 0.00e+00 6.81e-01 -1.7 8.43e-01 -0.1 7.90e-01 1.00e+00f 1 11 -2.1870779e+01 0.00e+00 1.22e+00 -1.7 4.93e+00 -0.6 9.76e-01 1.65e-01f 1 12 -2.4457195e+01 0.00e+00 1.69e+00 -1.7 1.02e+01 -1.1 2.55e-01 8.82e-02f 1 13 -2.4431280e+01 0.00e+00 2.03e+00 -1.7 1.82e+00 -0.6 8.59e-01 3.39e-02f 4 14 -2.4440263e+01 0.00e+00 1.15e+00 -1.7 7.14e-02 - 9.94e-01 5.00e-01f 2 15 -2.4432566e+01 0.00e+00 2.00e-07 -1.7 3.13e-02 - 1.00e+00 1.00e+00f 1 16 -2.4496204e+01 0.00e+00 5.06e-03 -3.8 4.80e-02 - 9.25e-01 9.72e-01f 1 17 -2.4498824e+01 0.00e+00 1.50e-09 -3.8 3.56e-02 - 1.00e+00 1.00e+00f 1 18 -2.4499812e+01 0.00e+00 5.53e-04 -5.7 1.90e-02 - 9.71e-01 1.00e+00f 1 19 -2.4499947e+01 0.00e+00 1.84e-11 -5.7 9.42e-03 - 1.00e+00 1.00e+00f 1 iter objective inf_pr inf_du lg(mu) ||d|| lg(rg) alpha_du alpha_pr ls 20 -2.4499982e+01 0.00e+00 1.84e-11 -5.7 4.79e-03 - 1.00e+00 1.00e+00f 1 21 -2.4499991e+01 0.00e+00 1.84e-11 -5.7 2.31e-03 - 1.00e+00 1.00e+00f 1 22 -2.4499999e+01 0.00e+00 1.77e-06 -8.6 1.34e-03 - 9.99e-01 1.00e+00f 1 23 -2.4500000e+01 0.00e+00 2.51e-14 -8.6 6.68e-04 - 1.00e+00 1.00e+00f 1 24 -2.4500000e+01 0.00e+00 2.51e-14 -8.6 3.33e-04 - 1.00e+00 1.00e+00f 1 25 -2.4500000e+01 0.00e+00 2.51e-14 -8.6 1.65e-04 - 1.00e+00 1.00e+00h 1 26 -2.4500000e+01 0.00e+00 2.51e-14 -8.6 7.89e-05 - 1.00e+00 1.00e+00h 1

Number of Iterations....: 26

                               (scaled)                 (unscaled)

Objective...............: -2.4500000338117488e+01 -2.4500000338117488e+01 Dual infeasibility......: 2.5059035596809423e-14 2.5059035596809423e-14 Constraint violation....: 0.0000000000000000e+00 0.0000000000000000e+00 Complementarity.........: 8.7300075264282586e-09 8.7300075264282586e-09 Overall NLP error.......: 8.7300075264282586e-09 8.7300075264282586e-09

Number of objective function evaluations = 32 Number of objective gradient evaluations = 27 Number of equality constraint evaluations = 0 Number of inequality constraint evaluations = 32 Number of equality constraint Jacobian evaluations = 0 Number of inequality constraint Jacobian evaluations = 27 Number of Lagrangian Hessian evaluations = 26 Total CPU secs in IPOPT (w/o function evaluations) = 2.623 Total CPU secs in NLP function evaluations = 0.023

EXIT: Optimal Solution Found. ==32182== Use of uninitialised value of size 8 ==32182== at 0x126341BC: ipfree_ (in /home/james/Programs/Ipopt/lib/libipopt.so.3.13.3) ==32182== by 0x109D35: MAIN (in /home/james/run_ipopt) ==32182== by 0x10A5F2: main (in /home/james/run_ipopt) ==32182== Uninitialised value was created by a stack allocation ==32182== at 0x10947F: MAIN (in /home/james/run_ipopt) ==32182== ==32182== Conditional jump or move depends on uninitialised value(s) ==32182== at 0x4C30CF1: free (in /usr/lib/valgrind/vgpreloadmemcheck-amd64-linux.so) ==32182== by 0x126341CC: ipfree (in /home/james/Programs/Ipopt/lib/libipopt.so.3.13.3) ==32182== by 0x109D35: MAIN (in /home/james/run_ipopt) ==32182== by 0x10A5F2: main (in /home/james/run_ipopt) ==32182== Uninitialised value was created by a stack allocation ==32182== at 0x10947F: MAIN (in /home/james/run_ipopt) ==32182==

**** CUTEst statistics ****

Package used : IPOPT Problem : BIGGSC4 # variables = 4 # constraints = 7 # objective functions = 0.6000000E+02 # objective gradients = 0.2800000E+02 # objective Hessians = 0.2700000E+02 # Hessian-vector prdct = 0.0000000E+00 # constraints functions = 0.6100000E+02 # constraints gradients = 0.2900000E+02 # constraints Hessians = 0.2700000E+02 Exit code = 0 Final f = -0.2450000E+02 Set up time = 0.08 seconds Solve time = 3.56 seconds

==32182== ==32182== HEAP SUMMARY: ==32182== in use at exit: 2,639 bytes in 21 blocks ==32182== total heap usage: 35,865 allocs, 35,844 frees, 302,556,517 bytes allocated ==32182== ==32182== 28 bytes in 1 blocks are definitely lost in loss record 2 of 21 ==32182== at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==32182== by 0x1098E7: MAIN__ (in /home/james/run_ipopt) ==32182== by 0x10A5F2: main (in /home/james/run_ipopt) ==32182== ==32182== 70 bytes in 1 blocks are definitely lost in loss record 14 of 21 ==32182== at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==32182== by 0x1099C3: MAIN__ (in /home/james/run_ipopt) ==32182== by 0x10A5F2: main (in /home/james/run_ipopt) ==32182== ==32182== LEAK SUMMARY: ==32182== definitely lost: 98 bytes in 2 blocks ==32182== indirectly lost: 0 bytes in 0 blocks ==32182== possibly lost: 0 bytes in 0 blocks ==32182== still reachable: 2,541 bytes in 19 blocks ==32182== suppressed: 0 bytes in 0 blocks ==32182== Reachable blocks (those to which a pointer was found) are not shown. ==32182== To see them, rerun with: --leak-check=full --show-leak-kinds=all ==32182== ==32182== For counts of detected and suppressed errors, rerun with: -v ==32182== ERROR SUMMARY: 6 errors from 6 contexts (suppressed: 0 from 0)

nimgould commented 4 years ago

Dear James

I downloaded and installed the latest Ipopt this morning ... but I don't see anything out of the ordinary -

% runcutest --package ipopt --decode BIGGSC4 sifdecoder -A pc64.lnx.gfo -st BIGGSC4

Problem name: BIGGSC4

Double precision version will be formed

The objective function uses 1 nonlinear group

There are 7 linear inequality constraints

There are 4 variables bounded from below and above

File successfully decoded CUTEST: tools (double precision version) compiled successfully CUTEst: ipopt (double precision version) compiled successfully /usr/bin/x86_64-linux-gnu-ld: warning: libgfortran.so.4, needed by //usr/local/lib/libcoinmumps.so.2, may conflict with libgfortran.so.3

This program contains Ipopt, a library for large-scale nonlinear optimization. Ipopt is released as open source code under the Eclipse Public License (EPL). For more information visit http://projects.coin-or.org/Ipopt

This is Ipopt version 3.13.3, running with linear solver mumps. NOTE: Other linear solvers might be more efficient (see Ipopt documentation).

Number of nonzeros in equality constraint Jacobian...: 0 Number of nonzeros in inequality constraint Jacobian.: 16 Number of nonzeros in Lagrangian Hessian.............: 6

Total number of variables............................: 4 variables with only lower bounds: 0 variables with lower and upper bounds: 4 variables with only upper bounds: 0 Total number of equality constraints.................: 0 Total number of inequality constraints...............: 7 inequality constraints with only lower bounds: 1 inequality constraints with lower and upper bounds: 6 inequality constraints with only upper bounds: 0

iter objective inf_pr inf_du lg(mu) ||d|| lg(rg) alpha_du alpha_pr ls 0 -1.9999960e-04 4.96e+00 3.67e-01 -1.0 0.00e+00 - 0.00e+00 0.00e+00 0 1 -1.4654948e-03 4.89e+00 1.20e+00 -1.0 1.42e+00 - 7.45e-03 1.58e-02h 1 2 -3.1187792e+00 0.00e+00 7.82e+01 -1.0 1.66e+00 - 2.97e-02 1.00e+00f 1 3 -3.2911450e+00 0.00e+00 2.11e+00 -1.0 2.31e-01 - 6.48e-01 1.00e+00f 1 4 -7.4033627e+00 0.00e+00 8.95e-01 -1.0 2.23e+00 - 2.02e-01 1.00e+00f 1 5 -9.3688176e+00 0.00e+00 7.61e-01 -1.0 7.61e-01 0.0 1.00e+00 1.00e+00f 1 6 -1.2634455e+01 0.00e+00 9.40e-01 -1.7 5.23e+00 -0.5 3.90e-01 2.36e-01f 1 7 -1.5042427e+01 0.00e+00 8.31e-01 -1.7 1.20e+01 -1.0 2.78e-01 5.61e-02f 1 8 -1.5089210e+01 0.00e+00 1.33e+00 -1.7 5.67e-02 - 4.60e-01 1.00e+00f 1 9 -1.7926460e+01 0.00e+00 5.64e-01 -1.7 1.91e+00 -0.5 6.65e-01 1.00e+00f 1 iter objective inf_pr inf_du lg(mu) ||d|| lg(rg) alpha_du alpha_pr ls 10 -1.9765668e+01 0.00e+00 6.81e-01 -1.7 8.43e-01 -0.1 7.90e-01 1.00e+00f 1 11 -2.1870779e+01 0.00e+00 1.22e+00 -1.7 4.93e+00 -0.6 9.76e-01 1.65e-01f 1 12 -2.4457195e+01 0.00e+00 1.69e+00 -1.7 1.02e+01 -1.1 2.55e-01 8.82e-02f 1 13 -2.4431280e+01 0.00e+00 2.03e+00 -1.7 1.82e+00 -0.6 8.59e-01 3.39e-02f 4 14 -2.4440263e+01 0.00e+00 1.15e+00 -1.7 7.14e-02 - 9.94e-01 5.00e-01f 2 15 -2.4432566e+01 0.00e+00 2.00e-07 -1.7 3.13e-02 - 1.00e+00 1.00e+00f 1 16 -2.4496204e+01 0.00e+00 5.06e-03 -3.8 4.80e-02 - 9.25e-01 9.72e-01f 1 17 -2.4498824e+01 0.00e+00 1.50e-09 -3.8 3.56e-02 - 1.00e+00 1.00e+00f 1 18 -2.4499812e+01 0.00e+00 5.53e-04 -5.7 1.90e-02 - 9.71e-01 1.00e+00f 1 19 -2.4499947e+01 0.00e+00 1.84e-11 -5.7 9.42e-03 - 1.00e+00 1.00e+00f 1 iter objective inf_pr inf_du lg(mu) ||d|| lg(rg) alpha_du alpha_pr ls 20 -2.4499982e+01 0.00e+00 1.84e-11 -5.7 4.79e-03 - 1.00e+00 1.00e+00f 1 21 -2.4499991e+01 0.00e+00 1.84e-11 -5.7 2.31e-03 - 1.00e+00 1.00e+00f 1 22 -2.4499999e+01 0.00e+00 1.77e-06 -8.6 1.34e-03 - 9.99e-01 1.00e+00f 1 23 -2.4500000e+01 0.00e+00 2.51e-14 -8.6 6.68e-04 - 1.00e+00 1.00e+00f 1 24 -2.4500000e+01 0.00e+00 2.51e-14 -8.6 3.33e-04 - 1.00e+00 1.00e+00f 1 25 -2.4500000e+01 0.00e+00 2.51e-14 -8.6 1.65e-04 - 1.00e+00 1.00e+00h 1 26 -2.4500000e+01 0.00e+00 2.51e-14 -8.6 7.89e-05 - 1.00e+00 1.00e+00h 1

Number of Iterations....: 26

                               (scaled)                 (unscaled)

Objective...............: -2.4500000338117491e+01 -2.4500000338117491e+01 Dual infeasibility......: 2.5059035596809423e-14 2.5059035596809423e-14 Constraint violation....: 0.0000000000000000e+00 0.0000000000000000e+00 Complementarity.........: 8.7300075264463060e-09 8.7300075264463060e-09 Overall NLP error.......: 8.7300075264463060e-09 8.7300075264463060e-09

Number of objective function evaluations = 36 Number of objective gradient evaluations = 27 Number of equality constraint evaluations = 0 Number of inequality constraint evaluations = 36 Number of equality constraint Jacobian evaluations = 0 Number of inequality constraint Jacobian evaluations = 27 Number of Lagrangian Hessian evaluations = 26 Total CPU secs in IPOPT (w/o function evaluations) = 0.014 Total CPU secs in NLP function evaluations = 0.001

EXIT: Optimal Solution Found.

**** CUTEst statistics ****

Package used : IPOPT Problem : BIGGSC4

variables = 4

constraints = 7

objective functions = 0.6400000E+02

objective gradients = 0.2800000E+02

objective Hessians = 0.2700000E+02

Hessian-vector prdct = 0.0000000E+00

constraints functions = 0.6500000E+02

constraints gradients = 0.2900000E+02

constraints Hessians = 0.2700000E+02

Exit code = 0 Final f = -0.2450000E+02 Set up time = 0.00 seconds Solve time = 0.03 seconds

Of course, this doesn't say that there isn't an error, but it will be hard to track.

The uninitialized variables appear to happen in the libipopt binaries, but this may be that cutest is passing incorrect data. I'll have a look using valgrind when I get a chance ... I am not a C++ programmer so it might be worth checking with the ipopt mailing list to see if anyone there has seen anything similar. In particular, ask about the "Use of uninitialised value of size 8".

Nick

chrundle commented 4 years ago

Nick,

Thank you for the quick response. As you mentioned, I thought it might be possible that cutest is passing incorrect data which is why I thought to ask about the issue here. I will check in with the ipopt mailing list and let you know what I hear. Thanks again.

svigerske commented 4 years ago

One thing one needs to get right is that Ipopt and CUTEst need to agree on the size of a void*. Ipopt's build doesn't communicate this, so CUTEst is mainly guessing. In ipopt_main.f, this are these lines:

CNOT64 INTEGER :: iproblem, IPCREATE
CIS64  INTEGER*8 :: iproblem, IPCREATE

That is one possible source for the valgrind warnings for ipsolve_ and ipfree_. If you build Ipopt with --enable-debug, valgrind may tell more.

I don't really know about that NULL pointer access within OpenMP within MKL within Mumps.

chrundle commented 4 years ago

@svigerske Thank you for your comment. I was able to resolve the segmentation fault by declaring iproblem and IPCREATE as variables of type INTEGER*8. In particular, I edited the lines

CNOT64 INTEGER :: iproblem, IPCREATE
CIS64  INTEGER*8 :: iproblem, IPCREATE

in ipopt_main.f down to 

    INTEGER*8 :: iproblem, IPCREATE

and I no longer encounter a segmentation fault when solving CUTEst problems using Ipopt. I am not sure if this was the intended approach but wanted to share that it resolved the issue for me.

Thanks to both @nimgould and @svigerske for your time and input.

nimgould commented 4 years ago

Hi James

Super, I'm glad that worked for you. I only just noticed that you compiled CUTEst for 32 bit machines ... there is a 64 bit version offered as one of the install options. The 64-bit version would have exposed the CIS64 comment, and that would have given you what you needed. The results I quoted were for the pc64.lnx.gfo version, I should have noticed.

I'll close this now.

Nick