Issues with 3.5 pre-release Ipopt binaries

Robbybp commented 5 months ago

Testing the Ipopt binaries for our next planned release, I have noticed some issues:

Ubuntu2004/x86: HSL solvers do not work. Ipopt appears to be looking for libhsl.so. I get the following error:
```
Ipopt 3.13.2: print_user_options=yes
linear_solver=ma57
```

Exception of type: OPTION_INVALID in file "IpAlgBuilder.cpp" at line 295: Exception message: Selected linear solver MA57 not available. Tried to obtain MA57 from shared library "libhsl.so", but the following error occured: libhsl.so: cannot open shared object file: No such file or directory

EXIT: Invalid option encountered.

- MacOS/ARM (only built with Mumps so far): I get a segfault when trying to solve with Mumps. When I compile locally and link this IDAES-Ipopt to my system-installed HSL, the HSL solvers work just fine. The segfault appears to be coming from the Mumps `dmumps_c` symbolic factorization function. The output I see is:
```console
<copyright header>

This is Ipopt version 3.13.2, running with linear solver mumps.
NOTE: Other linear solvers might be more efficient (see Ipopt documentation).

Number of nonzeros in equality constraint Jacobian...:        2
Number of nonzeros in inequality constraint Jacobian.:        0
Number of nonzeros in Lagrangian Hessian.............:        3

ipopt(1751,0x1f7092500) malloc: *** error for object 0x16fdfc240: pointer being freed was not allocated
ipopt(1751,0x1f7092500) malloc: *** set a breakpoint in malloc_error_break to debug

These are the only binaries I have tested so far.

Robbybp commented 5 months ago

When I use the --without-metis flag when building Mumps, the Mac segfault goes away. This could be a Metis/Mumps version incompatibility?

Robbybp commented 5 months ago

The associated Ipopt library seems to work with CyIpopt. While testing, however, I noticed an error message "Problem does not have any continuous variables" from libpynumero_ASL, if I (accidentally) used the version of this library that we distribute. I'm not sure what could be going on here.

To fix the Mumps segfault, we need to either (a) patch the compile-solvers script to use --without-metis for Mumps, or (b) figure out the version incompatibility and install a metis version that works. It looks like ThirdParty/Metis, which we use in compile_solvers.sh, downloads Metis 4.0.3. I wouldn't be surprised if the lastest version of Mumps (5.6.0) requires Metis 5.X. For HSL, we use an old, no-variadic version of Metis, rather than ThirdParty/Metis, which raises the question of whether we even need ThirdParty/Metis at all.

Robbybp commented 5 months ago

While testing, however, I noticed an error message "Problem does not have any continuous variables" from libpynumero_ASL, if I (accidentally) used the version of this library that we distribute. I'm not sure what could be going on here.

This error doesn't exist when I build libpynumero_ASL with v3.4.0 of this repository. I'm not sure what could have changed in the build or linking of this library.

My current plan is to omit this library from the next Mumps-only beta release that I push, and we can try to debug later.

Robbybp commented 5 months ago

While testing, however, I noticed an error message "Problem does not have any continuous variables" from libpynumero_ASL, if I (accidentally) used the version of this library that we distribute. I'm not sure what could be going on here.

This error doesn't exist when I build libpynumero_ASL with v3.4.0 of this repository. I'm not sure what could have changed in the build or linking of this library.

This was on M1 Mac, but I couldn't reproduce on another ARM mac. I will wait to debug until we have the ARM machine that we will actually use to compile these binaries with HSL for the release.

Robbybp commented 5 months ago

Ubuntu2004/x86: HSL solvers do not work. Ipopt appears to be looking for libhsl.so.

Debugging, I notice the following:

ldd ipopt gives me the following:

linux-vdso.so.1 (0x00007ffc93731000)
libgfortran.so.5 => /usr/lib/x86_64-linux-gnu/libgfortran.so.5 (0x00007f0c4d44a000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f0c4d427000)
liblapack.so.3 => /usr/lib/x86_64-linux-gnu/liblapack.so.3 (0x00007f0c4b413000)
libblas.so.3 => /usr/lib/x86_64-linux-gnu/libblas.so.3 (0x00007f0c49a14000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f0c49a0e000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f0c4982c000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f0c496db000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f0c496c0000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f0c494ce000)
/lib64/ld-linux-x86-64.so.2 (0x00007f0c4db57000)
libquadmath.so.0 => /usr/lib/x86_64-linux-gnu/libquadmath.so.0 (0x00007f0c49484000)

We are not attempting to dynamically link HSL (or Mumps) at build time. The error appears to be that Ipopt is not compiled with HSL, so when we request the solver, it looks for libhsl.so at runtime.

This Ipopt has Mumps as the default linear solver, which works.
The ipopt.pc patch is rejected. The offending line is
```
Requires.private: coinmumps lapack blas
```
This should be fixed by #267, which switches the patch command to a sed command. This is not directly related to the HSL issue, but the lack of coinhsl in this line also indicates that Ipopt was not built with HSL.

Robbybp commented 5 months ago

With https://github.com/IDAES/idaes-ext/pull/269, the segfault using Mumps with Mac-ARM-Ipopt appears to be gone.

ksbeattie commented 4 months ago

@Robbybp any progress on this? In time for the Feb (next 2 weeks) release?

Robbybp commented 4 months ago

This is waiting on @andrewlee94 to compile and test the ARM binaries, and to share some files with @MarcusHolly, who will compile the x86 binaries. Assuming no new technical issues arise, there is no reason this can't be ready in two weeks, it's just a matter of coordinating things.

ksbeattie commented 3 months ago

First step here: #270

ksbeattie commented 3 months ago

@Robbybp any updates?

ksbeattie commented 2 months ago

@Robbybp news?

Robbybp commented 2 months ago

@andrewlee94 and I have built and tested the Mac binaries. They are uploaded as a pre-release to this repo. I have not seen the ASL issue re-occur, so I assume this was something particular to the system I first built these binaries on.

Even when using sed instead of patch, we get errors trying to build the Linux x86 binaries on @MarcusHolly's machine (via Docker). As far as I remember, the error seemed fairly straightforward ("file does not exist" from bash, or something, indicating we just have name/location mismatch somewhere). This is waiting for me to have time to review @MarcusHolly's build logs and work with him to figure out exactly what is going on.

ksbeattie commented 2 months ago

@Robbybp & @MarcusHolly can we get a new update on this?

MarcusHolly commented 2 months ago

Same status as Robby's previous update as far as I know.

ksbeattie commented 1 month ago

@Robbybp I'm moving this off of the May release, to the Aug release. Let me know if you'll be able to make some progress on it.

Robbybp commented 1 month ago

@ksbeattie I likely won't have time until the second week of June.

ksbeattie commented 3 weeks ago

Sounds like we are waiting to see if Sandia can help with this.

IDAES / idaes-ext

Issues with 3.5 pre-release Ipopt binaries #268