'pyomo help -s' inconsistent with SolverFactory

haraldg commented 11 months ago

Summary

I have the cyipopt module installed and pyomo lists it as viable solver. But when trying to use it with SolverFactory instead of using cyipopt, pyomo falls back to ASL which crashes as one would expect.

Steps to reproduce the issue

$ pyomo help -s
...
    cplex                         The CPLEX LP/MIP solver
    cplex_direct                  Direct python interface to CPLEX
    cplex_persistent              Persistent python interface to CPLEX
   +cyipopt                       Cyipopt: direct python bindings to
                                  the Ipopt NLP solver
    gams                          The GAMS modeling language
   +gdpopt              22.5.13   The GDPopt decomposition-based
...

# example.py
import pyomo.environ as pyo
opt = pyo.SolverFactory('cypiopt')

Error Message

>>> import pyomo.environ as pyo
>>> opt = pyo.SolverFactory('cypiopt')
WARNING: Failed to create solver with name 'cypiopt': Failed to set executable
for solver asl. File with name=cypiopt either does not exist or it is not
executable. To skip this validation, call set_executable with validate=False.
Traceback (most recent call last):
  File "/home/harald/.local/lib/python3.11/site-packages/pyomo/opt/base/solvers.py", line 165, in __call__
    opt = self._cls[_implicit_solvers[mode]](**kwds)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/harald/.local/lib/python3.11/site-packages/pyomo/solvers/plugins/solvers/ASL.py", line 45, in __init__
    SystemCallSolver.__init__(self, **kwds)
  File "/home/harald/.local/lib/python3.11/site-packages/pyomo/opt/solver/shellcmd.py", line 67, in __init__
    self.set_executable(name=executable, validate=validate)
  File "/home/harald/.local/lib/python3.11/site-packages/pyomo/opt/solver/shellcmd.py", line 116, in set_executable
    raise ValueError(
ValueError: Failed to set executable for solver asl. File with name=cypiopt either does not exist or it is not executable. To skip this validation, call set_executable with validate=False.
>>>

Information on your system

Pyomo version: Pyomo 6.6.2 Python version: CPython 3.11.2 Operating system: Linux 6.1.0-9-amd64 (Debian) How Pyomo was installed (PyPI, conda, source): PyPI Solver (if applicable): cyipopt

Additional information

mrmundt commented 11 months ago

Thanks for this report, @haraldg . We are actively in the process of refactoring the solver interfaces (see #1030 ). We'll keep this on our radar as we work through that.

Robbybp commented 11 months ago

Hi @haraldg, it looks like you have a typo in your example: pyo.SolverFactory("cyiopt"). pyo.SolverFactory("cyipopt") works for me.

haraldg commented 11 months ago

Indeed. How deeply embarrassing. Though it seems even using cyipopt the proprietary ASL is still needed. So not sure, whether this actually will be better then the ipopt command line solver for me.

At the very least, cyipopt seems quite a bit faster for my test problem. Thank you very much!

Robbybp commented 11 months ago

FYI, the ASL automatic differentiation library, i.e. https://github.com/ampl/asl, is open-source. (As opposed to the AMPL modeling language, which is proprietary.) Our CyIpopt interface indeed uses ASL for automatic differentiation. However, it allows users to interact with Ipopt "directly", which can be useful for debugging. It also potentially allows users to switch out the AD backend, although we have not implemented any other AD interfaces at this point.

At the very least, cyipopt seems quite a bit faster for my test problem.

The only difference between the ipopt and cyipopt Pyomo solvers should be the small amount of Python code that shuttles arrays between ASL and Ipopt in CyIpopt and our interface interface. Therefore, all else equal, I would expect cyipopt to be (marginally) slower. The speed-up you observe may be due to different Ipopt versions (or executables), or different linear solvers, but is likely not due to any functional difference between the solver interfaces. That said, I am glad that it is working well for you.

haraldg commented 11 months ago

FYI, the ASL automatic differentiation library, i.e. https://github.com/ampl/asl, is open-source

Indeed. I have been thinking open source != free software, but it seems ASL is actually BSD 3-Clause licensed. Then why all the fuss about Third-Party-ASL and the dire warnings like:

It is YOUR RESPONSIBILITY to ensure that you are entitled to download and use this third party package.

What am I missing?

The speed-up you observe may be due to different Ipopt versions (or executables), or different linear solvers, but is likely not due to any functional difference between the solver interfaces.

Interesting. AFAIK ipopt supports many smart things like declaring part of the problem linear to speed it up. I had been thinking, that maybe one interface take advantage of this more the the other. I will investigate, where the difference actually comes from.

jsiirola commented 11 months ago

Some additional notes:

both Pyomo's ipopt and cyipopt interfaces use the same "NL writer" for converting your Pyomo model to an "nl file" that the ASL can read (both cyipopt and the command line ipopt executable leverage "nl files" and the ASL for passing the Pyomo model to ipopt). The NL writer already identifies linear expressions in your model and passes them to the NL file / ASL as such (it also identifies the linear portions of nonlinear expressions)
the one thing that comes to mind where cyipopt could be faster is that the cyipopt interface directly links the ipopt library to the running python process, whereas the ipopt executable has to incur the cost of starting up / shutting down a subprocess.

Pyomo / pyomo