search

ERGO-Code / HiGHS

Linear optimization software
MIT License
979 stars 181 forks source link

Make HiGHS installable via PyPI #925

Closed l-kotzur closed 8 months ago

l-kotzur commented 2 years ago

I would be great if one could call pip install highs. This would make the solver in general much easier installable and accessible.

@michaelbynum wrote already an appsi wrapper from Pyomo to HiGHS, as pointed in Pyomo/pyomo#2094. Unfortunately, this has stalled and has not been merged to the main branch. One of the main reasons is that HiGHS is not installable via a package manager. Therefore, it cannot be integrated into a proper CI-pipeline, I guess.

@mckib2 wrote wrappers for sckit and for scipy #156 and brought older versions to PyPI. Would it be possible to make HiGHs as an own PyPI package such that Pyomo and others can more easily use it?

Unfortunately, I personally do not have any experience in packaging C++ libraries.

mckib2 commented 2 years ago

@mckib2 wrote wrappers for sckit and for scipy #156 and brought older versions to PyPI. Would it be possible to make HiGHs as an own PyPI package such that Pyomo and others can more easily use it?

The old scikit that you linked to was a proof-of-concept for integrating into the SciPy build system. All that work has been integrated into SciPy and improved upon with the MIP solver accessible in the upcoming 1.9.0 release. Instead of the old buildutils build system, we now use meson. We expect to develop an object oriented interface in the coming months. 1.9.0 should also contain a very recent version of HiGHS with SciPy binaries available across many platforms and installable via pip.

Any reason you can't use/build on SciPy's linprog function? I believe cvxpy is taking this approach and you'll get support for Python specific build and usage issues

jajhall commented 2 years ago

Any reason you can't use/build on SciPy's linprog function? I believe cvxpy is taking this approach and you'll get support for Python specific build and usage issues

Thanks for the pointer on this @mckib2 We need HiGHS to be installable via a package manager for a number of projects. @michaelbynum and @galabovaa have managed this for MacOS and Linux, but Windows is proving to be very tricky.

I don't know whether we can use/build on SciPy's linprog function, but @galabovaa is soon back from holiday and will have a view

l-kotzur commented 2 years ago

@jajhall Only supporting Linux and MacOS would be fine at least for me. Coin-CBC is only supported on Linux and MacOS as well atm, if I am not wrong.

jajhall commented 2 years ago

For the prototype "thin" Python wrapper highspy around the C++ class, pip install highspy works on two Ubuntu installations, and is not known to fail on any other Linux installation.

l-kotzur commented 2 years ago

@michaelbynum Would this be sufficient to integrate the appsi-HiGHS interface to pyomo?

jajhall commented 2 years ago

pip install highspy also works on (at least one) MacOS. Windows is WIP.

stephane-caron commented 2 years ago

Having highspy on PyPI is great! It helped me a lot in adding an interface for HiGHS in qpsolvers.

Looking forward to the upcoming release (since currently 1.1.2.dev3 is pre-release) :smiley:

Also, not part of this issue but likely a follow-up feature request, I'll be looking forward for the addition of get/setHotStart to the interface (keeping track in https://github.com/stephane-caron/qpsolvers/issues/94).

jajhall commented 2 years ago

Also, not part of this issue but likely a follow-up feature request, I'll be looking forward for the addition of get/setHotStart to the interface (keeping track in stephane-caron/qpsolvers#94).

Hot starting the QP solver is not possible at present, and might not be possible for a long time. My other comments relate to hot-starting the LP solver

michaelbynum commented 2 years ago

I could not find highspy on PyPI. I know pip install works locally, but is it on PyPI yet? If so, could anyone share a link?

@l-kotzur, yes - having highspy on PyPI would be sufficient to integrate the appsi-HiGHS interface into Pyomo.

stephane-caron commented 2 years ago

You can find it at https://pypi.org/project/highspy/. It does not appear in search because it is still a pre-release.

michaelbynum commented 2 years ago

Thanks!

jackbrucesimpson commented 2 years ago

Hi, do you know when/if there were plans for the pip installed version to work on Windows? Thanks so much.

jajhall commented 2 years ago

We certainly "plan" to get pip install highspy to work on Windows, but achieving it is another matter!

michaelbynum commented 2 years ago

@l-kotzur, The Pyomo-Appsi interface to HiGHS is now in Pyomo main and will be in the next release.

l-kotzur commented 2 years ago

@l-kotzur, The Pyomo-Appsi interface to HiGHS is now in Pyomo main and will be in the next release.

@michaelbynum I have seen it. Those are great news, thanks. I am looking forward to test it.

trigaut commented 1 year ago

Hello @jajhall, what is the issue to pip install on windows? I would be glad to help if you think I could. We develop on windows where I work.

The HiGHS Rust interface works on Windows when we install cmake and llvm. But I imagine it is more challenging with PyPI?

jajhall commented 1 year ago

Hello @jajhall, what is the issue to pip install on windows? I would be glad to help if you think I could. We develop on windows where I work.

I don't know specifically, because it's being worked on by @galabovaa. Currently she's combining work on this with integrating HiGHS into another major solver package. Perhaps there's some specific issue that you could help her with, but I'll leave her to ask. Many thanks!

galabovaa commented 1 year ago

Hi @trigaut

Currently on Windows, pip install seems to be generating a wheel, however, when I call "import highspy" from python I get a "DLL not found" error.

I am not an expert on Windows and I have been struggling to get the paths correct.

Any help would be much appreciated. You could have a look at the win branch which contains some minor changes to the CMake files.

Thanks, Ivet

jajhall commented 1 year ago

@trigaut: It's worth (re-?)emphasising that pip install highspy works on Linux and MacOS

galabovaa commented 1 year ago

I just managed to build a wheel for Windows 64bit cp39: https://pypi.org/project/highspy/

If you would please try to install from your windows machines and test if it works on your side that would be great!

l-kotzur commented 1 year ago

The example from the README is working for my case on Windows 10. Great support, thanks!

jackbrucesimpson commented 1 year ago

Version 1.1.2.dev3 was working on Linux but when I just tested out updating to 1.4.0.dev0 it appears to fail.

[image: image.png]

On Tue, 22 Nov 2022 at 01:56, Leander Kotzur @.***> wrote:

The example from the README is working for my case on Windows 10. Great support, thanks!

— Reply to this email directly, view it on GitHub https://github.com/ERGO-Code/HiGHS/issues/925#issuecomment-1322189189, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABQC6PK4N6HDQTKG5PL42UDWJOESRANCNFSM542DWKZA . You are receiving this because you commented.Message ID: @.***>

jajhall commented 1 year ago

Version 1.1.2.dev3 was working on Linux but when I just tested out updating to 1.4.0.dev0 it appears to fail.

1.4.0.dev0 doesn't have a recent branch from @galabovaa merged into it

galabovaa commented 1 year ago

Yesterday I uploaded highspy 1.4.0 only for Windows so trying to update on Linux would give users an error. I will update linux and macOS as well to 1.4.0 once we merge the latest changes into master.

jackbrucesimpson commented 1 year ago

Thanks so much!

On Tue, 22 Nov 2022 at 20:54, galabovaa @.***> wrote:

Yesterday I uploaded highspy 1.4.0 only for Windows so trying to update on Linux would give users an error. I will update linux and macOS as well to 1.4.0 once we merge the latest changes into master.

— Reply to this email directly, view it on GitHub https://github.com/ERGO-Code/HiGHS/issues/925#issuecomment-1323394499, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABQC6PLF5BI5FBKQ3MF6TXTWJSJ5LANCNFSM542DWKZA . You are receiving this because you commented.Message ID: @.***>

trigaut commented 1 year ago

Just tested pip install highspy and to solve many problems modeled with pyomo and using the "appsi_highs" SolverFactory. It works like a charm! Thanks !

l-kotzur commented 1 year ago

For me the same. Therefore also again many thanks to @michaelbynum and the whole HiGHS-team. For my instance, I had at least a factor three faster solving time in comparison to coin-cbc. Just waiting for the official Pyomo-release to set it as default solver.

jajhall commented 1 year ago

@galabovaa I've tested it on my Windows laptop, and it works perfectly!

michaelbynum commented 1 year ago

Excellent news! I would also like to thank @galabovaa for all the effort putting together the PyPI releases. It made things much easier for Pyomo.

julian-belina commented 1 year ago

Is it planned to add a source distribution to the pypi release? This would simplify the creation of conda package.

jajhall commented 1 year ago

Is it planned to add a source distribution to the pypi release? This would simplify the creation of conda package.

If it's not difficult, then I'll ask @galabovaa to add one

rschwarz commented 1 year ago

Another related question: I saw that a version v1.5.1 was released on GitHub some days ago, but the most recent version for highspy on PyPI is still v1.5.0.dev0. Does that mean that the highspy packages are created & published manually?

jajhall commented 1 year ago

The highspy packages are created & published manually. Automating this is WIP

yasirroni commented 1 year ago

pip install highspy is no longer works in Windws in my case, returning "DLL not found" error, again

Requirement already satisfied: highspy (1.5.0.dev0)
Requirement already satisfied: pyomo (from highspy) (6.5.0)
Requirement already satisfied: pybind11 (from highspy) (2.10.4)
Requirement already satisfied: numpy (from highspy) (1.23.5)
Requirement already satisfied: ply (from pyomo->highspy) (3.11)

Specify previous version seems to not works

pip uninstall highspy
WARNING: Skipping highspy as it is not installed.

pip install highspy==1.4.0.dev0
ERROR: Could not find a version that satisfies the requirement highspy==1.4.0.dev0 (from versions: 1.5.0.dev0)
ERROR: No matching distribution found for highspy==1.4.0.dev0

Further reading, the highspy==1.4.0.dev0 only provide py39.

jajhall commented 1 year ago

Sorry about this. highspy is still WIP by @galabovaa, in particular its installation on Windows.

ryanjoneil commented 1 year ago

Are you all looking for some help with this? :smile_cat:

jajhall commented 1 year ago

If you've got someone experienced in building wheels for PyPI, then @galabovaa would welcome someone with whom to discuss issues

galabovaa commented 1 year ago

Some help would be much appreciated, thank you @ryanjoneil

I will send you an email with some details about what the particular issues are, if you could point me to a feasible direction that would be great!

galabovaa commented 8 months ago

highspy is now on PyPi, with a pre-release v1.7.1.dev1 already up and v1.7.1 coming up very soon!

julian-belina commented 8 months ago

@galabovaa will v1.7.1 include an sdist file? This would simplify the creation of a conda package.

galabovaa commented 8 months ago

Yes, it will include an sdist, apologies for the delay. If you wish to check whether the conda build will work, v 1.7.1.dev1 should already contain one:

https://pypi.org/project/highspy/1.7.1.dev1/#files

julian-belina commented 8 months ago

Seems to work, thank you verry much!

Jeromeschmidt commented 5 months ago

Hey @julian-belina @galabovaa , were either of you able to get conda build working for just highspy? If so, did you need to make any modifications to get the build to finish successfully?

I'm getting ModuleNotFoundError: No module named 'highspy._core' when the conda build gets to the test phase.

julian-belina commented 5 months ago

Hey @Jeromeschmidt , unfortunately I was not able to make the conda forge build really work. I am not sure whats causing the problem.

Maybe its a problem with Highs as discussed in this Issue https://github.com/ERGO-Code/HiGHS/issues/1119 . The Issue was fixed in the pre releaese 1.7.1.dev1, but it has not been published as a stable release so far. I dont know how if there is a way to use the pre release to build conda-forge package from a pre release.

There is also the possibility that the recipe is not setup correctly. I am not too familiar with cmake. If you are, feel free to take a look a the recipe: https://github.com/conda-forge/highs-feedstock/tree/main/recipe

If you have suggestions to improve the recipe, I can integrate them or make you a maintainer.