search

sagemath / sage

Main repository of SageMath
https://www.sagemath.org
Other
1.41k stars 475 forks source link

Move optional sage optimization backends (COIN, CPLEX, Gurobi) to separate Cython packages to remove OptionalExtension problems #28175

Closed mkoeppe closed 4 years ago

mkoeppe commented 5 years ago

sage.numerical.backends depends on very little from Sage. We propose to split out the optional backends (COIN, CPLEX, Gurobi) to separate Cython packages.

With this ticket, sagelib no longer has the optional dependency on cbc. Instead, these new packages depend on the sage package (i.e., sagelib).

This makes it easier to reconfigure when new solvers are installed, as sagelib does not have to be recompiled. By eliminating the use of the optional extensions mechanisms from sagelib, this will also simplify packaging binary distributions.

The new packages are on PyPI:

wget -P upstream https://files.pythonhosted.org/packages/29/8a/65fd90a890fcfb56166c4071aa7a4fb5b865d313304f60474870f0e89998/sage_numerical_backends_cplex-9.0b12.tar.gz
wget -P upstream https://files.pythonhosted.org/packages/2c/5c/e1532bb6cde28cf86ebffd408ccb500d7b846bc29bbd1c5e479a2f18acb6/sage_numerical_backends_gurobi-9.0b12.tar.gz
wget -P upstream https://files.pythonhosted.org/packages/17/f4/308d3e151d82daf2c58ca26bce157fac31f763b2b60b5173243c09291e0a/sage_numerical_backends_coin-9.0b12.tar.gz

Development here:

See also:

CC: @isuruf @saraedum @kiwifb @dimpase @sagetrac-tmonteil @videlec @seblabbe

Component: numerical

Author: Matthias Koeppe

Branch: 1279897

Reviewer: Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/28175

dcoudert commented 5 years ago
comment:1

Can you be more precise on your plans.

These backends are used in graphs. So what will happen if linear programming backends are in a separate package ?

mkoeppe commented 5 years ago
comment:2

Replying to @dcoudert:

These backends are used in graphs. So what will happen if linear programming backends are in a separate package ?

In my plan, the backends would be a standard package of the sage distribution, and the cython modules it provides would be automatically imported when MixedIntegerLinearProgram is used.

mkoeppe commented 4 years ago

Description changed:

--- 
+++ 
@@ -1,9 +1,37 @@
-`sage.numerical.backends` depends on very little from Sage and should become a separate Cython package. This would make it easier to reconfigure it when new solvers are installed and eliminate some optional extensions from sagelib.
+`sage.numerical.backends` depends on very little from Sage and should become a separate Cython package. This would make it easier to reconfigure it when new solvers are installed and eliminate some optional extensions from sagelib, which would simplify packaging binary distributions.
+
+From `src/module_list.py`:
+
+```
+    OptionalExtension("sage.numerical.backends.gurobi_backend",
+              ["sage/numerical/backends/gurobi_backend.pyx"],
+              libraries = ["gurobi"],
+              condition = os.path.isfile(SAGE_INC + "/gurobi_c.h") and
+                  os.path.isfile(SAGE_LOCAL + "/lib/libgurobi.so")),
+
+    OptionalExtension("sage.numerical.backends.cplex_backend",
+              ["sage/numerical/backends/cplex_backend.pyx"],
+              libraries = ["cplex"],
+              condition = os.path.isfile(SAGE_INC + "/cplex.h") and
+                  os.path.isfile(SAGE_LOCAL + "/lib/libcplex.a")),
+
+    OptionalExtension("sage.numerical.backends.coin_backend",
+              ["sage/numerical/backends/coin_backend.pyx"],
+              language = 'c++',
+              libraries = ["Cbc", "CbcSolver", "Cgl", "Clp", "CoinUtils",
+                           "OsiCbc", "OsiClp", "Osi"] + lapack_libs,
+              library_dirs = lapack_library_dirs,
+              include_dirs = lapack_include_dirs,
+              package = 'cbc'),
+```
+

 Details:
 - `get_solver` uses base_ring
 - GLPKBackend depends on `sage.libs.glpk`, which should also become a separate Cython package (see https://en.wikibooks.org/wiki/GLPK/Python for a list of other glpk bindings)
-
- 
+- For Gurobi and CPLEX, we could alternatively get rid of these Cython modules entirely by making use of their standard Python API instead.

+
+
+
kiwifb commented 4 years ago
comment:5

For glpk it looks like https://github.com/biosustain/swiglpk/releases is the best maintained. Excluding sage and cvxopt.

mkoeppe commented 4 years ago

Description changed:

--- 
+++ 
@@ -31,6 +31,8 @@
 - GLPKBackend depends on `sage.libs.glpk`, which should also become a separate Cython package (see https://en.wikibooks.org/wiki/GLPK/Python for a list of other glpk bindings)
 - For Gurobi and CPLEX, we could alternatively get rid of these Cython modules entirely by making use of their standard Python API instead.

+See also:
+- #26511 - Meta-ticket: Use Python optimization interfaces: PuLP, Pyomo, cylp...
mkoeppe commented 4 years ago

Description changed:

--- 
+++ 
@@ -33,7 +33,7 @@

 See also:
 - #26511 - Meta-ticket: Use Python optimization interfaces: PuLP, Pyomo, cylp...
+- https://groups.google.com/forum/#!topic/sage-packaging/_FLDkqD2X8c 

-
mkoeppe commented 4 years ago
comment:8

I have prepared a first module, https://github.com/mkoeppe/sage-numerical-backends-coin. It depends on Sage (as it inherits from sage.numerical.backends.generic_backend.GenericBackend).

mkoeppe commented 4 years ago

Description changed:

--- 
+++ 
@@ -1,4 +1,4 @@
-`sage.numerical.backends` depends on very little from Sage and should become a separate Cython package. This would make it easier to reconfigure it when new solvers are installed and eliminate some optional extensions from sagelib, which would simplify packaging binary distributions.
+`sage.numerical.backends` depends on very little from Sage. We propose to split out the optional backends to separate Cython packages. This would make it easier to reconfigure when new solvers are installed, as sagelib does not have to be recompiled. By eliminating the use of the optional extensions mechanisms from sagelib, this  would also simplify packaging binary distributions.

 From `src/module_list.py`:
mkoeppe commented 4 years ago

Description changed:

--- 
+++ 
@@ -1,35 +1,24 @@
-`sage.numerical.backends` depends on very little from Sage. We propose to split out the optional backends to separate Cython packages. This would make it easier to reconfigure when new solvers are installed, as sagelib does not have to be recompiled. By eliminating the use of the optional extensions mechanisms from sagelib, this  would also simplify packaging binary distributions.
+`sage.numerical.backends` depends on very little from Sage. We propose to split out the optional backends (COIN, CPLEX, Gurobi) to separate Cython packages. This would make it easier to reconfigure when new solvers are installed, as sagelib does not have to be recompiled. By eliminating the use of the optional extensions mechanisms from sagelib, this will also simplify packaging binary distributions.

-From `src/module_list.py`:
+The new packages are on PyPI:
dimpase commented 4 years ago
comment:10

Is there a git branch to try, replacing Sage's current COIN-OR backend with this one?

mkoeppe commented 4 years ago
comment:11

Not yet (working on it right now), but the 3 backends already pass their test suites separately.

mkoeppe commented 4 years ago

Branch: u/mkoeppe/move_optional_sage_optimization_backends_to_separate_cython_packages_to_remove_optionalextension_problems

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 4 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

437fec5Fixup dependencies
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 4 years ago

Commit: 437fec5

mkoeppe commented 4 years ago
comment:14

Now the packages can be installed using sage -i -c... please test

dimpase commented 4 years ago
comment:15

remains of copypaste, I see PyNormaliz in the branch...

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 4 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

846ca9aReplace CoinBackend, CPLEXBackend, GurobiBackend by their versions imported from sage_numerical_backends_*
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 4 years ago

Changed commit from 437fec5 to 846ca9a

mkoeppe commented 4 years ago

Author: Matthias Koeppe

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 4 years ago

Changed commit from 846ca9a to 3c557b4

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 4 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

3c557b4build/pkgs/sage_numerical_backends_*/spkg-install: Adjust error message
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 4 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

84c83f2src/doc/en/thematic_tutorials/linear_programming.rst: Update COIN/CPLEX/Gurobi install instructions
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 4 years ago

Changed commit from 3c557b4 to 84c83f2

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 4 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

4828766MixedIntegerLinearProgram: Update docstrings for class and `__init__`, refer to thematic tutorial, and reduce copy-paste
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 4 years ago

Changed commit from 84c83f2 to 4828766

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 4 years ago

Changed commit from 4828766 to 189d60b

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 4 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

189d60bget_solver: Reduce copy-paste in docstring
mkoeppe commented 4 years ago

Description changed:

--- 
+++ 
@@ -5,7 +5,7 @@

wget -P upstream https://files.pythonhosted.org/packages/2a/1f/4fc55a0013bc5050ecf253b8bce1c4548380c15d57310a1fae5841405984/sage_numerical_backends_cplex-9.0b9.tar.gz wget -P upstream https://files.pythonhosted.org/packages/72/07/8b52ab9411caec6f4593decebc5660c6ce955f437510d8bd58533866ff31/sage_numerical_backends_gurobi-9.0b9.tar.gz -wget -P upstream https://files.pythonhosted.org/packages/40/01/0b9580bc9db5671cc2446ea089ba1f28bacf85ef0e9f05a4a781bf9dcd7d/sage_numerical_backends_coin-9.0b9.tar.gz +wget -P upstream https://files.pythonhosted.org/packages/24/04/173c29f0d5cdeccfa335ecc00149834e25b9d7fb00279eed9d7a8757e3a6/sage_numerical_backends_coin-9.0b10.tar.gz


 Development here:
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 4 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

357e966src/doc/en/thematic_tutorials/linear_programming.rst: Update license info, URL for cbc
53f3774Update sage_numerical_backends_coin to 9.0b10
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 4 years ago

Changed commit from 189d60b to 53f3774

tscrim commented 4 years ago
comment:25

Since these are currently part of the Sage distribution, it makes sense to me to have them as standard packages (in regards to comment:2). Similarly I would argue they do not require a vote on sage-devel, but I would just post something quick there to see if there are any objections.

mkoeppe commented 4 years ago
comment:26

They can't be standard packages because they only compile if the respective solvers are installed.

mkoeppe commented 4 years ago
comment:27

Note that I narrowed down the ticket recently; the early comments were on a broader proposal to move ALL backends out to separate packages.

tscrim commented 4 years ago
comment:28

Ah, I see. Sorry for the noise.

kiwifb commented 4 years ago
comment:29

coin is problematic for me on sage-on-gentoo, and has been for a while. So, making it a standard package would be annoying. The coin interface doesn't seem to be very stable and coin in sage is not updated often (neither it is in gentoo). Version mismatch usually lead to a failure to build the interface.

mkoeppe commented 4 years ago
comment:30

Well, the point of this ticket is to make the sage build less dependent on coin, not more.

Also, I'd appreciate any help with testing https://github.com/mkoeppe/sage-numerical-backends-coin on sage installations such as the one provided on gentoo. My current test scripts (using GitHub actions) cover conda-forge so far, and I'm working on various debian/ubuntu flavors.

mkoeppe commented 4 years ago

Description changed:

--- 
+++ 
@@ -1,4 +1,8 @@
-`sage.numerical.backends` depends on very little from Sage. We propose to split out the optional backends (COIN, CPLEX, Gurobi) to separate Cython packages. This would make it easier to reconfigure when new solvers are installed, as sagelib does not have to be recompiled. By eliminating the use of the optional extensions mechanisms from sagelib, this will also simplify packaging binary distributions.
+`sage.numerical.backends` depends on very little from Sage. We propose to split out the optional backends (COIN, CPLEX, Gurobi) to separate Cython packages. 
+
+With this ticket, sagelib no longer has the optional dependency on cbc. Instead, these new packages depend on the `sage` package (i.e., sagelib).
+
+This makes it easier to reconfigure when new solvers are installed, as sagelib does not have to be recompiled. By eliminating the use of the optional extensions mechanisms from sagelib, this will also simplify packaging binary distributions.

 The new packages are on PyPI:
kiwifb commented 4 years ago
comment:32

Replying to @mkoeppe:

Well, the point of this ticket is to make the sage build less dependent on coin, not more.

Also, I'd appreciate any help with testing https://github.com/mkoeppe/sage-numerical-backends-coin on sage installations such as the one provided on gentoo. My current test scripts (using GitHub actions) cover conda-forge so far, and I'm working on various debian/ubuntu flavors.

I will have a look at it. It looks interesting but where does it install itself? I cannot promise speedy review as I about to have my annual internet /computer detox for a few days :)

mkoeppe commented 4 years ago
comment:33

It just installs itself as a Python package named sage_numerical_backends_coin.coin_backend (note, underscores, not dots). The https://github.com/mkoeppe/sage-numerical-backends-coin/blob/master/README.md explains how to use it, including how to patch it into a sage for testing purposes.

dimpase commented 4 years ago
comment:34

for coin, there is a verssion mismatch. On pypi and in the ticket description the minor version is b9, in build/pkgs/ it is b10, the latest on github is b12 :-)

So I took b12 from github, renamed the tarfile (arrgh), bumped up the version,

diff --git a/build/pkgs/sage_numerical_backends_coin/checksums.ini b/build/pkgs/sage_numerical_backends_coin/checksums.ini
index 03b250b653..3062652fc1 100644
--- a/build/pkgs/sage_numerical_backends_coin/checksums.ini
+++ b/build/pkgs/sage_numerical_backends_coin/checksums.ini
@@ -1,4 +1,4 @@
 tarball=sage_numerical_backends_coin-VERSION.tar.gz
-sha1=fd96cfc224f1f45bad6bacaaf37df3904fa7a561
-md5=98e5a002648cdf3167df2dafb16cde47
-cksum=869439056
+sha1=0c424f8d01bdd44169630bc898738a4ee146ea2c
+md5=09b6b74d7c8b9b3526424cf8850f8700
+cksum=1339126990
diff --git a/build/pkgs/sage_numerical_backends_coin/package-version.txt b/build/pkgs/sage_numerical_backends_coin/package-version.txt
index 5d6a14cdd9..1700f7be00 100644
--- a/build/pkgs/sage_numerical_backends_coin/package-version.txt
+++ b/build/pkgs/sage_numerical_backends_coin/package-version.txt
@@ -1 +1 @@
-9.0b10
+9.0b12

and then 

$ SAGE_CHECK=yes ./sage -i sage_numerical_backends_coin

worked on Debian with system's cbc 2.9.9. (using #28908)

dimpase commented 4 years ago
comment:35

Where are the tests from now gone sage/numerical/backends/coin_backend.pyx ? Are they all being run while installing with SAGE_CHECK=yes ?

kiwifb commented 4 years ago
comment:36

In a way related to #26511, upstream devs for cbc have published their own python interface https://git.patrikdufresne.com/pdsl/cbcpy the fact that it uses swig may be an obstacle for immediate inclusion but switching to something maintained externally is something to think about. The project is quite new (August 2019 according to https://pypi.org/project/cbcpy/ ) which is why it may escaped attention before.

dimpase commented 4 years ago
comment:37

in the environment as above for coin,

SAGE_CHECK=yes ./sage -f sage_numerical_backends_gurobi

fails with

...
[sage_numerical_backends_gurobi-9.0b9] Running the test suite for sage_numerical_backends_gurobi-9.0b9...
[sage_numerical_backends_gurobi-9.0b9] running test
[sage_numerical_backends_gurobi-9.0b9] Searching for sage-package
[sage_numerical_backends_gurobi-9.0b9] Reading https://pypi.org/simple/sage-package/
[sage_numerical_backends_gurobi-9.0b9] Using gurobi_include_directories=['/home/dimpase/sage/upstream/gurobi900/linux64/include'], libraries=['gurobi90'], library_dirs=['/home/dimpase/sage/upstream/gurobi900/linux64/lib']
[sage_numerical_backends_gurobi-9.0b9] Download error on https://pypi.org/simple/sage-package/: [Errno 110] Connection timed out -- Some packages may not be found!
[sage_numerical_backends_gurobi-9.0b9] Couldn't find index page for 'sage-package' (maybe misspelled?)
[sage_numerical_backends_gurobi-9.0b9] Scanning index of all packages (this may take a while)
[sage_numerical_backends_gurobi-9.0b9] Reading https://pypi.org/simple/
[sage_numerical_backends_gurobi-9.0b9] Download error on https://pypi.org/simple/: [Errno 110] Connection timed out -- Some packages may not be found!
[sage_numerical_backends_gurobi-9.0b9] No local packages or working download links found for sage-package
[sage_numerical_backends_gurobi-9.0b9] error: Could not find suitable distribution for Requirement.parse('sage-package')
[sage_numerical_backends_gurobi-9.0b9] 
[sage_numerical_backends_gurobi-9.0b9] real     1m4.289s
[sage_numerical_backends_gurobi-9.0b9] user     0m0.833s
[sage_numerical_backends_gurobi-9.0b9] sys      0m0.148s
[sage_numerical_backends_gurobi-9.0b9] ************************************************************************
[sage_numerical_backends_gurobi-9.0b9] Error testing package sage_numerical_backends_gurobi-9.0b9
mkoeppe commented 4 years ago
comment:38

Replying to @dimpase:

So I took b12 from github, renamed the tarfile (arrgh)...

I recommend to take the release files from pypi - they have been built with setup.py sdist.

mkoeppe commented 4 years ago
comment:39

Replying to @dimpase:

Where are the tests from now gone sage/numerical/backends/coin_backend.pyx ? Are they all being run while installing with SAGE_CHECK=yes ?

They are here: https://github.com/mkoeppe/sage-numerical-backends-coin/blob/master/sage_numerical_backends_coin/coin_backend.pyx They are run in the CI scripts of the package, and also when installing the spkg with SAGE_CHECK=yes.

dimpase commented 4 years ago
comment:40

Replying to @mkoeppe:

Replying to @dimpase:

So I took b12 from github, renamed the tarfile (arrgh)...

I recommend to take the release files from pypi - they have been built with setup.py sdist.

sure, but b12 was not there yet. https://pypi.org/project/sage-numerical-backends-coin/#files says it made it there ~10 hours ago.

mkoeppe commented 4 years ago

Description changed:

--- 
+++ 
@@ -17,14 +17,9 @@
 - https://github.com/mkoeppe/sage-numerical-backends-coin
 - https://github.com/mkoeppe/sage-numerical-backends-gurobi

----
-
-(In a follow-up ticket, one could consider moving the entire backend machinery to a separate packages. Some notes regarding this:
-- `get_solver` uses base_ring
-- GLPKBackend depends on `sage.libs.glpk`, which should also become a separate Cython package (see https://en.wikibooks.org/wiki/GLPK/Python for a list of other glpk bindings).)
-(In another follow-up ticket, for Gurobi and CPLEX, we could alternatively get rid of these Cython modules entirely by making use of their standard Python API instead.)

 See also:
+- Follow-up: #28920: Move sage optimization backend framework (`sage.numerical.backends`) to separate Cython packages
 - #26511 - Meta-ticket: Use Python optimization interfaces: PuLP, Pyomo, cylp...
 - https://groups.google.com/forum/#!topic/sage-packaging/_FLDkqD2X8c
mkoeppe commented 4 years ago
comment:41

Created follow-up ticket, cleaned up ticket description

dimpase commented 4 years ago
comment:42

Replying to @mkoeppe:

Created follow-up ticket, cleaned up ticket description

some links to packages are not up to date.

mkoeppe commented 4 years ago
comment:43

Replying to @kiwifb:

In a way related to #26511, upstream devs for cbc have published their own python interface https://git.patrikdufresne.com/pdsl/cbcpy the fact that it uses swig may be an obstacle for immediate inclusion but switching to something maintained externally is something to think about. The project is quite new (August 2019 according to https://pypi.org/project/cbcpy/ ) which is why it may escaped attention before.

Thank you! I've added this information to #26511.