Closed mkoeppe closed 4 years ago
Can you be more precise on your plans.
These backends are used in graphs. So what will happen if linear programming backends are in a separate package ?
Replying to @dcoudert:
These backends are used in graphs. So what will happen if linear programming backends are in a separate package ?
In my plan, the backends would be a standard package of the sage distribution, and the cython modules it provides would be automatically imported when MixedIntegerLinearProgram is used.
Description changed:
---
+++
@@ -1,9 +1,37 @@
-`sage.numerical.backends` depends on very little from Sage and should become a separate Cython package. This would make it easier to reconfigure it when new solvers are installed and eliminate some optional extensions from sagelib.
+`sage.numerical.backends` depends on very little from Sage and should become a separate Cython package. This would make it easier to reconfigure it when new solvers are installed and eliminate some optional extensions from sagelib, which would simplify packaging binary distributions.
+
+From `src/module_list.py`:
+
+```
+ OptionalExtension("sage.numerical.backends.gurobi_backend",
+ ["sage/numerical/backends/gurobi_backend.pyx"],
+ libraries = ["gurobi"],
+ condition = os.path.isfile(SAGE_INC + "/gurobi_c.h") and
+ os.path.isfile(SAGE_LOCAL + "/lib/libgurobi.so")),
+
+ OptionalExtension("sage.numerical.backends.cplex_backend",
+ ["sage/numerical/backends/cplex_backend.pyx"],
+ libraries = ["cplex"],
+ condition = os.path.isfile(SAGE_INC + "/cplex.h") and
+ os.path.isfile(SAGE_LOCAL + "/lib/libcplex.a")),
+
+ OptionalExtension("sage.numerical.backends.coin_backend",
+ ["sage/numerical/backends/coin_backend.pyx"],
+ language = 'c++',
+ libraries = ["Cbc", "CbcSolver", "Cgl", "Clp", "CoinUtils",
+ "OsiCbc", "OsiClp", "Osi"] + lapack_libs,
+ library_dirs = lapack_library_dirs,
+ include_dirs = lapack_include_dirs,
+ package = 'cbc'),
+```
+
Details:
- `get_solver` uses base_ring
- GLPKBackend depends on `sage.libs.glpk`, which should also become a separate Cython package (see https://en.wikibooks.org/wiki/GLPK/Python for a list of other glpk bindings)
-
-
+- For Gurobi and CPLEX, we could alternatively get rid of these Cython modules entirely by making use of their standard Python API instead.
+
+
+
For glpk it looks like https://github.com/biosustain/swiglpk/releases is the best maintained. Excluding sage and cvxopt.
Description changed:
---
+++
@@ -31,6 +31,8 @@
- GLPKBackend depends on `sage.libs.glpk`, which should also become a separate Cython package (see https://en.wikibooks.org/wiki/GLPK/Python for a list of other glpk bindings)
- For Gurobi and CPLEX, we could alternatively get rid of these Cython modules entirely by making use of their standard Python API instead.
+See also:
+- #26511 - Meta-ticket: Use Python optimization interfaces: PuLP, Pyomo, cylp...
Description changed:
---
+++
@@ -33,7 +33,7 @@
See also:
- #26511 - Meta-ticket: Use Python optimization interfaces: PuLP, Pyomo, cylp...
+- https://groups.google.com/forum/#!topic/sage-packaging/_FLDkqD2X8c
-
I have prepared a first module, https://github.com/mkoeppe/sage-numerical-backends-coin.
It depends on Sage (as it inherits from sage.numerical.backends.generic_backend.GenericBackend
).
Description changed:
---
+++
@@ -1,4 +1,4 @@
-`sage.numerical.backends` depends on very little from Sage and should become a separate Cython package. This would make it easier to reconfigure it when new solvers are installed and eliminate some optional extensions from sagelib, which would simplify packaging binary distributions.
+`sage.numerical.backends` depends on very little from Sage. We propose to split out the optional backends to separate Cython packages. This would make it easier to reconfigure when new solvers are installed, as sagelib does not have to be recompiled. By eliminating the use of the optional extensions mechanisms from sagelib, this would also simplify packaging binary distributions.
From `src/module_list.py`:
Description changed:
---
+++
@@ -1,35 +1,24 @@
-`sage.numerical.backends` depends on very little from Sage. We propose to split out the optional backends to separate Cython packages. This would make it easier to reconfigure when new solvers are installed, as sagelib does not have to be recompiled. By eliminating the use of the optional extensions mechanisms from sagelib, this would also simplify packaging binary distributions.
+`sage.numerical.backends` depends on very little from Sage. We propose to split out the optional backends (COIN, CPLEX, Gurobi) to separate Cython packages. This would make it easier to reconfigure when new solvers are installed, as sagelib does not have to be recompiled. By eliminating the use of the optional extensions mechanisms from sagelib, this will also simplify packaging binary distributions.
-From `src/module_list.py`:
+The new packages are on PyPI:
OptionalExtension("sage.numerical.backends.gurobi_backend",
["sage/numerical/backends/gurobi_backend.pyx"],
libraries = ["gurobi"],
condition = os.path.isfile(SAGE_INC + "/gurobi_c.h") and
os.path.isfile(SAGE_LOCAL + "/lib/libgurobi.so")), +wget -P upstream https://files.pythonhosted.org/packages/2a/1f/4fc55a0013bc5050ecf253b8bce1c4548380c15d57310a1fae5841405984/sage_numerical_backends_cplex-9.0b9.tar.gz +wget -P upstream https://files.pythonhosted.org/packages/72/07/8b52ab9411caec6f4593decebc5660c6ce955f437510d8bd58533866ff31/sage_numerical_backends_gurobi-9.0b9.tar.gz +wget -P upstream https://files.pythonhosted.org/packages/40/01/0b9580bc9db5671cc2446ea089ba1f28bacf85ef0e9f05a4a781bf9dcd7d/sage_numerical_backends_coin-9.0b9.tar.gz +```
OptionalExtension("sage.numerical.backends.cplex_backend",
["sage/numerical/backends/cplex_backend.pyx"],
libraries = ["cplex"],
condition = os.path.isfile(SAGE_INC + "/cplex.h") and
os.path.isfile(SAGE_LOCAL + "/lib/libcplex.a")), +Development here: +- https://github.com/mkoeppe/sage-numerical-backends-cplex +- https://github.com/mkoeppe/sage-numerical-backends-coin +- https://github.com/mkoeppe/sage-numerical-backends-gurobi
OptionalExtension("sage.numerical.backends.coin_backend",
["sage/numerical/backends/coin_backend.pyx"],
language = 'c++',
libraries = ["Cbc", "CbcSolver", "Cgl", "Clp", "CoinUtils",
"OsiCbc", "OsiClp", "Osi"] + lapack_libs,
library_dirs = lapack_library_dirs,
include_dirs = lapack_include_dirs,
package = 'cbc'), -``` +---
-Details: +(In a follow-up ticket, one could consider moving the entire backend machinery to a separate packages. Some notes regarding this:
get_solver
uses base_ring
-- GLPKBackend depends on sage.libs.glpk
, which should also become a separate Cython package (see https://en.wikibooks.org/wiki/GLPK/Python for a list of other glpk bindings)
-- For Gurobi and CPLEX, we could alternatively get rid of these Cython modules entirely by making use of their standard Python API instead.
+- GLPKBackend depends on sage.libs.glpk
, which should also become a separate Cython package (see https://en.wikibooks.org/wiki/GLPK/Python for a list of other glpk bindings).)
+(In another follow-up ticket, for Gurobi and CPLEX, we could alternatively get rid of these Cython modules entirely by making use of their standard Python API instead.)See also:
Is there a git branch to try, replacing Sage's current COIN-OR backend with this one?
Not yet (working on it right now), but the 3 backends already pass their test suites separately.
Branch pushed to git repo; I updated commit sha1. New commits:
437fec5 | Fixup dependencies |
Commit: 437fec5
Now the packages can be installed using sage -i -c
... please test
remains of copypaste, I see PyNormaliz
in the branch...
Branch pushed to git repo; I updated commit sha1. New commits:
846ca9a | Replace CoinBackend, CPLEXBackend, GurobiBackend by their versions imported from sage_numerical_backends_* |
Author: Matthias Koeppe
Branch pushed to git repo; I updated commit sha1. New commits:
3c557b4 | build/pkgs/sage_numerical_backends_*/spkg-install: Adjust error message |
Branch pushed to git repo; I updated commit sha1. New commits:
84c83f2 | src/doc/en/thematic_tutorials/linear_programming.rst: Update COIN/CPLEX/Gurobi install instructions |
Branch pushed to git repo; I updated commit sha1. New commits:
4828766 | MixedIntegerLinearProgram: Update docstrings for class and `__init__`, refer to thematic tutorial, and reduce copy-paste |
Branch pushed to git repo; I updated commit sha1. New commits:
189d60b | get_solver: Reduce copy-paste in docstring |
Description changed:
---
+++
@@ -5,7 +5,7 @@
wget -P upstream https://files.pythonhosted.org/packages/2a/1f/4fc55a0013bc5050ecf253b8bce1c4548380c15d57310a1fae5841405984/sage_numerical_backends_cplex-9.0b9.tar.gz wget -P upstream https://files.pythonhosted.org/packages/72/07/8b52ab9411caec6f4593decebc5660c6ce955f437510d8bd58533866ff31/sage_numerical_backends_gurobi-9.0b9.tar.gz -wget -P upstream https://files.pythonhosted.org/packages/40/01/0b9580bc9db5671cc2446ea089ba1f28bacf85ef0e9f05a4a781bf9dcd7d/sage_numerical_backends_coin-9.0b9.tar.gz +wget -P upstream https://files.pythonhosted.org/packages/24/04/173c29f0d5cdeccfa335ecc00149834e25b9d7fb00279eed9d7a8757e3a6/sage_numerical_backends_coin-9.0b10.tar.gz
Development here:
Since these are currently part of the Sage distribution, it makes sense to me to have them as standard packages (in regards to comment:2). Similarly I would argue they do not require a vote on sage-devel, but I would just post something quick there to see if there are any objections.
They can't be standard packages because they only compile if the respective solvers are installed.
Note that I narrowed down the ticket recently; the early comments were on a broader proposal to move ALL backends out to separate packages.
Ah, I see. Sorry for the noise.
coin
is problematic for me on sage-on-gentoo, and has been for a while. So, making it a standard package would be annoying. The coin
interface doesn't seem to be very stable and coin
in sage is not updated often (neither it is in gentoo). Version mismatch usually lead to a failure to build the interface.
Well, the point of this ticket is to make the sage build less dependent on coin, not more.
Also, I'd appreciate any help with testing https://github.com/mkoeppe/sage-numerical-backends-coin on sage installations such as the one provided on gentoo. My current test scripts (using GitHub actions) cover conda-forge so far, and I'm working on various debian/ubuntu flavors.
Description changed:
---
+++
@@ -1,4 +1,8 @@
-`sage.numerical.backends` depends on very little from Sage. We propose to split out the optional backends (COIN, CPLEX, Gurobi) to separate Cython packages. This would make it easier to reconfigure when new solvers are installed, as sagelib does not have to be recompiled. By eliminating the use of the optional extensions mechanisms from sagelib, this will also simplify packaging binary distributions.
+`sage.numerical.backends` depends on very little from Sage. We propose to split out the optional backends (COIN, CPLEX, Gurobi) to separate Cython packages.
+
+With this ticket, sagelib no longer has the optional dependency on cbc. Instead, these new packages depend on the `sage` package (i.e., sagelib).
+
+This makes it easier to reconfigure when new solvers are installed, as sagelib does not have to be recompiled. By eliminating the use of the optional extensions mechanisms from sagelib, this will also simplify packaging binary distributions.
The new packages are on PyPI:
Replying to @mkoeppe:
Well, the point of this ticket is to make the sage build less dependent on coin, not more.
Also, I'd appreciate any help with testing https://github.com/mkoeppe/sage-numerical-backends-coin on sage installations such as the one provided on gentoo. My current test scripts (using GitHub actions) cover conda-forge so far, and I'm working on various debian/ubuntu flavors.
I will have a look at it. It looks interesting but where does it install itself? I cannot promise speedy review as I about to have my annual internet /computer detox for a few days :)
It just installs itself as a Python package named sage_numerical_backends_coin.coin_backend
(note, underscores, not dots).
The https://github.com/mkoeppe/sage-numerical-backends-coin/blob/master/README.md explains how to use it, including how to patch it into a sage for testing purposes.
for coin, there is a verssion mismatch. On pypi and in the ticket description the minor version is b9, in build/pkgs/ it is b10, the latest on github is b12 :-)
So I took b12 from github, renamed the tarfile (arrgh), bumped up the version,
diff --git a/build/pkgs/sage_numerical_backends_coin/checksums.ini b/build/pkgs/sage_numerical_backends_coin/checksums.ini
index 03b250b653..3062652fc1 100644
--- a/build/pkgs/sage_numerical_backends_coin/checksums.ini
+++ b/build/pkgs/sage_numerical_backends_coin/checksums.ini
@@ -1,4 +1,4 @@
tarball=sage_numerical_backends_coin-VERSION.tar.gz
-sha1=fd96cfc224f1f45bad6bacaaf37df3904fa7a561
-md5=98e5a002648cdf3167df2dafb16cde47
-cksum=869439056
+sha1=0c424f8d01bdd44169630bc898738a4ee146ea2c
+md5=09b6b74d7c8b9b3526424cf8850f8700
+cksum=1339126990
diff --git a/build/pkgs/sage_numerical_backends_coin/package-version.txt b/build/pkgs/sage_numerical_backends_coin/package-version.txt
index 5d6a14cdd9..1700f7be00 100644
--- a/build/pkgs/sage_numerical_backends_coin/package-version.txt
+++ b/build/pkgs/sage_numerical_backends_coin/package-version.txt
@@ -1 +1 @@
-9.0b10
+9.0b12
and then
$ SAGE_CHECK=yes ./sage -i sage_numerical_backends_coin
worked on Debian with system's cbc 2.9.9. (using #28908)
Where are the tests from now gone sage/numerical/backends/coin_backend.pyx
?
Are they all being run while installing with SAGE_CHECK=yes
?
In a way related to #26511, upstream devs for cbc have published their own python interface https://git.patrikdufresne.com/pdsl/cbcpy the fact that it uses swig may be an obstacle for immediate inclusion but switching to something maintained externally is something to think about. The project is quite new (August 2019 according to https://pypi.org/project/cbcpy/ ) which is why it may escaped attention before.
in the environment as above for coin,
SAGE_CHECK=yes ./sage -f sage_numerical_backends_gurobi
fails with
...
[sage_numerical_backends_gurobi-9.0b9] Running the test suite for sage_numerical_backends_gurobi-9.0b9...
[sage_numerical_backends_gurobi-9.0b9] running test
[sage_numerical_backends_gurobi-9.0b9] Searching for sage-package
[sage_numerical_backends_gurobi-9.0b9] Reading https://pypi.org/simple/sage-package/
[sage_numerical_backends_gurobi-9.0b9] Using gurobi_include_directories=['/home/dimpase/sage/upstream/gurobi900/linux64/include'], libraries=['gurobi90'], library_dirs=['/home/dimpase/sage/upstream/gurobi900/linux64/lib']
[sage_numerical_backends_gurobi-9.0b9] Download error on https://pypi.org/simple/sage-package/: [Errno 110] Connection timed out -- Some packages may not be found!
[sage_numerical_backends_gurobi-9.0b9] Couldn't find index page for 'sage-package' (maybe misspelled?)
[sage_numerical_backends_gurobi-9.0b9] Scanning index of all packages (this may take a while)
[sage_numerical_backends_gurobi-9.0b9] Reading https://pypi.org/simple/
[sage_numerical_backends_gurobi-9.0b9] Download error on https://pypi.org/simple/: [Errno 110] Connection timed out -- Some packages may not be found!
[sage_numerical_backends_gurobi-9.0b9] No local packages or working download links found for sage-package
[sage_numerical_backends_gurobi-9.0b9] error: Could not find suitable distribution for Requirement.parse('sage-package')
[sage_numerical_backends_gurobi-9.0b9]
[sage_numerical_backends_gurobi-9.0b9] real 1m4.289s
[sage_numerical_backends_gurobi-9.0b9] user 0m0.833s
[sage_numerical_backends_gurobi-9.0b9] sys 0m0.148s
[sage_numerical_backends_gurobi-9.0b9] ************************************************************************
[sage_numerical_backends_gurobi-9.0b9] Error testing package sage_numerical_backends_gurobi-9.0b9
Replying to @dimpase:
So I took b12 from github, renamed the tarfile (arrgh)...
I recommend to take the release files from pypi - they have been built with setup.py sdist
.
Replying to @dimpase:
Where are the tests from now gone
sage/numerical/backends/coin_backend.pyx
? Are they all being run while installing withSAGE_CHECK=yes
?
They are here:
https://github.com/mkoeppe/sage-numerical-backends-coin/blob/master/sage_numerical_backends_coin/coin_backend.pyx
They are run in the CI scripts of the package, and also when installing the spkg with SAGE_CHECK=yes
.
Replying to @mkoeppe:
Replying to @dimpase:
So I took b12 from github, renamed the tarfile (arrgh)...
I recommend to take the release files from pypi - they have been built with
setup.py sdist
.
sure, but b12 was not there yet. https://pypi.org/project/sage-numerical-backends-coin/#files says it made it there ~10 hours ago.
Description changed:
---
+++
@@ -17,14 +17,9 @@
- https://github.com/mkoeppe/sage-numerical-backends-coin
- https://github.com/mkoeppe/sage-numerical-backends-gurobi
----
-
-(In a follow-up ticket, one could consider moving the entire backend machinery to a separate packages. Some notes regarding this:
-- `get_solver` uses base_ring
-- GLPKBackend depends on `sage.libs.glpk`, which should also become a separate Cython package (see https://en.wikibooks.org/wiki/GLPK/Python for a list of other glpk bindings).)
-(In another follow-up ticket, for Gurobi and CPLEX, we could alternatively get rid of these Cython modules entirely by making use of their standard Python API instead.)
See also:
+- Follow-up: #28920: Move sage optimization backend framework (`sage.numerical.backends`) to separate Cython packages
- #26511 - Meta-ticket: Use Python optimization interfaces: PuLP, Pyomo, cylp...
- https://groups.google.com/forum/#!topic/sage-packaging/_FLDkqD2X8c
Created follow-up ticket, cleaned up ticket description
Replying to @mkoeppe:
Created follow-up ticket, cleaned up ticket description
some links to packages are not up to date.
Replying to @kiwifb:
In a way related to #26511, upstream devs for cbc have published their own python interface https://git.patrikdufresne.com/pdsl/cbcpy the fact that it uses swig may be an obstacle for immediate inclusion but switching to something maintained externally is something to think about. The project is quite new (August 2019 according to https://pypi.org/project/cbcpy/ ) which is why it may escaped attention before.
Thank you! I've added this information to #26511.
sage.numerical.backends
depends on very little from Sage. We propose to split out the optional backends (COIN, CPLEX, Gurobi) to separate Cython packages.With this ticket, sagelib no longer has the optional dependency on cbc. Instead, these new packages depend on the
sage
package (i.e., sagelib).This makes it easier to reconfigure when new solvers are installed, as sagelib does not have to be recompiled. By eliminating the use of the optional extensions mechanisms from sagelib, this will also simplify packaging binary distributions.
The new packages are on PyPI:
Development here:
See also:
sage.numerical.backends
) to separate Cython packages26511 - Meta-ticket: Use Python optimization interfaces: PuLP, Pyomo, cylp...
CC: @isuruf @saraedum @kiwifb @dimpase @sagetrac-tmonteil @videlec @seblabbe
Component: numerical
Author: Matthias Koeppe
Branch:
1279897
Reviewer: Dima Pasechnik
Issue created by migration from https://trac.sagemath.org/ticket/28175