Upgrade to cvxopt 1.2.x

[dimpase]

This ticket is to upgrade to cvxopt 1.2.3

• Download page: https://pypi.org/project/cvxopt/#files
• Tarballs:
  • https://files.pythonhosted.org/packages/00/02/13657bcd12d93f53b64a19b68d204f43d5f3247c29287c9eaeefcfa58fb9/cvxopt-1.2.3.tar.gz
  • https://github.com/DrTimothyAldenDavis/SuiteSparse/archive/v5.6.0.tar.gz

[Somehow downloads as SuiteSparse-5.6.0.tar.gz in chromium] Since cvxopt 1.1.9, SuiteSparse is no longer included, thus it will have to be packaged separately.

Ticket #19687 was our previous upgrade, to cvxopt 1.1.8 (released 2015-09-22).

CC: @antonio-rojas @embray @kiwifb @timokau @infinity0 @pcpa @saraedum @slel @strogdon @tobihan @sagetrac-tmonteil

Component: packages: standard

Keywords: upgrade, cvxopt

Author: François Bissey

Branch: e23d5c5

Reviewer: Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/22380

[dimpase]

Attachment: LICENSE.txt

suite-sparse license

[dimpase]

Description changed:

--- 
+++ 
@@ -1 +1,4 @@
 cvxopt 1.1.9 was released in Nov 2016, so we need to update.
+
+[SuiteSparse http://faculty.cse.tamu.edu/davis/suitesparse.html] is no longer included, thus we will have to package
+it seperately.

[dimpase]

comment:1

the 1st task would be to understand whether the Suite tarfile needs to be cut to exclude non-GPL compatible parts (see attachment)...

[kiwifb]

comment:2

I don't know about debian and other but we split suitesparse in components in Gentoo. I would be shocked if the other didn't do the same. It looks like upstream integrated a lot of the autoconfig patches that were floating around which makes life easier. What cvxopt needs:

• suitesparseconfig amd umfpack colamd cholmod

Indentation is for dependency. That's without optional or more complicated configuration - a minimal setup (although options using blas/lapack could be enabled without extra burden). All the above in mimimum configuration is LGPL 2.1. In non-mimimum configuration some switch to GPL-2.

[kiwifb]

comment:3

From setup.py - this can all be overridden, but it horrifies me

# Guess SUITESPARSE_LIB_DIR and SUITESPARSE_INC_DIR
if platform.platform().startswith("Darwin"):
    # macOS
    SUITESPARSE_LIB_DIR = '/usr/local/lib'
    SUITESPARSE_INC_DIR = '/usr/local/include'
else: 
    # default: Ubuntu/Debian
    SUITESPARSE_LIB_DIR = "/usr/lib/x86_64-linux-gnu"
    SUITESPARSE_INC_DIR = "/usr/include/suitesparse"

The only line that is valid is the default include directory for debian. The rest is accessible by default without defining anything. Basically it destroys stuff that would work out of the box without any values. This is relevant in that it means we will have to override things manually when it could work out of the box if left alone. That's not just suitesparse. gsl, glpk... Of course there are limitations of distutils that complicate things (an empty includedir statement generate a -I in the compilation line instead of nothing at all, the compiler then gets confused. The best default is probably '.' if you have to put something).

[kiwifb]

comment:4

Good news, cvxopt-1.1.9 is in gentoo now and no doctests were harmed by the upgrade in sage-on-gentoo. The bad news is that writing the spkgs for this is not high on my TODO list.

[slel]

comment:5

Sage-on-Gentoo just upgraded to cvxopt 1.2.1.

See https://github.com/cschwan/sage-on-gentoo/issues/524

[slel]

Description changed:

--- 
+++ 
@@ -1,4 +1,8 @@
-cvxopt 1.1.9 was released in Nov 2016, so we need to update.
+This ticket is to upgrade to cvxopt 1.2.1, released 2018-08-31.

-[SuiteSparse http://faculty.cse.tamu.edu/davis/suitesparse.html] is no longer included, thus we will have to package
-it seperately.
+Tarball: https://files.pythonhosted.org/packages/50/f5/254cbdccf67eb86fb6dec91da48430d16ac4195e2c30100239e54a79eebe/cvxopt-1.2.1.tar.gz
+
+Since cvxopt 1.1.9, [SuiteSparse](http://faculty.cse.tamu.edu/davis/suitesparse.html)
+is no longer included, thus it will have to be packaged separately.
+
+Ticket #19687 was our previous upgrade, to cvxopt 1.1.8 (released 2015-09-22).

[slel]

Changed keywords from none to upgrade, cvxopt

[dimpase]

comment:6

It would be great if Gentoo's suite-sparse can be lifted to become a Sage package.

For starters I should get sage-on-gentoo running on my Gentoo laptop. How do I start with it?

[kiwifb]

comment:7

Replying to @dimpase:

It would be great if Gentoo's suite-sparse can be lifted to become a Sage package.

For starters I should get sage-on-gentoo running on my Gentoo laptop. How do I start with it?

Well we are a bit old school but the instructions of the README at https://github.com/cschwan/sage-on-gentoo are still working. One day I'll update them to something a little bit more modern. Note that you also need the science overlay. I won't finish sage 8.4 tongiht I am afraid. Too much to do, it will be ready in 12 hours.

[Konrad127123]

comment:8

Note: cvxopt 1.2.2 was released a few days ago.

[slel]

comment:9

Updating ticket description following the release of cvxopt 1.2.2.

[slel]

Description changed:

--- 
+++ 
@@ -1,6 +1,7 @@
-This ticket is to upgrade to cvxopt 1.2.1, released 2018-08-31.
+This ticket is to upgrade to cvxopt 1.2.2, released 2018-10-19.

-Tarball: https://files.pythonhosted.org/packages/50/f5/254cbdccf67eb86fb6dec91da48430d16ac4195e2c30100239e54a79eebe/cvxopt-1.2.1.tar.gz
+- **Download page**: https://pypi.org/project/cvxopt/#files
+- **Tarball**: https://files.pythonhosted.org/packages/53/9c/cb7f362e651ff6eb1c6c490435ffded2a2c4074d54fa87bf9a99be4d7763/cvxopt-1.2.2.tar.gz

 Since cvxopt 1.1.9, [SuiteSparse](http://faculty.cse.tamu.edu/davis/suitesparse.html)
 is no longer included, thus it will have to be packaged separately.

[dimpase]

comment:10

Replying to @kiwifb:

Well we are a bit old school but the instructions of the README at https://github.com/cschwan/sage-on-gentoo are still working.

A somewhat cryptic error:

!!! The ebuild selected to satisfy "sage" has unmet requirements.
- sci-mathematics/sage-8.4::sage-on-gentoo USE="X doc-html-bin -bliss -debug -doc-html -doc-pdf -doc-pdf-bin -jmol -latex -modular_decomposition -sagenb -testsuite" ABI_X86="(64)" L10N="-ca -de -en -es -fr -hu -it -ja -pt -ru -tr" PYTHON_TARGETS="python2_7 python3_6"

  The following REQUIRED_USE flag constraints are unsatisfied:
    doc-html-bin? ( l10n_en )

  The above constraints are a subset of the following complete expression:
    doc-html? ( l10n_en sagenb ) doc-pdf? ( sagenb ) testsuite? ( any-of ( doc-html doc-html-bin ) ) doc-html-bin? ( !doc-html !doc-pdf l10n_en ) doc-pdf-bin? ( !doc-html !doc-pdf l10n_en ) any-of ( python_targets_python2_7 python_targets_python3_6 )

no idea what's wrong. Changing doc-html-bin to -doc-html-bin allows the build to go forward.

[kiwifb]

comment:11

That's part of the possible useflags. Basically I allow you to choose which language you want to build but because the english version is the only complete one, if you elect to have documentation, the english one is compulsory. Put

L10N="en"

in make.conf and you'll get pre-built english documentation installed.

[kiwifb]

comment:12

I am going to make an attempt at a "reduced" suitesparse spkg. Reduced because I will not make default packages (which aren't used by anything else) that use cmake. Note that suitesparse will move to cmake for the whole thing in the near future.

The fun bit is that suitesparse is effectively a small mini distribution itself (but most of the upstreams are now totally absorbed by the project). It doesn't currently work very nicely with building phases unless we break down the various components. It does install directly on the file system and because it is like a mini distribution, trying to stop it will break the build system (where are my libraries and my include files).

Once it is completely cmake-ified it may actually behave better in that respect.

[kiwifb]

Commit: 32e028e

[kiwifb]

Description changed:

--- 
+++ 
@@ -1,7 +1,9 @@
 This ticket is to upgrade to cvxopt 1.2.2, released 2018-10-19.

 - **Download page**: https://pypi.org/project/cvxopt/#files
-- **Tarball**: https://files.pythonhosted.org/packages/53/9c/cb7f362e651ff6eb1c6c490435ffded2a2c4074d54fa87bf9a99be4d7763/cvxopt-1.2.2.tar.gz
+- **Tarballs**: 
+  * https://files.pythonhosted.org/packages/53/9c/cb7f362e651ff6eb1c6c490435ffded2a2c4074d54fa87bf9a99be4d7763/cvxopt-1.2.2.tar.gz
+  * http://faculty.cse.tamu.edu/davis/SuiteSparse/SuiteSparse-5.3.0.tar.gz

 Since cvxopt 1.1.9, [SuiteSparse](http://faculty.cse.tamu.edu/davis/suitesparse.html)
 is no longer included, thus it will have to be packaged separately.

[kiwifb]

Branch: u/fbissey/cvxopt-1.2.2

[kiwifb]

comment:13

SPKG.txt for suitesparse needs a bit of polish but this is a good start for anyone who wants to tests this. This is open for reviews even though I am expecting a bit more work needs to be done.

---

New commits:

94517d2	basic suitsparse spkg
903741f	initial update to cvxopt 1.2.2
093e5f3	Merge branch 'develop' into cvxopt-1.22
32e028e	add suitesparse to cvxopt dependencies

[kiwifb]

Author: François Bissey

[Konrad127123]

comment:14

I'm getting an Invalid checksum error for cvxopt-1.2.2.tar.gz.

cvxopt/checksums.ini says:

tarball=cvxopt-VERSION.tar.gz
sha1=725ce85fe122cd0fcda9d1b1a45ae43adf6fea2c
md5=ef9ab5f939380e9488e57b113103d96a
cksum=4099798435

but I get:

$ sha1sum ../../upstream/cvxopt-1.2.2.tar.gz 
43113a138e5e8f3ba42dc53ccb8c3c7501d5576b  ../../upstream/cvxopt-1.2.2.tar.gz
$ md5sum ../../upstream/cvxopt-1.2.2.tar.gz 
fc2546fdba2faa6c4996e9b40fce5269  ../../upstream/cvxopt-1.2.2.tar.gz
$ cksum ../../upstream/cvxopt-1.2.2.tar.gz
3288957477 1877253 ../../upstream/cvxopt-1.2.2.tar.gz

[kiwifb]

comment:15

My bad I used the wrong tarball (github instead of pypi) when creating the checksum. I'll fix it shortly.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

21e7280

fix cvxopt checksum

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 32e028e to 21e7280

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

87f0048

Flesh out SPKG.txt

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 21e7280 to 87f0048

[kiwifb]

comment:18

I would be OK for things to ship as they are now but I am open to suggestions. An alternative is to split SuiteSparse into components but that's more work.

[Konrad127123]

comment:19

Building with SAGE_CHECK="yes" still fails when running headless (#24657).

[kiwifb]

comment:20

I see, I was not aware of this issue. I see that matplotlib is involved, probably in the same way some doctest could be failing on debian. That will require some code inspection.

[dimpase]

comment:21

Without openmp, or with openmp installed in /usr/local/, I get I get

[suitesparse-5.3.0] /usr/bin/ld: error: unable to find library -lomp
[suitesparse-5.3.0] cc: error: linker command failed with exit code 1 (use -v to see invocation)

Is openmp a pre-req?

One way or another, the installation ought to be smarter as far as the location of openmp is concerned.

[kiwifb]

comment:22

Yes it should be smarter. Now can you give me more info? Is this OS X? which sub-package of suitesparse? In short a longer log please.

[dimpase]

comment:23

This is FreeBSD 12.0. With openmp installed (in /usr/local, which is "standard" location on such a system), everything works (passes relevant Sage tests too) if I export LDFLAGS="$LDFLAGS -L/usr/local/lib" before installation. If LDFLAGS are not passed, it breaks at CHOLMOD as follows:

...
[suitesparse-5.3.0] a - cholmod_l_nesdis.o
[suitesparse-5.3.0] a - cholmod_l_camd.o
[suitesparse-5.3.0] cc -L/usr/home/dimpase/Sage/sagemath/local/lib -Wl,-rpath,/usr/home/dimpase/Sage/sagemath/local/lib  -L/usr/home/dimpase/Sage/sagemath/local/lib -L/usr/home/dimpase/Sage/sagemath/local/lib -L/usr/home/dimpase/Sage/sagemath/local/lib -shared -Wl,-soname -Wl,libcholmod.so.3 -Wl,--no-undefined -fopenmp cholmod_aat.o cholmod_add.o cholmod_band.o cholmod_change_factor.o cholmod_common.o cholmod_complex.o cholmod_copy.o cholmod_dense.o cholmod_error.o cholmod_factor.o cholmod_memory.o cholmod_sparse.o cholmod_transpose.o cholmod_triplet.o cholmod_version.o cholmod_check.o cholmod_read.o cholmod_write.o cholmod_amd.o cholmod_analyze.o cholmod_colamd.o cholmod_etree.o cholmod_factorize.o cholmod_postorder.o cholmod_rcond.o cholmod_resymbol.o cholmod_rowcolcounts.o cholmod_rowfac.o cholmod_solve.o cholmod_spsolve.o cholmod_drop.o cholmod_horzcat.o cholmod_norm.o cholmod_scale.o cholmod_sdmult.o cholmod_ssmult.o cholmod_submatrix.o cholmod_vertcat.o cholmod_symmetry.o cholmod_rowadd.o cholmod_rowdel.o cholmod_updown.o cholmod_super_numeric.o cholmod_super_solve.o cholmod_super_symbolic.o cholmod_ccolamd.o cholmod_csymamd.o cholmod_metis.o cholmod_nesdis.o cholmod_camd.o cholmod_l_aat.o cholmod_l_add.o cholmod_l_band.o cholmod_l_change_factor.o cholmod_l_common.o cholmod_l_complex.o cholmod_l_copy.o cholmod_l_dense.o cholmod_l_error.o cholmod_l_factor.o cholmod_l_memory.o cholmod_l_sparse.o cholmod_l_transpose.o cholmod_l_triplet.o cholmod_l_version.o cholmod_l_check.o cholmod_l_read.o cholmod_l_write.o cholmod_l_amd.o cholmod_l_analyze.o cholmod_l_colamd.o cholmod_l_etree.o cholmod_l_factorize.o cholmod_l_postorder.o cholmod_l_rcond.o cholmod_l_resymbol.o cholmod_l_rowcolcounts.o cholmod_l_rowfac.o cholmod_l_solve.o cholmod_l_spsolve.o cholmod_l_drop.o cholmod_l_horzcat.o cholmod_l_norm.o cholmod_l_scale.o cholmod_l_sdmult.o cholmod_l_ssmult.o cholmod_l_submatrix.o cholmod_l_vertcat.o cholmod_l_symmetry.o cholmod_l_rowadd.o cholmod_l_rowdel.o cholmod_l_updown.o cholmod_l_super_numeric.o cholmod_l_super_solve.o cholmod_l_super_symbolic.o cholmod_l_ccolamd.o cholmod_l_csymamd.o cholmod_l_metis.o cholmod_l_nesdis.o cholmod_l_camd.o -o /usr/home/dimpase/Sage/sagemath/local/lib/libcholmod.so.3.0.12 -lm -lamd -lcolamd -lsuitesparseconfig -lccolamd -lcamd -lopenblas  -lopenblas 
[suitesparse-5.3.0] ranlib libcholmod.a
[suitesparse-5.3.0] /usr/bin/ld: error: unable to find library -lomp
[suitesparse-5.3.0] cc: error: linker command failed with exit code 1 (use -v to see invocation)
[suitesparse-5.3.0] gmake[4]: *** [Makefile:544: /usr/home/dimpase/Sage/sagemath/local/lib/libcholmod.so.3.0.12] Error 1
[suitesparse-5.3.0] gmake[4]: Leaving directory '/usr/home/dimpase/Sage/sagemath/local/var/tmp/sage/build/suitesparse-5.3.0/src/CHOLMOD/Lib'

And with openmp not installed, it also breaks at CHOLMOD, although the output is differently ordered, somehow.

...
[suitesparse-5.3.0] a - cholmod_l_nesdis.o
[suitesparse-5.3.0] a - cholmod_l_camd.o
[suitesparse-5.3.0] ranlib libcholmod.a
[suitesparse-5.3.0] /usr/bin/ld: error: unable to find library -lomp
[suitesparse-5.3.0] cc: error: linker command failed with exit code 1 (use -v to see invocation)
[suitesparse-5.3.0] gmake[4]: *** [Makefile:544: /usr/home/dimpase/Sage/sagemath/local/lib/libcholmod.so.3.0.12] Error 1
[suitesparse-5.3.0] gmake[4]: Leaving directory '/usr/home/dimpase/Sage/sagemath/local/var/tmp/sage/build/suitesparse-5.3.0/src/CHOLMOD/Lib'

[kiwifb]

comment:24

Yes it is unusual. And very strange. There is some control for openmp in the main makefile I think so it has to be turned off there (or autodetected properly there). But not finding stuff in /usr/local/lib is highly unusual - and it appears this is not your compiler's fault but ld's fault. Is it GNU ld, some kind of BSD ld or llvm's ld?

[kiwifb]

comment:25

Is there urgency to have this in 8.5? I have been thinking that I have a longer time frame I may want to try splitting suitesparse into just the components needed for cvxopt.

[dimpase]

comment:26

PS. I hear that openmp support on OSX is an especially brutal form of hell, so it has got to be tested there too.

[kiwifb]

comment:27

Replying to @dimpase:

PS. I hear that openmp support on OSX is an especially brutal form of hell, so it has got to be tested there too.

If I remember correctly it is just turned off as soon as darwin is detected.

[dimpase]

comment:28

For one, if openmp is not installed then -fopenmp should not even be mentioned in the calls, but it does get mentioned.

[kiwifb]

comment:29

We unfortunately deal with a meta build driven by a master makefile. No configuration is done apart possibly some flags passed to make.

[kiwifb]

comment:30

The good stuff to look at configuration and what they do SuiteSparse_config/SuiteSparse_config.mk and to see what happen with the flags you want make ${my_flags} config I actually run it at the beginning of the build so it is logged. Setting CFOPENMP="" should kill openmp.

[dimpase]

comment:31

I'd rather see a way to use system's SuiteSparse (as in #26286) rather than spending time on cutting custom tarballs...

[kiwifb]

comment:32

A lot of system split suitesparse so we would have to check the components individually. In any case I would not use spkg-src to make little tarballs. I would deal with it the same way gfortran and gcc are dealt with but on a bigger scale.

[dimpase]

comment:33

Replying to @kiwifb:

Yes it is unusual. And very strange. There is some control for openmp in the main makefile I think so it has to be turned off there (or autodetected properly there). But not finding stuff in /usr/local/lib is highly unusual - and it appears this is not your compiler's fault but ld's fault. Is it GNU ld, some kind of BSD ld or llvm's ld?

$ ld -v
LLD 6.0.1 (FreeBSD 335540-1200005) (compatible with GNU linkers)

[kiwifb]

comment:34

I believe this is llvm's ld then.

[dimpase]

comment:35

Replying to @kiwifb:

I believe this is llvm's ld then.

yep. If it is called directly, it does not know where to look for libraries, so one much pass -L/lib -L/usr/lib -L etc to it.

[kiwifb]

comment:36

Replying to @dimpase:

Replying to @kiwifb:

I believe this is llvm's ld then.

yep. If it is called directly, it does not know where to look for libraries, so one much pass -L/lib -L/usr/lib -L etc to it.

Definitely not called directly (I checked) the compiler is called to do the linking so -fopenmp should set all the flags properly. I am calling that a bug in linking driver of the compiler.

[dimpase]

comment:37

Replying to @kiwifb:

Replying to @dimpase:

Replying to @kiwifb:

I believe this is llvm's ld then.

yep. If it is called directly, it does not know where to look for libraries, so one much pass -L/lib -L/usr/lib -L etc to it.

Definitely not called directly (I checked) the compiler is called to do the linking so -fopenmp should set all the flags properly. I am calling that a bug in linking driver of the compiler.

I think I agree. It's ludicrous that LDFLAGS need to be set for openMP to work. And it appears to be well-known: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222858

[kiwifb]

comment:38

Basically the -fopenmp switch doesn't work. Well this is half assed, it should either work or not work, but not partially work. Obviously it still add -lomp to the linker call.

[dimpase]

comment:39

Replying to @kiwifb:

A lot of system split suitesparse so we would have to check the components individually. In any case I would not use spkg-src to make little tarballs. I would deal with it the same way gfortran and gcc are dealt with but on a bigger scale.

Checking components would be needed anyway, as it's installed as a bunch of separate libraries, whether they come from one blob or many.

Anyhow, send this one to the bots!