Switch from PolyBoRi to BRiAl

[ohanar]

Right now PolyBoRi relies on scons (which is still Python 2 only), and has some Python 2 specific syntax in its bindings. These issues will need to be resolved before Sage can support Python 3.

On this ticket we update to the first version of BRiAl which includes a new (incomplete, but sufficient for our purposes) autotools based system.

To achieve this, we fork upstream polybori, see https://github.com/BRiAl/BRiAl

Tarball: https://github.com/BRiAl/BRiAl/releases/download/0.8.4.3/brial-0.8.4.3.tar.bz2

Depends on #19085

CC: @malb

Component: packages: standard

Author: R. Andrew Ohana

Branch/Commit: 7ff3b09

Reviewer: François Bissey

Issue created by migration from https://trac.sagemath.org/ticket/18437

[kiwifb]

comment:38

Cool to see some move on this! spkg-install still has messages about polybori, it is especially important to update those not too confuse new-comers.

I see that the library name hasn't changed upstream (libpolybori*), that would be something to consider although it is not a show stopper for this.

I notice ipython has been dropped from the list of dependencies, is it on purpose?

[kiwifb]

comment:39

Do I understand well that libgd is looked for if libpng is not there? It is all a bit strange. If m4ri doesn't have support for png I guess have_libpng will be empty and then libgd will be checked. The line

        AC_DEFINE([PBORI_HAVE_GD],[],[has gd png support])

is a bit misleading as if you arrive there, you have libgd support which may or may not have png included in it.

Also why shouldn't the compilation abort if m4ri_png=yes but you cannot find libpng? As far as I can tell m4ri won't use libgd for libpng unless you abuse the configuration options, so that whole structure looks very strange.

I guess on closer look I have an issue with the way m4ri/m4rie/polybory deal with some of their optional dependencies, and that may be something to look into in the future. For starter if m4ri is compiled with png support, the used libpng should be included in output of pkg-config m4ri --libs.

[ohanar]

comment:40

Replying to @kiwifb:

Cool to see some move on this! spkg-install still has messages about polybori, it is especially important to update those not too confuse new-comers.

Sure, I'll do that shortly.

I see that the library name hasn't changed upstream (libpolybori*), that would be something to consider although it is not a show stopper for this.

I didn't want to have to deal with this on the sage side of things at the moment, so I didn't change this. There are a ton of places that the polybori name is used throughout the source code, and it would be a pain to change them, although changing the library name and headers shouldn't be too bad.

I notice ipython has been dropped from the list of dependencies, is it on purpose?

It shouldn't have been a dependency in the first place from what I can tell.

[ohanar]

comment:41

Replying to @kiwifb:

I basically just took the sconstruct logic and moved it into configure.ac (with the appropriate language translation). I don't claim it is good logic.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

5cbf983

rename error messages PolyBoRi -> BRiAl

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from fdff733 to 5cbf983

[kiwifb]

comment:43

I say we save my last comment for future work because I think m4ri configure script needs surgery first.

---

New commits:

5cbf983

rename error messages PolyBoRi -> BRiAl

---

New commits:

5cbf983

rename error messages PolyBoRi -> BRiAl

[kiwifb]

comment:44

I just greped the sage source code for traces of polybori import. Since it now installs in brial unless I missed a mechanism the followings need to be updated

Mirage:sage fbissey$ grep -r "from polybori" *
rings/polynomial/multi_polynomial_sequence.py:        from polybori import gauss_on_polys
rings/polynomial/multi_polynomial_sequence.py:        from polybori.ll import eliminate,ll_encode,ll_red_nf_redsb
rings/polynomial/multi_polynomial_sequence.py:            from polybori.interred import interred as inter_red
rings/polynomial/pbori.pyx:    sage: from polybori import *
rings/polynomial/pbori.pyx:        #from polybori.interpolate import interpolate_smallest_lex
rings/polynomial/pbori.pyx:        sage: from polybori import BooleanMonomialMonoid
rings/polynomial/pbori.pyx:            sage: from polybori import BooleanMonomialMonoid
rings/polynomial/pbori.pyx:            sage: from polybori import BooleanMonomialMonoid
rings/polynomial/pbori.pyx:            sage: from polybori import BooleanMonomialMonoid
rings/polynomial/pbori.pyx:            sage: from polybori import BooleanMonomialMonoid
rings/polynomial/pbori.pyx:            sage: from polybori import BooleanMonomialMonoid
rings/polynomial/pbori.pyx:            sage: from polybori import BooleanMonomialMonoid
rings/polynomial/pbori.pyx:            sage: from polybori import BooleanMonomialMonoid
rings/polynomial/pbori.pyx:            sage: from polybori import BooleanMonomialMonoid
rings/polynomial/pbori.pyx:            sage: from polybori import BooleanMonomialMonoid
rings/polynomial/pbori.pyx:            sage: from polybori import BooleanMonomialMonoid
rings/polynomial/pbori.pyx:            sage: from polybori import BooleanMonomialMonoid
rings/polynomial/pbori.pyx:            sage: from polybori import BooleanMonomialMonoid
rings/polynomial/pbori.pyx:        sage: from polybori import BooleanMonomialMonoid, BooleanMonomial
rings/polynomial/pbori.pyx:            sage: from polybori import BooleanMonomialMonoid, BooleanMonomial
rings/polynomial/pbori.pyx:            sage: from polybori import BooleanMonomialMonoid
rings/polynomial/pbori.pyx:            sage: from polybori import BooleanMonomialMonoid
rings/polynomial/pbori.pyx:            sage: from polybori import BooleanMonomialMonoid
rings/polynomial/pbori.pyx:            sage: from polybori import BooleanMonomialMonoid
rings/polynomial/pbori.pyx:            sage: from polybori import BooleanMonomialMonoid
rings/polynomial/pbori.pyx:            sage: from polybori import BooleanMonomialMonoid
rings/polynomial/pbori.pyx:            sage: from polybori import BooleanMonomialMonoid
rings/polynomial/pbori.pyx:            sage: from polybori import BooleanMonomialMonoid
rings/polynomial/pbori.pyx:            sage: from polybori import BooleanMonomialMonoid
rings/polynomial/pbori.pyx:            sage: from polybori import BooleanMonomialMonoid
rings/polynomial/pbori.pyx:            sage: from polybori import BooleanMonomialMonoid
rings/polynomial/pbori.pyx:            sage: from polybori import BooleSet
rings/polynomial/pbori.pyx:        sage: from polybori import BooleanPolynomial
rings/polynomial/pbori.pyx:        from polybori.parallel import _encode_polynomial
rings/polynomial/pbori.pyx:            sage: from polybori import BooleSet
rings/polynomial/pbori.pyx:        from polybori import red_tail
rings/polynomial/pbori.pyx:            sage: from polybori import *
rings/polynomial/pbori.pyx:            sage: from polybori import *
rings/polynomial/pbori.pyx:            sage: from polybori import *
rings/polynomial/pbori.pyx:            sage: from polybori import *
rings/polynomial/pbori.pyx:        from polybori.gbcore import groebner_basis
rings/polynomial/pbori.pyx:        from polybori import red_tail
rings/polynomial/pbori.pyx:        sage: from polybori import BooleSet
rings/polynomial/pbori.pyx:        sage: from polybori import *
rings/polynomial/pbori.pyx:            sage: from polybori import BooleSet
rings/polynomial/pbori.pyx:            sage: from polybori import BooleSet
rings/polynomial/pbori.pyx:        sage: from polybori import BooleanPolynomialVector
rings/polynomial/pbori.pyx:            sage: from polybori import BooleanPolynomialVector
rings/polynomial/pbori.pyx:            sage: from polybori import BooleanPolynomialVector
rings/polynomial/pbori.pyx:            sage: from polybori import BooleanPolynomialVector
rings/polynomial/pbori.pyx:            sage: from polybori import BooleanPolynomialVector
rings/polynomial/pbori.pyx:            sage: from polybori import BooleanPolynomialVector
rings/polynomial/pbori.pyx:            sage: from polybori import BooleanPolynomialVector
rings/polynomial/pbori.pyx:            sage: from polybori import *
rings/polynomial/pbori.pyx:            sage: from polybori import *
rings/polynomial/pbori.pyx:            sage: from polybori import *
rings/polynomial/pbori.pyx:            sage: from polybori import *
rings/polynomial/pbori.pyx:            sage: from polybori import *
rings/polynomial/pbori.pyx:            sage: from polybori import *
rings/polynomial/pbori.pyx:            sage: from polybori import *
rings/polynomial/pbori.pyx:            sage: from polybori import *
rings/polynomial/pbori.pyx:            sage: from polybori import *
rings/polynomial/pbori.pyx:            sage: from polybori import *
rings/polynomial/pbori.pyx:            sage: from polybori import *
rings/polynomial/pbori.pyx:            sage: from polybori import *
rings/polynomial/pbori.pyx:            sage: from polybori import GroebnerStrategy
rings/polynomial/pbori.pyx:            sage: from polybori import GroebnerStrategy
rings/polynomial/pbori.pyx:            sage: from polybori import *
rings/polynomial/pbori.pyx:            sage: from polybori import *
rings/polynomial/pbori.pyx:            sage: from polybori import *
rings/polynomial/pbori.pyx:            sage: from polybori import GroebnerStrategy
rings/polynomial/pbori.pyx:            sage: from polybori import groebner_basis
rings/polynomial/pbori.pyx:            sage: from polybori import GroebnerStrategy
rings/polynomial/pbori.pyx:            sage: from polybori import BooleanPolynomialVector
rings/polynomial/pbori.pyx:            sage: from polybori import *
rings/polynomial/pbori.pyx:            sage: from polybori import GroebnerStrategy
rings/polynomial/pbori.pyx:            sage: from polybori import groebner_basis
rings/polynomial/pbori.pyx:            sage: from polybori import GroebnerStrategy
rings/polynomial/pbori.pyx:            sage: from polybori import GroebnerStrategy
rings/polynomial/pbori.pyx:            sage: from polybori import GroebnerStrategy
rings/polynomial/pbori.pyx:            sage: from polybori import BooleanMonomialMonoid
rings/polynomial/pbori.pyx:        sage: from polybori import *
rings/polynomial/pbori.pyx:        sage: from polybori import *
rings/polynomial/pbori.pyx:        sage: from polybori import map_every_x_to_x_plus_one
rings/polynomial/pbori.pyx:        sage: from polybori import zeros
rings/polynomial/pbori.pyx:        sage: from polybori.interpolate import *
rings/polynomial/pbori.pyx:        sage: from polybori.interpolate import *
rings/polynomial/pbori.pyx:        sage: from polybori import ll_red_nf_redsb
rings/polynomial/pbori.pyx:        sage: from polybori import ll_red_nf_noredsb
rings/polynomial/pbori.pyx:        sage: from polybori import ll_red_nf_noredsb_single_recursive_call
rings/polynomial/pbori.pyx:        sage: from polybori import if_then_else
rings/polynomial/pbori.pyx:        sage: from polybori import top_index
rings/polynomial/pbori.pyx:        sage: from polybori import *
rings/polynomial/pbori.pyx:        sage: from polybori import substitute_variables
rings/polynomial/pbori.pyx:        sage: from polybori import substitute_variables
rings/polynomial/pbori.pyx:        sage: from polybori import random_set, set_random_seed
rings/polynomial/pbori.pyx:        sage: from polybori import random_set, set_random_seed
rings/polynomial/pbori.pyx:# todo: merge with pickling from polybori.parallel
rings/polynomial/pbori.pyx:# todo: merge with pickling from polybori.parallel
rings/polynomial/pbori.pyx:    from polybori.parallel import _decode_polynomial
rings/polynomial/pbori.pyx:# todo: merge with pickling from polybori.parallel
rings/polynomial/pbori.pyx:            sage: from polybori import BooleConstant
rings/polynomial/pbori.pyx:            sage: from polybori import BooleConstant
rings/polynomial/pbori.pyx:            sage: from polybori import BooleConstant
rings/polynomial/pbori.pyx:            sage: from polybori import BooleConstant
rings/polynomial/pbori.pyx:            sage: from polybori import BooleConstant
rings/polynomial/pbori.pyx:            sage: from polybori import BooleConstant
rings/polynomial/pbori.pyx:            sage: from polybori import BooleConstant
rings/polynomial/pbori.pyx:            sage: from polybori import BooleConstant
rings/polynomial/pbori.pyx:            sage: from polybori import *
rings/polynomial/pbori.pyx:            sage: from polybori import *
rings/polynomial/pbori.pyx:            sage: from polybori import *
rings/polynomial/pbori.pyx:            sage: from polybori import *
rings/polynomial/pbori.pyx:            sage: from polybori import *
rings/polynomial/pbori.pyx:            sage: from polybori import *
rings/polynomial/pbori.pyx:            sage: from polybori import *
rings/polynomial/pbori.pyx:            sage: from polybori import *
sat/converters/__init__.py:from polybori import CNFEncoder as PolyBoRiCNFEncoder

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 5cbf983 to 51a1e98

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

51a1e98

fix imports polybori -> brial

[ohanar]

Description changed:

--- 
+++ 
@@ -4,4 +4,4 @@

 To achieve this, we fork upstream `polybori`, see [https://github.com/BRiAl/BRiAl](https://github.com/BRiAl/BRiAl)

-**Tarball**: [https://github.com/BRiAl/BRiAl/releases/download/0.8.4/brial-0.8.4.tar.bz2](https://github.com/BRiAl/BRiAl/releases/download/0.8.4/brial-0.8.4.tar.bz2)
+**Tarball**: [https://github.com/BRiAl/BRiAl/releases/download/0.8.4.1/brial-0.8.4.1.tar.bz2](https://github.com/BRiAl/BRiAl/releases/download/0.8.4.1/brial-0.8.4.1.tar.bz2)

[ohanar]

New commits:

51a1e98

fix imports polybori -> brial

[kiwifb]

comment:47

One last thing. I think we should implement a removal of old polybori install like in the old polybori spkg and while we are at it cleaning of a previous brial install in case of updating or re-install. We should do it before the call to configure.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 51a1e98 to bf3c56f

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

bf3c56f

brial: clean out old polybori and brial installations before installing

[kiwifb]

comment:49

I think it is ready for testing. One more thing that I just spotted

              libraries=['polybori', 'polybori_groebner', 'm4ri', 'gd', 'png12'],

shouldn't include gd we have removed the dependency in polybori and a check in sage-on-gentoo (all compiled with -Wl,-as-needed) shows that it is not used in pbori.so, there is only one instance of a direct linking to libgd in the whole sage, I'll check where it is coming from.

[kiwifb]

comment:50

matrix_mod2_dense is where there is a direct linking to gd and it is explicit in module_list.py. The other 2 instances sage.modules.vector_mod2_dense and sage.rings.polynomial.pbori (whose libraries's line was the object of my previous comment) are probably linked to the presence of m4ri and can be removed. Actually the presence in matrix_mod2_dense is curious and should be further investigated but that's not this ticket.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

7aadcf2

remove unused library gd from pbori

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from bf3c56f to 7aadcf2

[kiwifb]

comment:52

Ok just tested a build from scratch with the branch. The python bindings are being installed in SAGE_LOCAL/lib64/python2.7/site-packages that lib64 is a bit of a problem. We'll have to dig in the configuration script as libpolybori* is installed correctly under SAGE_LOCAL/lib.

[kiwifb]

comment:53

Adding

AC_SUBST(pythondir)

in configure.ac (yes in lower case see http://www.gnu.org/software/automake/manual/html_node/Python.html) seem to have solved the problem. That or my autotools did a better job. I am successfully building the doc now, which is where things fell apart the first time.

[kiwifb]

comment:54

Amusing, polybori was the last dependency pulling scons. Consequently my build from scratch doesn't include scons and the associated tests are failing. If we have an associated ticket for scons demotion to optional or removal it will depend on this one and vice-versa.

sage -t --long --warn-long 116.1 src/sage/tests/cmdline.py
**********************************************************************
File "src/sage/tests/cmdline.py", line 578, in sage.tests.cmdline.test_executable
Failed example:
    out.find("SCons") >= 0
Expected:
    True
Got:
    False
**********************************************************************
File "src/sage/tests/cmdline.py", line 580, in sage.tests.cmdline.test_executable
Failed example:
    err
Expected:
    ''
Got:
    'Traceback (most recent call last):\n  File "/usr/lib/python-exec/python2.7/scons", line 188, in <module>\n    import SCons.Script\nImportError: No module named SCons.Script\n'
**********************************************************************
File "src/sage/tests/cmdline.py", line 582, in sage.tests.cmdline.test_executable
Failed example:
    ret
Expected:
    0
Got:
    1
**********************************************************************

[kiwifb]

comment:55

OK I have failing tests because cnf.py is not installed. It is the only one missing, I am guessing this is because it is not listed in pyroot/Makefile.am as I see it on github. However it is listed in the tarball you are linking from github. How did this happen? I cannot see a commit touching pyroot/Makefile.am other than the one moving it in pyroot.

[kiwifb]

comment:56

And it wasn't included in the tarball for 0.8.4.

[ohanar]

comment:57

Replying to @kiwifb:

OK I have failing tests because cnf.py is not installed. It is the only one missing, I am guessing this is because it is not listed in pyroot/Makefile.am as I see it on github. However it is listed in the tarball you are linking from github. How did this happen? I cannot see a commit touching pyroot/Makefile.am other than the one moving it in pyroot.

Yes, I fixed that for 0.8.4.1, look at the tag on github. I probably had a staging commit that didn't include cnf.py, and somehow that is on the master branch.

[kiwifb]

comment:58

OK see it under the tag but of course when I worked on the lib/lib64 I was working from master.

[ohanar]

comment:59

scons was already demoted to optional, but since we still had standard packages depending on it, it was in practice standard so these doctest errors weren't caught.

I fixed master with a force push (to reflect the correct commit). If you want to send a pull request fixing the lib64/lib issue, please do. I don't see it on OS X (obviously) and I don't have access to a decent linux box at the moment either.

[kiwifb]

comment:60

Will send the pull request in a few hours when I can properly focus again.

[jdemeyer]

comment:61

Replying to @kiwifb:

sage -t --long --warn-long 116.1 src/sage/tests/cmdline.py
**********************************************************************
File "src/sage/tests/cmdline.py", line 578, in sage.tests.cmdline.test_executable
Failed example:
    out.find("SCons") >= 0
Expected:
    True
Got:
    False
**********************************************************************
File "src/sage/tests/cmdline.py", line 580, in sage.tests.cmdline.test_executable
Failed example:
    err
Expected:
    ''
Got:
    'Traceback (most recent call last):\n  File "/usr/lib/python-exec/python2.7/scons", line 188, in <module>\n    import SCons.Script\nImportError: No module named SCons.Script\n'
**********************************************************************
File "src/sage/tests/cmdline.py", line 582, in sage.tests.cmdline.test_executable
Failed example:
    ret
Expected:
    0
Got:
    1
**********************************************************************

positive_review to a new ticket which just removes those 3 tests.

[kiwifb]

comment:62

Replying to @ohanar:

I fixed master with a force push (to reflect the correct commit). If you want to send a pull request fixing the lib64/lib issue, please do. I don't see it on OS X (obviously) and I don't have access to a decent linux box at the moment either.

Done. I may just take care of Jeroen's request - I will take it literally.

[kiwifb]

comment:63

19085 filled and we should depend on it.

[kiwifb]

Dependencies: 19085

[kiwifb]

Changed dependencies from 19085 to #19085

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 7aadcf2 to 2a799b9

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

2a799b9

BRiAl: minor version bump with site-packages fix

[ohanar]

Description changed:

--- 
+++ 
@@ -4,4 +4,4 @@

 To achieve this, we fork upstream `polybori`, see [https://github.com/BRiAl/BRiAl](https://github.com/BRiAl/BRiAl)

-**Tarball**: [https://github.com/BRiAl/BRiAl/releases/download/0.8.4.1/brial-0.8.4.1.tar.bz2](https://github.com/BRiAl/BRiAl/releases/download/0.8.4.1/brial-0.8.4.1.tar.bz2)
+**Tarball**: [https://github.com/BRiAl/BRiAl/releases/download/0.8.4.2/brial-0.8.4.2.tar.bz2](https://github.com/BRiAl/BRiAl/releases/download/0.8.4.2/brial-0.8.4.2.tar.bz2)

[kiwifb]

comment:67

Building from scratch on a clean machine...

[kiwifb]

Reviewer: François Bissey

[kiwifb]

comment:68

I think we are ready to go!

[vbraun]

comment:69

http://build.sagemath.org/release/builders/%20%20slow%20AIMS%20%20%28Debian%207%2032%20bit%29%20incremental/builds/71/steps/compile/logs/stdio

[ohanar]

comment:70

I'm guessing this issue comes down to not including the SIMD cflags of m4ri. I can't seem to figure out how to extract these from <m4ri/m4ri_config.h> using autotools. @kiwifb any ideas?

[kiwifb]

comment:71

If that's the case, yes I do have an idea. But I'll look at the log more in depth first. It looks like missing sse2 instruction but the build is i486 so it shouldn't have them either.

[kiwifb]

comment:72

I think I know what to do, expect a pull request shortly.

[malb]

comment:73

Here is what I do in M4RIE

https://bitbucket.org/malb/m4rie/src/HEAD/m4/ax_m4ri_flags.m4?at=master

but I think this should be replaced by pkg-config like this

https://autotools.io/pkgconfig/pkg_check_modules.html

[kiwifb]

comment:74

Funny, I wrote what I called a vile hack but at some point it was looking very very close to that. I just didn't have the step to use cat afterwards. For some reason I didn't want to go that way.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 2a799b9 to 61c0186

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

61c0186

brial: version bump which includes M4RI's SIMD CFLAGS

[ohanar]

Description changed:

--- 
+++ 
@@ -4,4 +4,4 @@

 To achieve this, we fork upstream `polybori`, see [https://github.com/BRiAl/BRiAl](https://github.com/BRiAl/BRiAl)

-**Tarball**: [https://github.com/BRiAl/BRiAl/releases/download/0.8.4.2/brial-0.8.4.2.tar.bz2](https://github.com/BRiAl/BRiAl/releases/download/0.8.4.2/brial-0.8.4.2.tar.bz2)
+**Tarball**: [https://github.com/BRiAl/BRiAl/releases/download/0.8.4.2/brial-0.8.4.3.tar.bz2](https://github.com/BRiAl/BRiAl/releases/download/0.8.4.2/brial-0.8.4.3.tar.bz2)

[ohanar]

comment:76

ok, hopefully now fixed

[jdemeyer]

comment:78

Isn't this a typo?

+ $(INST)/$(BRAIL) \