search

sagemath / sage

Main repository of SageMath
https://www.sagemath.org
Other
1.44k stars 480 forks source link

homebrew-standard: Fix several problems with library search paths #29607

Closed mkoeppe closed 4 years ago

mkoeppe commented 4 years ago

29562 broke the gfortran build in homebrew-macos-minimal (https://github.com/mkoeppe/sage/runs/624417033):

Checking LIBRARY_PATH variable... contains current directory
configure: error: 
*** LIBRARY_PATH shouldn't contain the current directory when
*** building gcc. Please change the environment variable
*** and run configure again.

29562 also broke flint/singular on homebrew when conflicting versions of libraries are in /usr/local, leading to doctest errors and segfaults such as this one:

File "src/sage/combinat/posets/posets.py", line 3816, in sage.combinat.posets.posets.FinitePoset.coxeter_smith_form
Failed example:
    prod(P.coxeter_smith_form()) == P.coxeter_polynomial()
Exception raised:
    Traceback (most recent call last):
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/.tox/local-homebrew-macos-standard/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 681, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/.tox/local-homebrew-macos-standard/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 1123, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.combinat.posets.posets.FinitePoset.coxeter_smith_form[3]>", line 1, in <module>
        prod(P.coxeter_smith_form()) == P.coxeter_polynomial()
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/.tox/local-homebrew-macos-standard/local/lib/python3.7/site-packages/sage/combinat/posets/posets.py", line 3846, in coxeter_smith_form
        L = sing_m.smith().sage().diagonal()
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/.tox/local-homebrew-macos-standard/local/lib/python3.7/site-packages/sage/interfaces/interface.py", line 1087, in sage
        return self._sage_(*args, **kwds)
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/.tox/local-homebrew-macos-standard/local/lib/python3.7/site-packages/sage/interfaces/singular.py", line 1949, in _sage_
        typ = self.type()
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/.tox/local-homebrew-macos-standard/local/lib/python3.7/site-packages/sage/interfaces/singular.py", line 2107, in type
        m = p.match(self.parent().eval("type(%s)"%self.name()))
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/.tox/local-homebrew-macos-standard/local/lib/python3.7/site-packages/sage/interfaces/singular.py", line 659, in eval
        raise SingularError('Singular error:\n%s'%s)
    sage.interfaces.singular.SingularError: Singular error:
           GCDL[j,i]=-STDM[j,j]/G;<<
    Segment fault/Bus error occurred (r:0)
    please inform the authors
    trying to restart...
    Singular : signal 11 (v: 4112):
    current line:>
**********************************************************************

Depends on #29562

CC: @jhpalmieri @dimpase @slel @kiwifb @isuruf @antonio-rojas @mwageringel @thierry-FreeBSD @vbraun

Component: build

Author: Matthias Koeppe

Branch/Commit: fcfbc10

Reviewer: John Palmieri, Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/29607

mkoeppe commented 4 years ago

Description changed:

--- 
+++ 
@@ -1 +1,10 @@
+This broke `homebrew-macos-minimal`, `gfortran`:

+```
+Checking LIBRARY_PATH variable... contains current directory
+configure: error: 
+*** LIBRARY_PATH shouldn't contain the current directory when
+*** building gcc. Please change the environment variable
+*** and run configure again.
+```
+
mkoeppe commented 4 years ago

Description changed:

--- 
+++ 
@@ -1,4 +1,4 @@
-This broke `homebrew-macos-minimal`, `gfortran`:
+#29562  broke `homebrew-macos-minimal`, `gfortran`:

Checking LIBRARY_PATH variable... contains current directory

mkoeppe commented 4 years ago

Dependencies: #29562

mkoeppe commented 4 years ago

Description changed:

--- 
+++ 
@@ -1,4 +1,4 @@
-#29562  broke `homebrew-macos-minimal`, `gfortran`:
+#29562  broke the `gfortran` build in `homebrew-macos-minimal` (https://github.com/mkoeppe/sage/runs/624417033):

Checking LIBRARY_PATH variable... contains current directory

mkoeppe commented 4 years ago

Branch: u/mkoeppe/another_fixup_forhomebrew_build_env_setting_cpathlibrary_path

mkoeppe commented 4 years ago

Author: Matthias Koeppe

mkoeppe commented 4 years ago

New commits:

e74c6c2.homebrew-build-env: Set CPATH, LIBRARY_PATH instead of CPPFLAGS, LDFLAGS
3371465Merge branch 't/29562/_homebrew_build_env__set_cpath_instead_of_cppflags' into t/29607/another_fixup_for__homebrew_build_env_setting_cpath__library_path
c1002b6.homebrew-build-env: Do not accidentally put the current directory in CPATH, LIBRARY_PATH
mkoeppe commented 4 years ago

Commit: c1002b6

mkoeppe commented 4 years ago
comment:6

Tests at https://github.com/mkoeppe/sage/actions/runs/89771836

jhpalmieri commented 4 years ago
comment:7

This works for me, with or without gfortran. With just #29562, gfortran does indeed fail. This branch doesn't apply cleanly on top of #29562, I think, but I don't know if that matters.

jhpalmieri commented 4 years ago

Reviewer: John Palmieri

mkoeppe commented 4 years ago
comment:8

Thanks! Actually this is fast-forward from #29562

mkoeppe commented 4 years ago
comment:10

Seems to need more work, to fix a problem with singular on homebrew.

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 4 years ago

Changed commit from c1002b6 to f593e66

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 4 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

f593e66src/bin/sage-env: Add to beginning, not end of CPATH, LIBRARY_PATH
mkoeppe commented 4 years ago

Description changed:

--- 
+++ 
@@ -8,3 +8,6 @@
 *** and run configure again.

+ +#29562 also broke singular on homebrew, leading to doctest errors and segfaults +

mkoeppe commented 4 years ago

Description changed:

--- 
+++ 
@@ -9,5 +9,37 @@

-#29562 also broke singular on homebrew, leading to doctest errors and segfaults +#29562 also broke singular on homebrew, leading to doctest errors and segfaults such as this one:

+``` +File "src/sage/combinat/posets/posets.py", line 3816, in sage.combinat.posets.posets.FinitePoset.coxeter_smith_form +Failed example:

mkoeppe commented 4 years ago
comment:15

cc'ing some of our singular experts

jhpalmieri commented 4 years ago
comment:16

But Sage doesn't use the system's Singular. I don't understand.

mkoeppe commented 4 years ago
comment:17

No, but some dependencies of Singular are coming from the system.

jhpalmieri commented 4 years ago
comment:18

Do you know which packages? I haven't seen these doctest failures, but I could install more homebrew packages to try to reproduce them.

mkoeppe commented 4 years ago
comment:19

The error can be reproduced in the standard homebrew environment set up by tox -e local-homebrew-macos-standard -- ptest.

I am now trying with EXTRA_CONFIGURE_ARGS="--with-mp=mpir" tox -e local-homebrew-macos-standard -- ptest to disable use of homebrew's GMP.

Any help with debugging this would be very welcome!

mkoeppe commented 4 years ago
comment:20

The failures can be seen, for example, in https://github.com/mkoeppe/sage/runs/628776278

kiwifb commented 4 years ago
comment:21

Looks like failures in the pexpect interface to singular if I read those logs correctly and I haven't missed anything. This make readline the number one suspect amongst singular's dependencies (ntl, gmp, cddlib, flint).

jhpalmieri commented 4 years ago
comment:22

I'll try tox -e local-homebrew-macos-standard -- ptest. In my current setup (in which I don't see any failures), the hints at the end of running ./configure only suggest installing cmake, which implies (?) that I have everything in homebrew-macos-standard already. (The comment in tox.ini says "Install all known system packages equivalent to standard packages that have spkg-configure.m4", but it must actually require both an spkg-configure.m4 and distros/homebrew.txt, right?)

mkoeppe commented 4 years ago
comment:23

Replying to @jhpalmieri:

I'll try tox -e local-homebrew-macos-standard -- ptest. In my current setup (in which I don't see any failures), the hints at the end of running ./configure only suggest installing cmake, which implies (?) that I have everything in homebrew-macos-standard already.

Note that tox installs its own copy of homebrew, independent of the one in /usr/local.

(The comment in tox.ini says "Install all known system packages equivalent to standard packages that have spkg-configure.m4", but it must actually require both an spkg-configure.m4 and distros/homebrew.txt, right?)

Right, with this clarification it describes correctly the behavior for the docker-.... environments. For local-homebrew, however, the implementation cheats a little bit (fixing this is one of the tasks of #29146). See line 323 of tox.ini to see how the system package list is determined:

    homebrew-standard:   bash -c 'PACKAGES=`sed "s/#.*//;" build/pkgs/*/distros/homebrew.txt`; {env:HOMEBREW}/bin/brew install $PACKAGES; {env:HOMEBREW}/bin/brew upgrade $PACKAGES'

So it really just installs every package listed in any homebrew.txt

mkoeppe commented 4 years ago
comment:24

Replying to @kiwifb:

Looks like failures in the pexpect interface to singular if I read those logs correctly and I haven't missed anything. This make readline the number one suspect amongst singular's dependencies (ntl, gmp, cddlib, flint).

Thanks, yes, that should be the top suspect!

mkoeppe commented 4 years ago
comment:25

Replying to @jhpalmieri:

In my current setup (in which I don't see any failures), the hints at the end of running ./configure only suggest installing cmake, which implies (?) that I have everything in homebrew-macos-standard already.

Yes, I think so

mkoeppe commented 4 years ago
comment:26

Replying to @mkoeppe:

The error can be reproduced in the standard homebrew environment set up by tox -e local-homebrew-macos-standard -- ptest.

The error also shows in local-homebrew-macos-python2-standard.

On the other hand: local-homebrew-macos-python3_pythonorg-minimal (https://github.com/mkoeppe/sage/runs/628776310) and local-homebrew-macos-python2-minimal are clean.

mkoeppe commented 4 years ago
comment:27

Replying to @mkoeppe:

I am now trying with EXTRA_CONFIGURE_ARGS="--with-mp=mpir" tox -e local-homebrew-macos-standard -- ptest to disable use of homebrew's GMP.

This build does not have the failure

mkoeppe commented 4 years ago
comment:28

EXTRA_CONFIGURE_ARGS="--without-system-readline" tox -e local-homebrew-macos-standard -- ptest has the failure

kiwifb commented 4 years ago
comment:29

Hum, I guess that could have predicted on the fact that only singular displayed the failure. Still, there is something fishy happening in sage/interface/singular.py. Incidentally, after scanning the singular libraries, it appears that mpfr is a direct dependency of singular - not just through flint. I certainly need to update the dependency list in Gentoo.

mkoeppe commented 4 years ago
comment:30

I'm testing EXTRA_CONFIGURE_ARGS="--without-system-mpfr" tox -e local-homebrew-macos-standard -- ptest at the moment

mkoeppe commented 4 years ago
comment:31

Replying to @mkoeppe:

I'm testing EXTRA_CONFIGURE_ARGS="--without-system-mpfr" tox -e local-homebrew-macos-standard -- ptest at the moment

... which has the failure.

mkoeppe commented 4 years ago
comment:32

Here's a self-contained interaction with Singular in the bad build:

$ Singular 
                     SINGULAR                                 /  Development
 A Computer Algebra System for Polynomial Computations       /   version 4.1.1
                                                           0<
 by: W. Decker, G.-M. Greuel, G. Pfister, H. Schoenemann     \   Feb 2018
FB Mathematik der Universitaet, D-67653 Kaiserslautern        \
> ring sage7=0,(x,y),dp;
> def sage8=9*y^8 - 9*x^2*y^7 - 18*x^3*y^6 - 18*x^5*y^6 +9*x^6*y^4 + 18*x^7*y^5 + 36*x^8*y^4 + 9*x^10*y^4 - 18*x^11*y^2 -9*x^12*y^3 - 18*x^13*y^2 + 9*x^16;
> def sage9=factorize(sage8);
Singular : signal 11 (v: 4112):
current line:>>def sage9=factorize(sage8);<<
Segment fault/Bus error occurred (r:1588272268)
please inform the authors
trying to restart...
mkoeppe commented 4 years ago
comment:33

Self-contained command line test:

echo "ring sage7=0,(x,y),dp; factorize(9*y^8 - 9*x^2*y^7 - 18*x^3*y^6 - 18*x^5*y^6 +9*x^6*y^4 + 18*x^7*y^5 + 36*x^8*y^4 + 9*x^10*y^4 - 18*x^11*y^2 -9*x^12*y^3 - 18*x^13*y^2 + 9*x^16); " | .tox/local-homebrew-macos-standard/local/bin/Singular | fgrep -q '[2]:'
jhpalmieri commented 4 years ago
comment:35

I can't even get Sage to build with tox -e local-homebrew-macos-standard -- ptest with this branch or without: the Sage library fails with

In file included from build/cythonized/sage/tests/stl_vector.cpp:663:
./sage/libs/ntl/ntlwrap.h:5:10: fatal error: 'NTL/ZZ.h' file not found
#include <NTL/ZZ.h>

Sage builds fine and passes tests (except for those at #29493) with make ptestlong.

mkoeppe commented 4 years ago
comment:36

Could you attach a log of the tox run?

mkoeppe commented 4 years ago
comment:37

I also see a testsuite segfault for FLINT

  [flint-2.5.2.p5]   make[3]: *** [../build/nmod_poly_mat/test/t-concat_vertical_RUN] Segmentation fault: 11
mkoeppe commented 4 years ago
comment:38

Replying to @jhpalmieri:

I can't even get Sage to build with tox -e local-homebrew-macos-standard -- ptest with this branch or without: the Sage library fails with

In file included from build/cythonized/sage/tests/stl_vector.cpp:663:
./sage/libs/ntl/ntlwrap.h:5:10: fatal error: 'NTL/ZZ.h' file not found
#include <NTL/ZZ.h>

Sage builds fine and passes tests (except for those at #29493) with make ptestlong.

Do you happen to have NTL from homebrew installed in /usr/local (brew info ntl)?

jhpalmieri commented 4 years ago
comment:39

Replying to @mkoeppe:

Replying to @jhpalmieri:

I can't even get Sage to build with tox -e local-homebrew-macos-standard -- ptest with this branch or without: the Sage library fails with

In file included from build/cythonized/sage/tests/stl_vector.cpp:663:
./sage/libs/ntl/ntlwrap.h:5:10: fatal error: 'NTL/ZZ.h' file not found
#include <NTL/ZZ.h>

Sage builds fine and passes tests (except for those at #29493) with make ptestlong.

Do you happen to have NTL from homebrew installed in /usr/local (brew info ntl)?

Yes, I do.

Could you attach a log of the tox run?

Which file do you want? One of the ones in logs?

-rw-r--r--    1 palmieri  staff  275651 Apr 30 12:17 config.log
-rw-r--r--    1 palmieri  staff  103409 Apr 30 13:17 install.log
-rw-r--r--    1 palmieri  staff     914 Apr 30 10:26 local-homebrew-macos-standard-0.log
-rw-r--r--    1 palmieri  staff     268 Apr 30 10:26 local-homebrew-macos-standard-1.log
drwxr-xr-x  153 palmieri  staff    4896 Apr 30 13:08 pkgs

Or should I just uninstall homebrew's ntl?

mkoeppe commented 4 years ago
comment:40

Just the terminal output if you still have it, otherwise install.log please

mkoeppe commented 4 years ago
comment:41

There is a problem with /usr/local interaction. FLINT's configure script is severely broken (that's not exactly news, but...): it puts /usr/local/include/ in as explicit defaults and stuff there can override what we have in $SAGE_LOCAL.

isuruf commented 4 years ago
comment:42

For flint, adding --with-gmp=$SAGE_LOCAL is fine even if gmp is not there and is found from the system.

jhpalmieri commented 4 years ago
comment:43

Attachment: tox.log

I cut-and-pasted this from the terminal window.

mkoeppe commented 4 years ago
comment:44

Replying to @isuruf:

For flint, adding --with-gmp=$SAGE_LOCAL is fine even if gmp is not there and is found from the system.

Yes, this is the fix that I am using now

mkoeppe commented 4 years ago
comment:45

Replying to @jhpalmieri:

Replying to @mkoeppe:

Replying to @jhpalmieri:

I can't even get Sage to build with tox -e local-homebrew-macos-standard -- ptest with this branch or without: the Sage library fails with

In file included from build/cythonized/sage/tests/stl_vector.cpp:663:
./sage/libs/ntl/ntlwrap.h:5:10: fatal error: 'NTL/ZZ.h' file not found
#include <NTL/ZZ.h>

Sage builds fine and passes tests (except for those at #29493) with make ptestlong.

Do you happen to have NTL from homebrew installed in /usr/local (brew info ntl)?

From your log:

[sagelib-9.1.rc2]   g++ -bundle -undefined dynamic_lookup -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk -L/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/tox/sage-9.1.rc2/.tox/local-homebrew-macos-standard/local/lib -Wl,-rpath,/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/tox/sage-9.1.rc2/.tox/local-homebrew-macos-standard/local/lib build/temp.macosx-10.15-x86_64-3.7/build/cythonized/sage/symbolic/substitution_map.o -L/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/tox/sage-9.1.rc2/.tox/local-homebrew-macos-standard/local/lib -L/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/tox/sage-9.1.rc2/.tox/local-homebrew-macos-standard/homebrew/lib -L/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/tox/sage-9.1.rc2/.tox/local-homebrew-macos-standard/homebrew/opt/openssl@1.1/lib -L/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/tox/sage-9.1.rc2/.tox/local-homebrew-macos-standard/homebrew/opt/sqlite/lib -lgmp -lpynac -lstdc++ -o build/lib.macosx-10.15-x86_64-3.7/sage/symbolic/substitution_map.cpython-37m-darwin.so
2400      [sagelib-9.1.rc2]   In file included from build/cythonized/sage/tests/stl_vector.cpp:663:
2401      [sagelib-9.1.rc2]   ./sage/libs/ntl/ntlwrap.h:5:10: fatal error: 'NTL/ZZ.h' file not found
2402      [sagelib-9.1.rc2]   #include <NTL/ZZ.h>
2403      [sagelib-9.1.rc2]            ^~~~~~~~~~

This is an interesting variant of the failure mode of #29000, #29448 - where ./configure finds a library but then the package needing it does not. In this case, Homebrew's NTL in /usr/local is found by ./configure. But we compile Python modules using g++ -bundle -undefined dynamic_lookup -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk, where the -isysroot disables the use of /usr/local.

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 4 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

0b5d19ebuild/pkgs/flint/spkg-install.in: Always pass --with-{gmp,mpfr,ntl}=$SAGE_LOCAL to get rid of /usr/local
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 4 years ago

Changed commit from f593e66 to 0b5d19e

mkoeppe commented 4 years ago
comment:47

Replying to @mkoeppe:

Replying to @isuruf:

For flint, adding --with-gmp=$SAGE_LOCAL is fine even if gmp is not there and is found from the system.

Yes, this is the fix that I am using now

... in the commit above.

Probably the spkg-install of other packages with the same build system as FLINT need to be changed in the same way.