port Sage to FreeBSD 11

dimpase commented 7 years ago

Please see #26249 for the followup (for FreeBSD 12).

FreeBSD as of version 10 uses clang by default, and is closer to OSX in other ways too, so it would be a good free testing ground for the use of clang with Sage.

preparing tools:

install gcc6 and create fake gfortran by making a link to gfortran6
make sure as is new enough: pkg install binutils will install as version 2.28, and you need it to come first in PATH (i.e. make sure that /usr/local/bin comes before /usr/bin).
remove stale /lib/libgcc_s.so.1 and point it to the right one in /usr/local/lib/gcc6/ (perhaps there is a better way...)
install gmake, gawk, gsed, and create fake make and sed to point to gmake and gsed
make sure /bin/sh is actually bash
install pkgconf (FreeBSD's replacement of pkg-config)
install wget (needed for AtlasRep GAP package).
install icu (needed for R)

building Sage itself

starting from a clean git repo:

export CFLAGS="-fPIC"
export CXXFLAGS="-fPIC"
export LDFLAGS="-L/usr/local/lib -Wl,--add-needed -lm"
./configure CC=cc CXX=c++  #(this is clang 3.8.0 a.t.m.)
MAKE="make -j7" make build
MAKE="make -j4" make # building docs is very memory-hungry

HISTORICAL: Sage itself:

hack configure.ac to make freebsd allowed platform
deal with cephes' complex.h conflicting with numpy - #22786
gf2x: only builds with SAGE_FAT_BINARY on
upgrade sqlite to version 3.17 - otherwise there are compiling problems with editline (see #22687)
flint: remove "-ascii -pedantic" flags from CFLAGS in configure
libraries (libgap, cddlib, etc) need -fPIC for CFLAGS (so we build with CC="cc -fPIC -pipe" ; (-pipe is probably not really needed)) - see #22784
but gap cannot be built with CC="cc -fPIC -pipe", as it does something weird in ./configure, setting BASECC to be cc, and generating broken Makefile (with -fPIC and -pipe on two separate lines); so build gap with CC=cc
giac needs CFLAGS="-I/usr/local/include" to find libintl.h and LDFLAGS="-L/usr/local/lib -lintl" to link.
either provide /bin/bash or fix this properly (#22689)
Sage builds and runs, modulo linking issues with flint and arb, which can be fixed ad hoc by hand.
deal with cephes' libm not pulling system's libm, unless some arcane linker flags are used (see comments below)
old freebsd workaround for libgd removed in #22785
...

Depends on #12426 Depends on #23700

CC: @sagetrac-stephen @kiwifb @jdemeyer @jpflori

Component: porting: BSD

Author: Dima Pasechnik

Branch/Commit: u/dimpase/fbsd-support @ 4aaeeb4

Reviewer: Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/22679

dimpase commented 7 years ago

Description changed:

--- 
+++ 
@@ -5,7 +5,7 @@
 * install gcc6 and create fake gfortran by making a link to gcc6
 * install gmake, gawk, gsed, and create fake make and sed to point to gmake and gsed
 * make sure /bin/sh is actually bash
-* hack configure.ac to make freebsd applowed platform
+* hack configure.ac to make freebsd allowed platform
 * get rid of cephes - it causes linking problems with libm, and is probably not needs any more anyway
 * figure out what to do with Sage's libz and libbz - they cause linking problems (needless to say, 
 * gf2x only builds with SAGE_FAT_BINARY on

dimpase commented 7 years ago

Description changed:

--- 
+++ 
@@ -9,4 +9,5 @@
 * get rid of cephes - it causes linking problems with libm, and is probably not needs any more anyway
 * figure out what to do with Sage's libz and libbz - they cause linking problems (needless to say, 
 * gf2x only builds with SAGE_FAT_BINARY on
+* upgrade sqlite to version 3.17 - otherwise there are compiling problems with editline
 * ...

dimpase commented 7 years ago

Description changed:

--- 
+++ 
@@ -5,9 +5,13 @@
 * install gcc6 and create fake gfortran by making a link to gcc6
 * install gmake, gawk, gsed, and create fake make and sed to point to gmake and gsed
 * make sure /bin/sh is actually bash
+* install pkgconf (FreeBSD's replacement of pkg-config)
 * hack configure.ac to make freebsd allowed platform
 * get rid of cephes - it causes linking problems with libm, and is probably not needs any more anyway
 * figure out what to do with Sage's libz and libbz - they cause linking problems (needless to say, 
-* gf2x only builds with SAGE_FAT_BINARY on
+* gf2x: only builds with SAGE_FAT_BINARY on
 * upgrade sqlite to version 3.17 - otherwise there are compiling problems with editline
+* flint: remove "-ascii -pedantic" flags from CFLAGS in configure
+* libgap: needs -fPIC for CFLAGS
 * ...
+

dimpase commented 7 years ago

Description changed:

--- 
+++ 
@@ -1,14 +1,19 @@
 FreeBSD as of version 10 uses clang by default, and is closer to OSX in other ways too, so it would be a good free testing ground for the use of clang with Sage. 

 Few details I figured out.
+
+\# preparing tools:

 * install gcc6 and create fake gfortran by making a link to gcc6
 * install gmake, gawk, gsed, and create fake make and sed to point to gmake and gsed
 * make sure /bin/sh is actually bash
 * install pkgconf (FreeBSD's replacement of pkg-config)
+
+\# Sage itself:
+
 * hack configure.ac to make freebsd allowed platform
 * get rid of cephes - it causes linking problems with libm, and is probably not needs any more anyway
-* figure out what to do with Sage's libz and libbz - they cause linking problems (needless to say, 
+* figure out what to do with Sage's libz and libbz - they cause linking problems (needless to say, system versions are perfectly good) 
 * gf2x: only builds with SAGE_FAT_BINARY on
 * upgrade sqlite to version 3.17 - otherwise there are compiling problems with editline
 * flint: remove "-ascii -pedantic" flags from CFLAGS in configure

dimpase commented 7 years ago

Description changed:

--- 
+++ 
@@ -2,14 +2,15 @@

 Few details I figured out.

-\# preparing tools:
+
+## preparing tools:

 * install gcc6 and create fake gfortran by making a link to gcc6
 * install gmake, gawk, gsed, and create fake make and sed to point to gmake and gsed
 * make sure /bin/sh is actually bash
 * install pkgconf (FreeBSD's replacement of pkg-config)

-\# Sage itself:
+## Sage itself:

 * hack configure.ac to make freebsd allowed platform
 * get rid of cephes - it causes linking problems with libm, and is probably not needs any more anyway
@@ -17,6 +18,6 @@
 * gf2x: only builds with SAGE_FAT_BINARY on
 * upgrade sqlite to version 3.17 - otherwise there are compiling problems with editline
 * flint: remove "-ascii -pedantic" flags from CFLAGS in configure
-* libgap: needs -fPIC for CFLAGS
+* libraries (libgap, cddlib, etc) need -fPIC for CFLAGS (so we build with CC="cc -fPIC -pipe" ; (-pipe is probably not really needed))
 * ...

dimpase commented 7 years ago

comment:6

cf my comments at https://github.com/numpy/numpy/issues/8530

for the need for /lib/libgcc_s.so.1 to be replaced.

Basically, gcc/gfortran and clang's toolchains are slightly incompatible; while workarounds are in place for OSX, linking on freebsd makes it more or less mandatory.

dimpase commented 7 years ago

Description changed:

--- 
+++ 
@@ -5,7 +5,8 @@

 ## preparing tools:

-* install gcc6 and create fake gfortran by making a link to gcc6
+* install gcc6 and create fake gfortran by making a link to gfortran6
+* remove stale /lib/libgcc_s.so.1 and point it to the right one in /usr/local/lib/gcc6/
 * install gmake, gawk, gsed, and create fake make and sed to point to gmake and gsed
 * make sure /bin/sh is actually bash
 * install pkgconf (FreeBSD's replacement of pkg-config)

dimpase commented 7 years ago

Description changed:

--- 
+++ 
@@ -20,5 +20,6 @@
 * upgrade sqlite to version 3.17 - otherwise there are compiling problems with editline
 * flint: remove "-ascii -pedantic" flags from CFLAGS in configure
 * libraries (libgap, cddlib, etc) need -fPIC for CFLAGS (so we build with CC="cc -fPIC -pipe" ; (-pipe is probably not really needed))
+* giac needs `CFLAGS="-I/usr/local/include"` to find `libintl.h`.
 * ...

dimpase commented 7 years ago

comment:9

one of internal giac tests fail (harmlessly, I hope):

< -sqrt(6)*I*sqrt(sqrt(13)+3),sqrt(6)*I*sqrt(sqrt(13)+3),-sqrt(6)*sqrt(sqrt(13)-3),sqrt(6)*sqrt(sqrt(13)-3),0,
---
> sqrt(6)*I*sqrt(sqrt(13)+3),-sqrt(6)*I*sqrt(sqrt(13)+3),-sqrt(6)*sqrt(sqrt(13)-3),sqrt(6)*sqrt(sqrt(13)-3),0,
FAIL: chk_fhan16

dimpase commented 7 years ago

Description changed:

--- 
+++ 
@@ -20,6 +20,7 @@
 * upgrade sqlite to version 3.17 - otherwise there are compiling problems with editline
 * flint: remove "-ascii -pedantic" flags from CFLAGS in configure
 * libraries (libgap, cddlib, etc) need -fPIC for CFLAGS (so we build with CC="cc -fPIC -pipe" ; (-pipe is probably not really needed))
-* giac needs `CFLAGS="-I/usr/local/include"` to find `libintl.h`.
+* giac needs `CFLAGS="-I/usr/local/include"` to find `libintl.h`
+and `LDFLAGS="-L/usr/local/lib -lintl"` to link.
 * ...

dimpase commented 7 years ago

Description changed:

--- 
+++ 
@@ -17,10 +17,9 @@
 * get rid of cephes - it causes linking problems with libm, and is probably not needs any more anyway
 * figure out what to do with Sage's libz and libbz - they cause linking problems (needless to say, system versions are perfectly good) 
 * gf2x: only builds with SAGE_FAT_BINARY on
-* upgrade sqlite to version 3.17 - otherwise there are compiling problems with editline
+* upgrade sqlite to version 3.17 - otherwise there are compiling problems with editline (see #22687)
 * flint: remove "-ascii -pedantic" flags from CFLAGS in configure
 * libraries (libgap, cddlib, etc) need -fPIC for CFLAGS (so we build with CC="cc -fPIC -pipe" ; (-pipe is probably not really needed))
-* giac needs `CFLAGS="-I/usr/local/include"` to find `libintl.h`
-and `LDFLAGS="-L/usr/local/lib -lintl"` to link.
+* giac needs `CFLAGS="-I/usr/local/include"` to find `libintl.h` and `LDFLAGS="-L/usr/local/lib -lintl"` to link.
 * ...

dimpase commented 7 years ago

Author: Dima Pasechnik

dimpase commented 7 years ago

Description changed:

--- 
+++ 
@@ -21,5 +21,6 @@
 * flint: remove "-ascii -pedantic" flags from CFLAGS in configure
 * libraries (libgap, cddlib, etc) need -fPIC for CFLAGS (so we build with CC="cc -fPIC -pipe" ; (-pipe is probably not really needed))
 * giac needs `CFLAGS="-I/usr/local/include"` to find `libintl.h` and `LDFLAGS="-L/usr/local/lib -lintl"` to link.
+* either provide /bin/bash or fix this properly (#22689)
 * ...

dimpase commented 7 years ago

Dependencies: #22687

dimpase commented 7 years ago

New commits:

`a44a934`	`updating sqlite to version 3.17`
`a32b0c7`	`let Sage accept freebsd`
`3178fbc`	`remove -ascii -pedantic flags from configure`
`4691d92`	`allow clang++ and libc++ combo on freebsd`
`90234a3`	`using libintl from /usr/local/`

dimpase commented 7 years ago

Commit: 90234a3

dimpase commented 7 years ago

Branch: u/dimpase/freebsd_experimental

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 7 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

df76ac2 temporary fix to mke building on freebsd work

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 7 years ago

Changed commit from 90234a3 to df76ac2

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 7 years ago

Changed commit from df76ac2 to f5a9b80

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 7 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

f5a9b80 disable cephes installation on freebsd version >= 10

dimpase commented 7 years ago

Description changed:

--- 
+++ 
@@ -20,6 +20,10 @@
 * upgrade sqlite to version 3.17 - otherwise there are compiling problems with editline (see #22687)
 * flint: remove "-ascii -pedantic" flags from CFLAGS in configure
 * libraries (libgap, cddlib, etc) need -fPIC for CFLAGS (so we build with CC="cc -fPIC -pipe" ; (-pipe is probably not really needed))
+* but gap cannot be built with CC="cc -fPIC -pipe", as it
+does something weird in ./configure, setting BASECC to be cc,
+and generating broken Makefile (with -fPIC and -pipe on two
+separate lines); so build gap with CC=cc 
 * giac needs `CFLAGS="-I/usr/local/include"` to find `libintl.h` and `LDFLAGS="-L/usr/local/lib -lintl"` to link.
 * either provide /bin/bash or fix this properly (#22689)
 * ...

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 7 years ago

Changed commit from f5a9b80 to 9b622a8

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 7 years ago

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

`516f029`	`Upgraded to R 3.3.3`
`cbe9134`	`Merge branch 'public/r311' of git://trac.sagemath.org/sage into finalr`
`f029f66`	`This simple patch removes an overkill check in R's configure.`
`c8a0b1b`	`Merge branch 'public/r311' of trac.sagemath.org:sage into sqliteupdate`
`d0d5ca7`	`update fplll/fpylll to 5.1.0/0.2.4dev`
`9f702f3`	`Merge branch 'u/malb/22643-fplll-upgrade' of trac.sagemath.org:sage into sqliteupdate`
`1201132`	`LONG_LONG_MAX is an obsolete gnu-ism`
`276e658`	`R on freebsd needs this for libcurl`
`0c0c603`	`Write custom create_extension() for Cython`
`9b622a8`	`Merge branch 'u/jdemeyer/use_create_extension___from_cython' of trac.sagemath.org:sage into sqliteupdate`

dimpase commented 7 years ago

Description changed:

--- 
+++ 
@@ -26,5 +26,6 @@
 separate lines); so build gap with CC=cc 
 * giac needs `CFLAGS="-I/usr/local/include"` to find `libintl.h` and `LDFLAGS="-L/usr/local/lib -lintl"` to link.
 * either provide /bin/bash or fix this properly (#22689)
-* ...
+* Sage builds and runs, modulo linking issues with flint and arb,
+which can be fixed ad hoc by hand.

dimpase commented 7 years ago

Changed dependencies from #22687 to #22687, #22643, #22554

dimpase commented 7 years ago

comment:17

builds and runs, modulo ad hoc fixing arb and flint linking issue:

$ ln -sf $SAGE_LOCAL/lib/libarb.so.1.1.1 $SAGE_LOCAL/lib/libarb.so.1
$ ln -sf $SAGE_LOCAL/lib/libflint.so.13.5.2 $SAGE_LOCAL/lib/libflint.so.13

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 7 years ago

Changed commit from 9b622a8 to a391ad6

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 7 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

a391ad6 cephes still is needed for cpow/clog

dimpase commented 7 years ago

comment:19

without cephes, missing cpow() (shows up during building docs). And this is how things are on official FreeBSD.

Thus reverting f5a9b80. After thus, make completes.

Running tests now.

New commits:

a391ad6 cephes still is needed for cpow/clog

dimpase commented 7 years ago

Changed dependencies from #22687, #22643, #22554 to #22687, #22643, #22554, #22692

dimpase commented 7 years ago

comment:22

Question to linking experts:

I am not sure whether letting cephes create libm.so rather than something like libm_complex.so is a good idea. At least I do not see how one can in such a configuration link an executable that needs things from the "normal" libm as well as from cephes---in particular as Sage has moved away from using things like LD_LIBRARY_PATH to using -rpath to link. E.g. consider

#include <math.h>
#include <complex.h>
#include <stdio.h>
int main()
{
        double e=exp2(2.0);
        double complex c=clog(2.0);
        printf("\n e=%f\n", e);
        printf("\n c=%f + i%f\n", creal(c), cimag(c));
        return 0;
}

With two different libm's, I don't see a way to create a working executable (with clang). However if I rename cephes' libm to libm_complex then (from SAGE_ROOT) the result of

cc -I./local/include t.c -lm -L./local/lib -lm_complex -Wl,-rpath=./local/lib

works just fine.

kiwifb commented 7 years ago

comment:23

Replying to @dimpase:

Question to linking experts:

I am not sure whether letting cephes create libm.so rather than something like libm_complex.so is a good idea. At least I do not see how one can in such a configuration link an executable that needs things from the "normal" libm as well as from cephes---in particular as Sage has moved away from using things like LD_LIBRARY_PATH to using -rpath to link. E.g. consider
#include <math.h>
#include <complex.h>
#include <stdio.h>
int main()
{
        double e=exp2(2.0);
        double complex c=clog(2.0);
        printf("\n e=%f\n", e);
        printf("\n c=%f + i%f\n", creal(c), cimag(c));
        return 0;
}
With two different libm's, I don't see a way to create a working executable (with clang). However if I rename cephes' libm to libm_complex then (from SAGE_ROOT) the result of
cc -I./local/include t.c -lm -L./local/lib -lm_complex -Wl,-rpath=./local/lib
works just fine.

I am not entirely sure because I think freeBSD doesn't use sonames. But linking with two shared libraries of the same name is at best awkward. The linker can take -lm several times and several paths with -L, not a problem. The problem is at runtime. Let's see an elf shared object on linux, say openblas:

fbissey@moonloop ~ $ readelf -d /usr/lib64/libopenblas_threads_haswellp-r0.2.15.so

Dynamic section at offset 0x3e3b60 contains 27 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000e (SONAME)             Library soname: [libopenblas_threads.so.0]
 0x000000000000000c (INIT)               0x2ab38
....

The linker effectively create a table of libraries to be loaded at runtime. It also can include runtime path to search first before the normal order from ld.so.conf. But you see that if two libraries have strictly the same name the first one to be found will be the only one used.

If you link a library the linker may not complain if it cannot resolve some symbols because it cannot combine the two libm.so. Undefined symbols are legal in a library. It is not in an executable. I am not quite sure in that case if the linker can even see the two different set of symbols at the same time. In any case you will still have the problem at run time.

Apart from different name the only way out is to have one of the two being static instead of dynamic and then you pass it to the linker as an object with its full path (not just -L$path -lm but $path/libm.a).

dimpase commented 7 years ago

comment:24

Replying to @kiwifb:

If you link a library the linker may not complain if it cannot resolve some symbols because it cannot combine the two libm.so. Undefined symbols are legal in a library. It is not in an executable. I am not quite sure in that case if the linker can even see the two different set of symbols at the same time. In any case you will still have the problem at run time.

There is no /lib/libm.so, only /lib/libm.so.5.

I think the idea at the time of old Sage port to FreeBSD (#9543) was that $SAGE_LOCAL/lib/libm.so serves as a replacement for system's libm. But it seems that even though the former (renamed) is linked to the latter,

$ ldd local/lib/libm_complex.so 
local/lib/libm_complex.so:
        libc.so.7 => /lib/libc.so.7 (0x800823000)
        libm.so.5 => /lib/libm.so.5 (0x80120c000)

this still leads to ugly linker errors complaining about absent exp2. More specifically, if I remove -lm from the invocation above I get

$ cc -I./local/include t.c -L./local/lib -lm_complex -Wl,-rpath=./local/lib

/usr/bin/ld: undefined reference to symbol `exp2@@FBSD_1.0' (try adding -lm)
//lib/libm.so.5: could not read symbols: Bad value

(the latter is puzzling - it knows where to look for exp2, but bails out...)

Apart from different name the only way out is to have one of the two being static instead of dynamic and then you pass it to the linker as an object with its full path (not just -L$path -lm but $path/libm.a).

this is not really better than creating a separate dynamic library, IMHO.

kiwifb commented 7 years ago

comment:25

Is /lib/libm.so.5 a real library? What does running file on it says?

dimpase commented 7 years ago

comment:26

Replying to @kiwifb:

Is /lib/libm.so.5 a real library? What does running file on it says?

$ file /lib/libm.so.5 
/lib/libm.so.5: ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, stripped

kiwifb commented 7 years ago

comment:27

OK that leaves us with the next stupid question: are you compiling in 32bits? Finally can you move -lm after -lm_complex?

dimpase commented 7 years ago

comment:28

Replying to @kiwifb:

OK that leaves us with the next stupid question: are you compiling in 32bits? Finally can you move -lm after -lm_complex?

it's 64-bit, and yes, I can build and run the result of

$ cc -I./local/include t.c -L./local/lib -lm_complex -lm -Wl,-rpath=./local/lib

just fine.

Certainly, having a new library to add, instead of providing a replacement for libm, is a pain, although a limited to few packages one.

kiwifb commented 7 years ago

comment:29

--as-needed is probably your default behavior which means order is important. I suspect you are using the "gold" linker instead of the "bfd" (classic) one [don't remember how to check for that]. Short of passing -Wl,--no-as-needed I don't know how to make the order irrelevant.

jpflori commented 7 years ago

comment:30

Just a reminder that https://svnweb.freebsd.org/ports/head/math/sage/ used to be maintained. Maybe some stuff can be pulled from there.

kcrisman commented 7 years ago

comment:31

See also #12858, #14651, #14392 and in general this report. I'm sure username stephen would be happy to help but we didn't really take much stuff in terms of patches lately so perhaps that is why it was no longer maintained.

dimpase commented 7 years ago

comment:32

Replying to @jpflori:

Just a reminder that https://svnweb.freebsd.org/ports/head/math/sage/ used to be maintained. Maybe some stuff can be pulled from there.

they have there (as well as in #14491) an ld wrapper that uses --copy-dt-needed-entries option to recursively search libraries. And indeed the following (with the "classical" GNU buntils ld) gives working a.out.

$ cc -c -I./local/include t.c
$ /usr/local/bin/ld --eh-frame-hdr -dynamic-linker /libexec/ld-elf.so.1 -L./local/lib --copy-dt-needed-entries -rpath=./local/lib /usr/lib/crt1.o /usr/lib/crti.o t.o -lc -lm_complex /usr/lib/crtn.o

But this option has a different name with the default linker, as far as testing and manuals say.

/usr/bin/ld: unrecognized option '--copy-dt-needed-entries'

The reason is that the default linker used by clang is

$ /usr/bin/ld -v
GNU ld 2.17.50 [FreeBSD] 2007-07-03

The corresponding option is called --add-needed. (It's possible to find this by calling /usr/bin/ld --help.) So the right command (giving working a.out) to pull system's libm with the custom (renamed to libm_complex) libm is

cc -I./local/include -L./local/lib -Wl,--add-needed -Wl,-rpath=./local/lib -lm_complex t.c

jdemeyer commented 7 years ago

comment:33

Replying to @kiwifb:

--as-needed is probably your default behavior which means order is important. I suspect you are using the "gold" linker instead of the "bfd" (classic) one [don't remember how to check for that]. Short of passing -Wl,--no-as-needed I don't know how to make the order irrelevant.

Have a look at library_order_list in src/module_list.py to deal with ordering of libraries.

dimpase commented 7 years ago

Description changed:

--- 
+++ 
@@ -14,8 +14,7 @@
 ## Sage itself:

 * hack configure.ac to make freebsd allowed platform
-* get rid of cephes - it causes linking problems with libm, and is probably not needs any more anyway
-* figure out what to do with Sage's libz and libbz - they cause linking problems (needless to say, system versions are perfectly good) 
+* deal with cephes' complex.h comflicting with numpy
 * gf2x: only builds with SAGE_FAT_BINARY on
 * upgrade sqlite to version 3.17 - otherwise there are compiling problems with editline (see #22687)
 * flint: remove "-ascii -pedantic" flags from CFLAGS in configure
@@ -28,4 +27,5 @@
 * either provide /bin/bash or fix this properly (#22689)
 * Sage builds and runs, modulo linking issues with flint and arb,
 which can be fixed ad hoc by hand.
- 
+* deal with cephes' libm not pulling system's libm, unless some arcane linker flags are used (see comments below)
+* ...

dimpase commented 7 years ago

comment:34

Somewhat ironically, numpy now conflics with cephes (via <complex.h> header), as it adapted some FreeBSD code by stephen for its use in complex arithmetic. :-) Fortunately, it appears to be only the header-level conflict.

dimpase commented 7 years ago

comment:35

running make as follows

LDFLAGS="-L./local/lib -Wl,--add-needed -lm" $MAKE build

allows the build do finish, and pass many tests. Some of docbuilding crashes with rather puzzling (most probably related to multi-processing/threading) errors like

[dochtml] [hyperboli] loading pickled environment... not yet created
[dochtml] 
[dochtml] Internal or unrecoverable error in:
[dochtml] Got signal before environment was installed on our thread
[dochtml]   [2: No such file or directory]
[dochtml] 
[dochtml] ;;; ECL C Backtrace
[dochtml] ;;;    0 ecl_internal_error (0x89a790705) 
[dochtml] ;;;    1 init_unixint (0x89a7b6b70) 
[dochtml] ;;;    2 init_unixint (0x89a7b6912) 
[dochtml] ;;;    3 pthread_sigmask (0x80105779d) 
[dochtml] ;;;    4 pthread_getspecific (0x801056d6f) 
[dochtml] ;;;    5 unknown (0x7ffffffff193) 
[dochtml] ;;;    6 GC_push_all_stacks (0x89ab1b7aa) 
[dochtml] ;;;    7 GC_mark_some (0x89ab10a93) 
[dochtml] ;;;    8 GC_stopped_mark (0x89ab08b3a) 
[dochtml] ;;;    9 GC_try_to_collect_inner (0x89ab08a31) 
[dochtml] ;;;   10 GC_init (0x89ab14593) 
[dochtml] ;;;   11 init_alloc (0x89a7ca9f9) 
[dochtml] ;;;   12 cl_boot (0x89a694a4b) 
[dochtml] ;;;   13 initecl (0x89a218bc0) 
[dochtml] ;;;   14 initecl (0x89a20a9f5) 
[dochtml] ;;;   15 initecl (0x89a208678) 
[dochtml] ;;;   16 _PyImport_LoadDynamicModule (0x8009334fc) 
[dochtml] ;;;   17 PyImport_AppendInittab (0x800931eff) 
[dochtml] ;;;   18 PyImport_AppendInittab (0x8009319b4) 
[dochtml] ;;;   19 PyImport_ImportModuleLevel (0x800930abe) 
[dochtml] ;;;   20 _PyBuiltin_Init (0x80090b187) 
[dochtml] ;;;   21 PyObject_Call (0x800871454) 
[dochtml] ;;;   22 PyEval_EvalFrameEx (0x800915f47) 
[dochtml] ;;;   23 PyEval_EvalCodeEx (0x800910505) 
[dochtml] ;;;   24 PyEval_EvalCode (0x80090fcd6) 
[dochtml] ;;;   25 PyImport_ExecCodeModuleEx (0x80092f521) 
[dochtml] ;;;   26 PyImport_AppendInittab (0x800932598) 
[dochtml] ;;;   27 PyImport_AppendInittab (0x800931eff) 
[dochtml] ;;;   28 PyImport_AppendInittab (0x8009319b4) 
[dochtml] ;;;   29 PyImport_ImportModuleLevel (0x800930abe) 
[dochtml] ;;;   30 _PyBuiltin_Init (0x80090b187) 
[dochtml] ;;;   31 PyEval_EvalFrameEx (0x800917ab0)

building docs with make -j1 works. Running tests now.

dimpase commented 7 years ago

comment:36

"interesting" error message: 10 such errors, leading to segfaults in total in testlong, all related to (un)picklings and libpng. What the meaning of "Application" in these messages? And what is the reason? Missing or wrong -I option for the compiler?

sage -t --long src/sage/coding/binary_code.pyx
    Killed due to segmentation fault
...
sage: from sage.coding.binary_code import * ## line 824 ##
sage: M = Matrix(GF(2), [[1,1,1,1]]) ## line 825 ##
sage: B = BinaryCode(M) ## line 826 ##
sage: loads(dumps(B)) == B ## line 827 ##
libpng warning: Application was compiled with png.h from libpng-1.6.27+apng
libpng warning: Application  is  running with png.c from libpng-1.2.51
GD Error: gd-png: fatal libpng error: Incompatible libpng version in application and library

The rest---for the early days like this---looks not too bad, apart from code related to elliptic curves (probably another well-localised bug, only not quite clear where):

sage/coding/binary_code.pyx  # Killed due to segmentation fault
sage/coding/linear_code.py  # Killed due to segmentation fault
sage/crypto/mq/sr.py  # Killed due to segmentation fault
sage/graphs/generic_graph.py  # Killed due to segmentation fault
sage/graphs/base/graph_backends.pyx  # Killed due to segmentation fault
sage/homology/chain_complex.py  # Killed due to illegal instruction
sage/interfaces/mwrank.py  # 5 doctests failed
sage/interfaces/singular.py  # 2 doctests failed
sage/interfaces/tests.py  # 1 doctest failed
sage/lfunctions/sympow.py  # 4 doctests failed
sage/libs/eclib/interface.py  # 34 doctests failed
sage/libs/eclib/mwrank.pyx  # 3 doctests failed
sage/matrix/matrix_gf2e_dense.pyx  # Killed due to segmentation fault
sage/matrix/matrix_mod2_dense.pyx  # Killed due to segmentation fault
sage/misc/benchmark.py  # 3 doctests failed
sage/numerical/sdp.pyx  # 1 doctest failed
sage/plot/graphics.py  # 1 doctest failed
sage/plot/plot.py  # 1 doctest failed
sage/rings/complex_double.pyx  # 2 doctests failed
sage/rings/padics/CR_template.pxi  # 2 doctests failed
sage/rings/polynomial/multi_polynomial_sequence.py  # Killed due to segmentation fault
sage/rings/polynomial/pbori.pyx  # Killed due to abort
sage/rings/polynomial/polydict.pyx  # 1 doctest failed
sage/rings/polynomial/polynomial_real_mpfr_dense.pyx  # 1 doctest failed
sage/schemes/elliptic_curves/BSD.py  # 23 doctests failed
sage/schemes/elliptic_curves/constructor.py  # 11 doctests failed
sage/schemes/elliptic_curves/ell_egros.py  # 9 doctests failed
sage/schemes/elliptic_curves/ell_generic.py  # 14 doctests failed
sage/schemes/elliptic_curves/ell_number_field.py  # 16 doctests failed
sage/schemes/elliptic_curves/ell_point.py  # 13 doctests failed
sage/schemes/elliptic_curves/ell_rational_field.py  # Timed out
sage/schemes/elliptic_curves/ell_tate_curve.py  # 4 doctests failed
sage/schemes/elliptic_curves/heegner.py  # 48 doctests failed
sage/schemes/elliptic_curves/height.py  # 2 doctests failed
sage/schemes/elliptic_curves/lseries_ell.py  # 2 doctests failed
sage/schemes/elliptic_curves/padic_lseries.py  # 2 doctests failed
sage/schemes/elliptic_curves/padics.py  # 34 doctests failed 
sage/schemes/elliptic_curves/sha_tate.py  # Killed due to segmentation fault
sage/structure/coerce.pyx  # 1 doctest failed
sage/structure/element.pyx  # 1 doctest failed
sage/structure/sage_object.pyx  # Killed due to bus error
sage/tests/benchmark.py  # 1 doctest failed
sage/tests/book_stein_ent.py  # 3 doctests failed
sage/tests/cmdline.py  # 3 doctests failed
doc/en/constructions/algebraic_geometry.rst  # 1 doctest failed
doc/en/developer/coding_basics.rst  # 1 doctest failed
doc/en/developer/coding_in_other.rst  # 1 doctest failed
doc/en/prep/Calculus.rst  # 1 doctest failed
doc/en/thematic_tutorials/explicit_methods_in_number_theory/elliptic_curves.rst  # 11 doctests failed

dimpase commented 7 years ago

Changed dependencies from #22687, #22643, #22554, #22692 to #22687, #22643, #22554, #22692, #22159

dimpase commented 7 years ago

comment:37

libpng-related segfaults are "cured" by #22159 ; this is of course fixing the symptom, not the root cause---as #22159 happens to carry the same version of libpng as the system-wide version.

kiwifb commented 7 years ago

comment:38

It is strange I would have expected runpath to cover you for that. If runpath don't allow to load a library other than the one in the system default location, what's the point? "Application" looks like it is coming from inside libpng itself, so that's an helpful message from png devs.

Do you have readelf available? If so could we have a look at readelf -d local/lib/python/site-packages/sage/coding/binary_code.so when it is in a failing state?

sagemath / sage