dimpase commented 5 years ago

In FreeBSD 12 the long-standing incompleteness of its libm has been helped. This simplifies a lot, compared to #22679.

We will not try to work around the toolchain problem related to gfortran/libgcc_s. Instead, we just add a line at the end of /etc/libmap.conf as follows:

libgcc_s.so.1 /usr/local/lib/gcc7/libgcc_s.so.1

Also, we create the following symbolic links:

$ ls -l ~/bin
total 0
lrwxr-xr-x  1 dima  dima  20 Apr 27 12:49 make -> /usr/local/bin/gmake
lrwxr-xr-x  1 dima  dima  21 Apr 27 13:46 patch -> /usr/local/bin/gpatch

and use the standard shell, /bin/sh. In ~/.profile we put

export LDFLAGS="-L/usr/local/lib"
export CFLAGS="-I/usr/local/include"
export CXXFLAGS=$CFLAGS
export CPPFLAGS=$CFLAGS
export FCFLAGS=$CFLAGS
export FFLAGS=$CFLAGS

to account for clang not looking into /usr/local by default.

Things to fix:

26250 (fixes to rubiks)
27764 (a fix to flint)
26253 (update of psutils)
yasm check is broken on FreeBSD (it's always YES!)
26602 (cephes removed)
26287 (update to gc, to the FreeBSD's version)
26288 (update to matplotlib, fixes compilation errors)
26190 (update to latte_int---which is an optional package)
26733 (update to openblas)
26315 (update to giac, to be able to use clang 6.0.1)
27262 (needed to link sagelib to libs from /usr/local/lib)

There is some hope that the libgcc_s.so.1 problem will finally be [fixed in FreeBSD https://lists.freebsd.org/pipermail/freebsd-ports/2019-April/115972.html]

The strategy which would pay off to make Sage an "easy" FreeBSD port is to add/enhance as many needed by Sage external libraries as separate ports into FreeBSD, so that Sage does not need to vendor this. To this end, #27330 is very relevant ticket.

A first foray into this is enhancing FreeBSD's port of flint2 to be linkable with NTL (something that is a requirement for flint2 in Sage), see here. The necessary work on the flint2 side has been done in the meantime, see https://github.com/wbhart/flint2/pull/543 - so now it's only the FreeBSD part that needs to be done.

Few obvious other candidates for a port in FreeBSD are gf2x and ecm.

Depends on #26253 Depends on #27764

CC: @mkoeppe

Component: porting: BSD

Author: Dima Pasechnik, Li-Wen Hsu

Issue created by migration from https://trac.sagemath.org/ticket/26249

dimpase commented 5 years ago

Description changed:

--- 
+++ 
@@ -7,3 +7,9 @@
 libgcc_s.so.1 /usr/local/lib/gcc7/libgcc_s.so.1

+Things to fix: + + #26250 + + ... +

dimpase commented 5 years ago

Description changed:

--- 
+++ 
@@ -11,5 +11,5 @@

 * #26250

-* ...
+* #26252

dimpase commented 5 years ago

Description changed:

--- 
+++ 
@@ -9,7 +9,7 @@

 Things to fix:

-* #26250
+* #26250 (fixes to rubiks)
+* #26252 (fixes to flint)
+* #26253 (update of psutils)

-* #26252
-

dimpase commented 5 years ago

Description changed:

--- 
+++ 
@@ -12,4 +12,4 @@
 * #26250 (fixes to rubiks)
 * #26252 (fixes to flint)
 * #26253 (update of psutils)
-
+* yasm check is broken on FreeBSD (it's always YES!)

dimpase commented 5 years ago

Description changed:

--- 
+++ 
@@ -13,3 +13,4 @@
 * #26252 (fixes to flint)
 * #26253 (update of psutils)
 * yasm check is broken on FreeBSD (it's always YES!)
+* cephes to be removed

lwhsu commented 5 years ago

Author: Dima Pasechnik, Li-Wen Hsu

dimpase commented 5 years ago

Description changed:

--- 
+++ 
@@ -14,3 +14,4 @@
 * #26253 (update of psutils)
 * yasm check is broken on FreeBSD (it's always YES!)
 * cephes to be removed
+* #26287 (update to gc, to the FreeBSD's version)

dimpase commented 5 years ago

Description changed:

--- 
+++ 
@@ -15,3 +15,4 @@
 * yasm check is broken on FreeBSD (it's always YES!)
 * cephes to be removed
 * #26287 (update to gc, to the FreeBSD's version)
+* #26288 (update to matplotlib, fixes compilation errors)

dimpase commented 5 years ago

Description changed:

--- 
+++ 
@@ -16,3 +16,4 @@
 * cephes to be removed
 * #26287 (update to gc, to the FreeBSD's version)
 * #26288 (update to matplotlib, fixes compilation errors)
+* #26190 (update to latte_int---which is an optional package)

dimpase commented 5 years ago

Description changed:

--- 
+++ 
@@ -13,7 +13,9 @@
 * #26252 (fixes to flint)
 * #26253 (update of psutils)
 * yasm check is broken on FreeBSD (it's always YES!)
-* cephes to be removed
+* #26602 (cephes removed)
 * #26287 (update to gc, to the FreeBSD's version)
 * #26288 (update to matplotlib, fixes compilation errors)
 * #26190 (update to latte_int---which is an optional package)
+* #26733 (update to openblas)
+* #26315 (update to giac, to be able to use clang 6.0.1)

dimpase commented 5 years ago

comment:11

This scipy problem is still there. (while this works with the system's python2) It's a matter of comparing the patches used there with what we do in Sage. Perhaps they disable multithreading...

embray commented 5 years ago

comment:12

I'm currently overextended as always and don't have any deep personal motivation to work on FreeBSD support, but I would definitely like to have it if we can so if there's anything I can help with here I'll make a FreeBSD 12 VM so I can have it to play around on.

dimpase commented 5 years ago

comment:13

Replying to @embray:

I'm currently overextended as always and don't have any deep personal motivation to work on FreeBSD support, but I would definitely like to have it if we can so if there's anything. I can help with here I'll make a FreeBSD 12 VM so I can have it to play around on.

I can give you access to the FreeBSD 12.0 VM I am running on GCE, this would be easier, I guess. (12.0 is still in beta/RC cycle, so installation is not straightforward, you basically need to build it from source).

dimpase commented 5 years ago

comment:14

currently testing what we have here (with Sage 8.6) on the released FreeBSD 12.0

dimpase commented 5 years ago

comment:15

the next issue we hit is Python3 on Sage 8.7.beta3: https://bugs.python.org/issue6299 (pyexpat module does not build)

dimpase commented 5 years ago

Dependencies: #27254

dimpase commented 5 years ago

comment:16

Replying to @dimpase:

the next issue we hit is Python3 on Sage 8.7.beta3: https://bugs.python.org/issue6299 (pyexpat module does not build)

for the time being, working around this by not installing setuptools and pip for py3

dimpase commented 5 years ago

Changed dependencies from #27254 to #27254, #/26253

dimpase commented 5 years ago

Description changed:

--- 
+++ 
@@ -19,3 +19,4 @@
 * #26190 (update to latte_int---which is an optional package)
 * #26733 (update to openblas)
 * #26315 (update to giac, to be able to use clang 6.0.1)
+* #27262 (needed to link `sagelib` to libs from /usr/local/lib)

dimpase commented 5 years ago

Changed dependencies from #27254, #/26253 to #27254, #26253, #27262

dimpase commented 5 years ago

Description changed:

--- 
+++ 
@@ -20,3 +20,12 @@
 * #26733 (update to openblas)
 * #26315 (update to giac, to be able to use clang 6.0.1)
 * #27262 (needed to link `sagelib` to libs from /usr/local/lib)
+
+---
+There is some hope that the `libgcc_s.so.1` problem will finally be [fixed in FreeBSD https://lists.freebsd.org/pipermail/freebsd-ports/2019-April/115972.html]
+
+The strategy which would pay off to make Sage an "easy" FreeBSD port is to add/enhance  as many needed by Sage external libraries as separate ports into FreeBSD, so that Sage does not need to vendor this. To this end, #27330 is very relevant ticket.
+
+A first foray into this is enhancing FreeBSD's port of flint2 to be linkable with NTL (something that is a requirement for flint2 in Sage), see [here](https://lists.freebsd.org/pipermail/freebsd-ports/2019-February/115504.html). The necessary work on the flint2 side has been done in the meantime, see https://github.com/wbhart/flint2/pull/543 - so now it's only the FreeBSD part that needs to be done.
+
+Few obvious other candidates for a port in FreeBSD are gf2x and ecm.

dimpase commented 5 years ago

Changed dependencies from #27254, #26253, #27262 to #27254, #26253, #27262, #27764

dimpase commented 5 years ago

Description changed:

--- 
+++ 
@@ -10,7 +10,7 @@
 Things to fix:

 * #26250 (fixes to rubiks)
-* #26252 (fixes to flint)
+* #27764 (a fix to flint)
 * #26253 (update of psutils)
 * yasm check is broken on FreeBSD (it's always YES!)
 * #26602 (cephes removed)

embray commented 5 years ago

comment:21

As the Sage-8.8 release milestone is pending, we should delete the sage-8.8 milestone for tickets that are not actively being worked on or that still require significant work to move forward. If you feel that this ticket should be included in the next Sage release at the soonest please set its milestone to the next release milestone (sage-8.9).

dimpase commented 5 years ago

Description changed:

--- 
+++ 
@@ -6,6 +6,26 @@

libgcc_s.so.1 /usr/local/lib/gcc7/libgcc_s.so.1

+Also, we create the following symbolic links:
+
+```
+$ ls -l ~/bin
+total 0
+lrwxr-xr-x  1 dima  dima  20 Apr 27 12:49 make -> /usr/local/bin/gmake
+lrwxr-xr-x  1 dima  dima  21 Apr 27 13:46 patch -> /usr/local/bin/gpatch
+```
+and use the standard shell, `/bin/sh`. In `~/.profile` we put
+
+```
+export LDFLAGS="-L/usr/local/lib"
+export CFLAGS="-I/usr/local/include"
+export CXXFLAGS=$CFLAGS
+export CPPFLAGS=$CFLAGS
+export FCFLAGS=$CFLAGS
+export FFLAGS=$CFLAGS
+```
+to account for clang not looking into `/usr/local` by default.
+

 Things to fix:

dimpase commented 5 years ago

comment:22

to use flang (which is a possibility on FreeBSD at least), one needs

--- a/build/pkgs/numpy/spkg-install
+++ b/build/pkgs/numpy/spkg-install
@@ -18,7 +18,7 @@ export FFLAGS="$FFLAGS -fPIC"
 export FCFLAGS="$FCFLAGS -fPIC"

 export NUMPY_FCONFIG="config_fc --noopt --noarch"
-
+export LDFLAGS="$LDFLAGS -lexecinfo" # needed for flang
 ################################################

 sage-python23 setup.py \

due to it being somewhat underlinked.

dimpase commented 5 years ago

comment:23

As I have to abandon the FreeBSD VM I've been using, I've pushed the current WIP branch to https://github.com/lwhsu/sagemath/commits/wip31082019

TBC on another FreeBSD machine.

thierry-FreeBSD commented 4 years ago

comment:24

A work-in-progress has been submitted to the FreeBSD Phabricator:

https://reviews.freebsd.org/D24195

Its aim is to upgrade the existing port to 9.0.

Do not hesitate to comment or modify!

dimpase commented 4 years ago

comment:25

I think it'd be much better to use the work done on

27330 - you don't even need to build a special Python version, but use one on the system!

(And, of course, use autoconf, don't mess around with generated ./configure)

thierry-FreeBSD commented 4 years ago

comment:26

I used the external packages when this is possible with the released Sage 9.0. This version still builds the bundled Python. I guess that the progress done with #27330 will be available in the next release?

Thanks for your comment, I'll try to use autoconf.

dimpase commented 4 years ago

comment:27

A large part of #27330 is already in 9.0. But yes, I think it's much better to wait for 9.1, or actually start using 9.1.betas (we're near the first release candidate).

IIRC FreeBSD has quite a few packages available, although they aren't always built with pre-reqs needed for Sage, e.g. Flint is not built with NTL support.

thierry-FreeBSD commented 4 years ago

comment:28

Great! I'll try to work on 9.1 ASAP.

You are right about Flint: the ports used Cmake, and the NTL option is not available in CMakeLists.txt (and it is not even enabled by default with configure).

But we can change that, and I just sent a PR to fix it:

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=245085

thierry-FreeBSD commented 4 years ago

comment:29

And I just added a port for Cliquer, from your autotoolized version:

https://www.freshports.org/math/cliquer/

dimpase commented 4 years ago

comment:30

Thanks. I suggest that every Sage library/package mentioned in #27330 be made a FreeBSD port (some of them are), this should be enough to finish this task.

We have plans to turn Sage into a "normal" pip-installable Python library, but this probably won't happen before the end of the year.

thierry-FreeBSD commented 4 years ago

comment:31

I recently also added eclib, m4ri, gf2x, gp2c and the Pari packages (elldata, galdata, galpol, nftables and seadata).

And still going on!

thierry-FreeBSD commented 4 years ago

comment:32

Some news: I have just updated the WIP-port to 9.1.beta9: it is available at https://reviews.freebsd.org/D24195

Every possible packages listed in #27330 are used as external packages, thanks for that!

Unfortunately, there remain some problems, mostly caused by staging (DESTDIR): a lot of libraries are referring to the directory where they have been built: e.g.

Error: 'lib/python3.7/site-packages/PIL/_imagingft.so' is referring to /usr/ports/math/sage/work/stage Error: 'lib/libldl.so.2.2.6' is referring to /usr/ports/math/sage/work/stage

Is there some possibility to let them refer to relative directories? How would you set SAGE_LOCAL?

thierry-FreeBSD commented 4 years ago

comment:33

To be more precise: I have played with SAGE_LOCAL and SAGE_DESTDIR.

The logical choice seemed to set SAGE_LOCAL=$PREFIX and SAGE_DESTDIR=$STAGEDIR, but this is not the case.

thierry-FreeBSD commented 4 years ago

comment:34

Actually, to be more positive, it mostly works!

Looking for the root of the problem, only 29 libraries are not fully staged - but of course, thereafter the other binaries using one of this library become affected.

ATM (with 9.1.beta9), my list of affected libraries is:

libamd.so.2 => /usr/ports/math/sage/work/stage/usr/local/lib/libamd.so.2

libbraiding.so.0 => /usr/ports/math/sage/work/stage/usr/local/lib/libbraiding.so.0

libbrial_groebner.so.3 => /usr/ports/math/sage/work/stage/usr/local/lib/libbrial_groebner.so.3

libbrial.so.3 => /usr/ports/math/sage/work/stage/usr/local/lib/libbrial.so.3

libbtf.so.1 => /usr/ports/math/sage/work/stage/usr/local/lib/libbtf.so.1

libcamd.so.2 => /usr/ports/math/sage/work/stage/usr/local/lib/libcamd.so.2

libccolamd.so.2 => /usr/ports/math/sage/work/stage/usr/local/lib/libccolamd.so.2

libcddgmp.so.0 => /usr/ports/math/sage/work/stage/usr/local/lib/libcddgmp.so.0

libcholmod.so.3 => /usr/ports/math/sage/work/stage/usr/local/lib/libcholmod.so.3

libcolamd.so.2 => /usr/ports/math/sage/work/stage/usr/local/lib/libcolamd.so.2

libecl.so.16.1 => /usr/ports/math/sage/work/stage/usr/local/lib/libecl.so.16.1

libfactory-4.1.1.so => /usr/ports/math/sage/work/stage/usr/local/lib/libfactory-4.1.1.so

libfflas.so.1 => /usr/ports/math/sage/work/stage/usr/local/lib/libfflas.so.1

libffpack.so.1 => /usr/ports/math/sage/work/stage/usr/local/lib/libffpack.so.1

libgap.so.0 => /usr/ports/math/sage/work/stage/usr/local/lib/libgap.so.0

libgc.so.1 => /usr/ports/math/sage/work/stage/usr/local/lib/libgc.so.1

libhomfly.so.0 => /usr/ports/math/sage/work/stage/usr/local/lib/libhomfly.so.0

libiml.so.1 => /usr/ports/math/sage/work/stage/usr/local/lib/libiml.so.1

liblinbox.so.0 => /usr/ports/math/sage/work/stage/usr/local/lib/liblinbox.so.0

liblrcalc.so.1 => /usr/ports/math/sage/work/stage/usr/local/lib/liblrcalc.so.1

libomalloc-0.9.6.so => /usr/ports/math/sage/work/stage/usr/local/lib/libomalloc-0.9.6.so

libpolys-4.1.1.so => /usr/ports/math/sage/work/stage/usr/local/lib/libpolys-4.1.1.so

libppl.so.14 => /usr/ports/math/sage/work/stage/usr/local/lib/libppl.so.14

libpynac.so.18 => /usr/ports/math/sage/work/stage/usr/local/lib/libpynac.so.18

libsingular_resources-4.1.1.so => /usr/ports/math/sage/work/stage/usr/local/lib/libsingular_resources-4.1.1.so

libSingular-4.1.1.so => /usr/ports/math/sage/work/stage/usr/local/lib/libSingular-4.1.1.so

libsuitesparseconfig.so.5 => /usr/ports/math/sage/work/stage/usr/local/lib/libsuitesparseconfig.so.5

libumfpack.so.5 => /usr/ports/math/sage/work/stage/usr/local/lib/libumfpack.so.5

libzn_poly-0.9.so => /usr/ports/math/sage/work/stage/usr/local/lib/libzn_poly-0.9.so

Some of them like libgc, brial, or fflas-ffpack should be fixed RSN (thanks to #27330 ).

I guess that the most problematic case is libsuitesparseconfig.so, which is widely used.

dimpase commented 4 years ago

comment:35

I cc Matthias, who has been doing a lot of work on improving Sage's installation. He'd be able to understand the issue better than me, hopefully.

thierry-FreeBSD commented 4 years ago

comment:36

The problem is documented in build/pkgs/suitesparse/patches/03-buildflags.patch:

"Note that -L$(INSTALL_LIB) cannot be removed from LDFLAGS unless the package is broken into components. It is necessary as the components are progressively installed in INSTALL_LIB one after the other."

It has been reported in #28832 and fixed for OS X with -install_name, but this does not fix it for FreeBSD. I'll have to try other patches - unless suiteparse could be used from a system package? It is not yet listed in #27330 but would be awesome!

dimpase commented 4 years ago

comment:37

Now we have #29502 for suitesparse - not done yet.

thierry-FreeBSD commented 4 years ago

comment:38

Some news: after #29502 for suitesparse and #29024 for singular, only 13 libraries (+ of course the modules built around them!) cause problems with staging: libbraiding libbrial_groebner libbrial libecl libfflas libffpack libgap libgc libhomfly liblinbox libppl libpynac libzn_poly-0.

Among them, all are listed in #27330 (with or without a ticket), but libpynac. We do not even have pynac in the FreeBSD ports tree (but we have math/GiNaC) => it will be my next target.

thierry-FreeBSD commented 4 years ago

comment:39

After giac (#29541) and pynac (#29542), I tried to use a system package for ECL (lang/ecl) and Maxima (math/maxima), but without success for the moment: see #29617. To be tried again when ECL will be upgraded (#22191)!

Meanwhile, our ports must be patched:

lang/ecl: add a symbolic link for $PREFIX/lib/ecl
math/maxima: this one is more tricky, because now ECL is not even supported and must be added, and to get a useful package we need to replace the default from SBCL to ECL (or create a slave port?)

They are not submitted yet, but if someone want to work on #29617 the patches are available at

dimpase commented 4 years ago

comment:40

Replying to @thierry-FreeBSD:

After giac (#29541) and pynac (#29542), I tried to use a system package for ECL (lang/ecl) and Maxima (math/maxima), but without success for the moment: see #29617. To be tried again when ECL will be upgraded (#22191)!

Meanwhile, our ports must be patched:

lang/ecl: add a symbolic link for $PREFIX/lib/ecl

math/maxima: this one is more tricky, because now ECL is not even supported and must be added, and to get a useful package we need to replace the default from SBCL to ECL (or create a slave port?)

there are many free Common Lisp implementations, I suppose there can be lang/cl as a "main" package with a number of slaves, perhaps SBCL as a standard one.

mkoeppe commented 4 years ago

comment:41

I'd recommend to add system package information for FreeBSD. See https://github.com/sagemath/sage-prod/issues/29354 where this is done for Slackware (except for the bits that use docker)

thierry-FreeBSD commented 4 years ago

comment:42

Well, every package listed in #27330 have a corresponding package from the FreeBSD ports tree and a spkg-configure.m4 to use it!

Next step: try to solve the problem with bad staging (non-respect of $DESTDIR) for the libraries built by Sage.

mkoeppe commented 4 years ago

comment:43

Replying to @thierry-FreeBSD:

Well, every package listed in #27330 have a corresponding package from the FreeBSD ports tree and a spkg-configure.m4 to use it!

Yes, and the trick is to put the names of these packages in a freebsd.txt file to record the correspondence of spkg to FreeBSD port.

thierry-FreeBSD commented 4 years ago

comment:44

OK, I'll do it!

thierry-FreeBSD commented 4 years ago

comment:45

Replying to @thierry-FreeBSD:

OK, I'll do it!

Done in #29653.

mkoeppe commented 4 years ago

comment:46

Next, if you're serious about FreeBSD support, I would recommend setting up testing infrastructure with our tox.ini and virtualbox (https://www.freebsd.org/doc/handbook/virtualization-guest-virtualbox.html) or a similar solution so that Sage developers and our CI infrastructure have a chance of testing on that platform.

sagemath / sage

port Sage to FreeBSD 12 #26249

26250 (fixes to rubiks)

27764 (a fix to flint)

26253 (update of psutils)

26602 (cephes removed)

26287 (update to gc, to the FreeBSD's version)

26288 (update to matplotlib, fixes compilation errors)

26190 (update to latte_int---which is an optional package)

26733 (update to openblas)

26315 (update to giac, to be able to use clang 6.0.1)

27262 (needed to link `sagelib` to libs from /usr/local/lib)

27330 - you don't even need to build a special Python version, but use one on the system!

sagemath / sage

port Sage to FreeBSD 12 #26249

26250 (fixes to rubiks)

27764 (a fix to flint)

26253 (update of psutils)

26602 (cephes removed)

26287 (update to gc, to the FreeBSD's version)

26288 (update to matplotlib, fixes compilation errors)

26190 (update to latte_int---which is an optional package)

26733 (update to openblas)

26315 (update to giac, to be able to use clang 6.0.1)

27262 (needed to link sagelib to libs from /usr/local/lib)

27330 - you don't even need to build a special Python version, but use one on the system!

27262 (needed to link `sagelib` to libs from /usr/local/lib)