search

void-linux / void-packages

The Void source packages collection
https://voidlinux.org
Other
2.56k stars 2.13k forks source link

Package request: SageMath #19090

Closed dkwo closed 2 years ago

dkwo commented 4 years ago

https://www.sagemath.org SageMath is a free open-source mathematics software system licensed under the GPL. It builds on top of many existing open-source packages: NumPy, SciPy, matplotlib, Sympy, Maxima, GAP, FLINT, R and many more. Access their combined power through a common, Python-based language or directly via interfaces or wrappers. Mission: Creating a viable free open source alternative to Magma, Maple, Mathematica and Matlab.

leahneukirchen commented 4 years ago

Sage is a terrible mess to compile yourself. I recommend just using the Docker container.

dkwo commented 4 years ago

It is now possible to build Sage on Void. More work is required to package the missing dependencies, and then install Sage (the library) as Python module.

motorto commented 4 years ago

Maybe do a check list of the missing packages to xbps-src , so that people could do them.
I can help !

dkwo commented 4 years ago

Here's a list of desiderata (see here, here and here). This Sage ticket tracks spkg's. It's useful to look at config options that Sage the distribution uses.

leahneukirchen commented 4 years ago

Sage is a great example of what Docker is useful for. *ducks and runs*

dkwo commented 3 years ago

I was previously able to build sage by installing these packages in Void:

CoinMP-devel
R
automake
boost-devel
cblas
cmake
ecl
ecm-devel
freetype-devel
gcc-fortran
gd-devel
gettext-devel
giac-devel
glpk-devel
gmpxx-devel
graphviz
harfbuzz
isl
libcurl-devel
libgda-devel
libgomp-devel
libmpc-devel
m4
mk-configure
mpc
mpir-devel
ninja
openblas-devel
pandoc
pari-devel
pari-elldata
pari-galdata
pari-galpol
pari-seadata
patch
pcre-devel
pcre2-devel
perl-Term-ReadLine-Gnu
pkgconf
ppl-devel
python3-Babel
python3-Pillow
python3-Sphinx
python3-devel
python3-matplotlib
python3-networkx
python3-pkgconfig
python3-scipy
python3-sympy
sqlite-devel
tox
yasm
zeromq-devel

    cblas
    harfbuzz
    libgda-devel
    mpir-devel
    python3-Babel
    python3-Pillow
    python3-Sphinx
    python3-devel
    python3-matplotlib
    python3-networkx
    python3-pkgconfig
    python3-scipy
    python3-sympy
tornaria commented 3 years ago

Note that sagemath includes (almost all) dependencies, and it's easy to compile with a minimal set of system packages:

gcc
make
m4
perl
binutils
git
tar
libgomp-devel

It's also recommended to install at least

gcc-fortran
libressl-devel

The difficult part would be to compile sagemath with as many system packages as possible.

dkwo commented 3 years ago

The difficult part would be to compile sagemath with as many system packages as possible.

Indeed. So that you don't get huge files and build times.

Eloitor commented 3 years ago

I would like to try to package sage, but I'm quite new to void. Where should I start?

dkwo commented 3 years ago

@Eloitor One could try to package some dependencies, see https://github.com/void-linux/void-packages/issues/19090#issuecomment-687717541

tornaria commented 3 years ago

I would like to try to package sage, but I'm quite new to void. Where should I start?

It's a long road, we are slowly packaging dependencies but it will take some time.

It could be useful if you can test and review some pending PR: #29997, #30034, #30035, #30036.

If you merge those 4 PR, then you can compile sage with all of them installed (FlintQS, lcalc-devel, mpfi-devel, givaro-devel, fflas_ffpack, linbox-devel). The more dependencies we have, fewer have to be compiled by sage. Other recently merged dependencies that you may want to install are: gp2c, gf2x-devel, ntl-devel, flintlib-devel, arb-devel, eclib-devel. At this point I can have sage-9.4.rc2 compiled in ~ 25 min with -j8. More important, all those dependencies can now be used for other purposes.

I suggest that you focus on one or two packages and try to get them in good shape, pass QA, and get merged. It will take some time, but in my experience it's not a good idea to make too many half-baked PR at the same time. For example: try to get your lean-community package merged, and try to package m4ri and m4rie (both in one PR since they depend). Learn from that, e.g. how to test your templates in -musl and -i686, and also how to try cross compilation to see if it works, etc.

dkwo commented 3 years ago

@tornaria re:doctesting sage9.4, don't you get sage -t --warn-long 28.3 --random-seed=0 src/sage/symbolic/integration/integral.py # 1 doctest failed if you do make test? I got it a few days ago with rc2

tornaria commented 3 years ago

@tornaria re:doctesting sage9.4, don't you get sage -t --warn-long 28.3 --random-seed=0 src/sage/symbolic/integration/integral.py # 1 doctest failed if you do make test? I got it a few days ago with rc2

Yes, this is expected. I'm using the patch from https://trac.sagemath.org/ticket/32275 (commit dd75684).

dkwo commented 3 years ago

Just a note to self: python3-jupyter_ipywidgets in jupyterlab, for use with sage.

dkwo commented 3 years ago

@tornaria Do you also get this same list with sage 9.4, or is your list shorter :) ?

configure: notice: the following SPKGs did not find equivalent system packages:
brial cddlib cliquer fplll gfan libbraiding libhomfly lrcalc nauty palp planarity rw suitesparse symmetrica sympow tachyon zn_poly
_recommended coxeter3 graphviz igraph libnauty libsemigroups lrslib perl_cpan_polymake_prereq perl_mongodb
tornaria commented 3 years ago

@tornaria Do you also get this same list with sage 9.4, or is your list shorter :) ?

configure: notice: the following SPKGs did not find equivalent system packages:
brial cddlib cliquer fplll gfan libbraiding libhomfly lrcalc nauty palp planarity rw suitesparse symmetrica sympow tachyon zn_poly_recommended coxeter3 graphviz igraph libnauty libsemigroups lrslib perl_cpan_polymake_prereq perl_mongodb

Are you able to compile sage on x86_64-musl ? I tried a few days ago and I the compilation stopped with an error in pyzmq.

dkwo commented 3 years ago

Thanks. I have not tried on musl yet, as that would be my laptop, but I can try one of the coming days.

dkwo commented 3 years ago

@tornaria Same here on musl.

[pyzmq-22.0.3]   gcc build/temp.linux-x86_64-3.9/tmp/timer_createrk_w_nbj.o -o build/temp.linux-x86_64-3.9/a.out
[pyzmq-22.0.3]   error: [Errno 2] No such file or directory: 'a.out'
[pyzmq-22.0.3]   Building wheel for pyzmq (PEP 517): finished with status 'error'
[pyzmq-22.0.3]   ERROR: Failed building wheel for pyzmq
[pyzmq-22.0.3] Failed to build pyzmq
[pyzmq-22.0.3] ERROR: Failed to build one or more wheels
[pyzmq-22.0.3] Exception information:
[pyzmq-22.0.3] Traceback (most recent call last):
[pyzmq-22.0.3]   File "/tmp/build/sage-9.4/local/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 180, in _main
[pyzmq-22.0.3]     status = self.run(options, args)
[pyzmq-22.0.3]   File "/tmp/build/sage-9.4/local/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 204, in wrapper
[pyzmq-22.0.3]     return func(self, options, args)
[pyzmq-22.0.3]   File "/tmp/build/sage-9.4/local/lib/python3.9/site-packages/pip/_internal/commands/wheel.py", line 174, in run
[pyzmq-22.0.3]     raise CommandError(
[pyzmq-22.0.3] pip._internal.exceptions.CommandError: Failed to build one or more wheels
[pyzmq-22.0.3] Removed build tracker: '/tmp/pip-req-tracker-htt11etf'

I hope soon the python3-* packages and maxima will be usable from system, so that we don't have to deal with this issue.

tornaria commented 3 years ago

@tornaria Same here on musl.

Can you try again with the following patch?

diff --git a/build/pkgs/pyzmq/spkg-install.in b/build/pkgs/pyzmq/spkg-install.in
index 0ce404ee5a..b7260c6d27 100644
--- a/build/pkgs/pyzmq/spkg-install.in
+++ b/build/pkgs/pyzmq/spkg-install.in
@@ -1,6 +1,9 @@
 # Since we use environment vars we have to generate setup.cfg

-echo "[build_ext]" > src/setup.cfg
+echo "[global]" > src/setup.cfg
+echo "skip_check_zmq = True" >> src/setup.cfg
+
+echo "[build_ext]" >> src/setup.cfg

 # (I tried putting quotes around $SAGE_LOCAL to allow for spaces in
 # the path---which is never used but is a good habit to support---but

With this for me sage-9.4 compiles and tests ok on musl, except for the following doctest failures:

sage -t --random-seed=0 src/sage/geometry/polyhedron/base.py  # 1 doctest failed
sage -t --random-seed=0 src/sage/lfunctions/sympow.py  # 3 doctests failed
sage -t --random-seed=0 src/sage/symbolic/expression.pyx  # 1 doctest failed
tornaria commented 3 years ago

Just for the record: besides what it is already in PRs, I have working templates for

What's left for compiling sage-9.4 with standard procedure is:

Besides that, the remaining big packages seem to be gap and singular. They are both very useful independently of sage, so packaging those two is probably a worthwhile goal: sooner or later sage will use them from system, as well as maxima.

Moreover, when we get to package sage we probably won't be using the standard compile procedure but compile sage-as-a-python-library on top of all system-installed dependencies, I think that's what arch does and it's the way to move forward; we can't PR a monolith into void.

Eloitor commented 3 years ago

I've build Sage-9.4 using the pending PR. Now I would like to be able to call sage from python, for example using

from sage.all import *

but I get ModuleNotFoundError: No module named 'sage' What can I do?

tornaria commented 3 years ago

I've build Sage-9.4 using the pending PR. Now I would like to be able to call sage from python, for example using

from sage.all import *

but I get ModuleNotFoundError: No module named 'sage' What can I do?

That's not expected to work, because sage installs python packages (in particular sagelib) into its own python venv (should be in SAGE_LOCAL).

I think the simplest way for you is to run sage -python which will run your system python but with sage venv. Then your from sage.all import * should work.

Note that you won't have access to other python packages installed in your system. If you need python packages that are not installed in sage venv, you can use sage -pip to run pip in sage venv and install whatever you need there.

When we get around to make a sage package, we will install sagelib in the system python venv as any other python package in void (in fact, I think sage will be just a python package with lots of dependencies).

Eloitor commented 3 years ago

Ok, thanks for your answer. I'm interested in installing sagelib in the system python venv because then I could use python debuggers... I'm not sure how to debug using sage -python.

tornaria commented 3 years ago

Can you install the python debugger in the sage venv? Try sage -sh for a shell with all environment setup for sage.

On Sat, Sep 4, 2021, 18:32 Eloi Torrents @.***> wrote:

Ok, thanks for your answer. I'm interested in installing sagelib in the system python venv because then I could use python debuggers... I'm not sure how to debug using sage -python.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/void-linux/void-packages/issues/19090#issuecomment-913043924, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMCC27HHVUEXB2SYMDPZ5TUAKF73ANCNFSM4KUOAGWQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

Eloitor commented 3 years ago

Thanks I try to use the debugger in VSCode... In arch it works well with the default python debugger. It uses the command

/usr/bin/env /bin/python /home/eloi/.vscode/extensions/ms-python.python-2021.9.1191016588/pythonFiles/lib/python/debugpy/launcher 37247 -- "example_file.py"

So I tried adding debugpy to sage venv using

sage -sh
pip install debugpy

But I don't know how to continue. I guess I should replace /usr/bin/env with /path/to/sage-9.4/build/bin/sage-venv and /bin/python with path/to/sage-9.4/sage -python. But I'm not very experienced with python environments nor debugging with VSCode...

dkwo commented 3 years ago

Just for the record: besides what it is already in PRs, I have working templates for

* zn_poly

* SuiteSparse

* igraph

* rankwidth

* perl-SVG

If you make PRs for these, would you mind linking back here as well?

Moreover, when we get to package sage we probably won't be using the standard compile procedure but compile sage-as-a-python-library on top of all system-installed dependencies, I think that's what arch does and it's the way to move forward; we can't PR a monolith into void.

Totally agree. Btw, it seems sage itself is (slowly) proceeding towards this, by making sage-the library pip installable.

Ps. Will test your musl patch soon.

dkwo commented 3 years ago

Can you try again with the following patch?

For me the build on musl still fails

 ImportError: Error relocating /tmp/build/sage-9.4/local/lib/python3.9/site-packages/zmq/backend/cython/error.cpython-39-x86_64-linux-gnu.so: zmq_errno: symbol not found
  Building wheel for ipykernel (PEP 517): finished with status 'error'
  ERROR: Failed building wheel for ipykernel

but I'm not using the PR's on my laptop, only packaged ones, so that may be the cause. (Either that, or newer musl I'm using.)

leahneukirchen commented 2 years ago

I merged a few PR, please tell how I can help else to make progress on this.

dkwo commented 2 years ago

@leahneukirchen Great! How about #32601 next?

tornaria commented 2 years ago

Unfortunately python3.10 is not yet supported, we should track https://trac.sagemath.org/ticket/30766

I'm testing the branch in that ticket (commit ​c73bc97), hopefully it will be merged soon.

leahneukirchen commented 2 years ago

We have all these bumps already, no?

dkwo commented 2 years ago

I guess yes (e.g. they're updating gmpy to work with python3.10, which we already have), so we only need those that affect sage the library (or stuff that we haven't packaged yet).

dkwo commented 2 years ago

@leahneukirchen How about #32641 next?

dkwo commented 2 years ago

@tornaria Do you think it's time to do gap? You mentioned you already have something, want to make a wip pr? The remaining packages all seem rather simple to package.

tornaria commented 2 years ago

@tornaria Do you think it's time to do gap? You mentioned you already have something, want to make a wip pr? The remaining packages all seem rather simple to package.

I'll search for my old gap package; I'm not using it, since it will not be used by sage and it seems quite tricky (there are tons of packages for gap). See https://trac.sagemath.org/ticket/29644 and https://trac.sagemath.org/ticket/31761.

It's a nice package to have, independently of sagemath.

At some point we should start figuring out what's the best way to package sagemath: a. using the standard installation procedure, but using as many system packages as possible. This means sage runs its own local directory (and python venv) b. installing it as a python module in the system. For this we need ALL dependencies to be installed and working (including all python dependencies), and presumably passing doctests will be harder because of differences in package versions or compile options. This is how it's done in arch (https://archlinux.org/packages/community/x86_64/sagemath/)

For (a) we don't need (can't use) a system gap. For (b) we require a system gap.

The remaining packages that sagemath will use from system if available are:

I'll try to keep doing 1 or 2 a day from the standard list.

Here there's a template for a meta package "sage-deps" which will install all deps to build sage using all possible system packages.

https://github.com/tornaria/void-packages/blob/sage-deps/srcpkgs/sage-deps/template

leahneukirchen commented 2 years ago

I suggest we start with (a), because we need to do the work for that anyway, and making a template isn't hard, and we can ship something.

Ultimately, (b) would be nice to avoid duplication of all the python modules.

tornaria commented 2 years ago

I suggest we start with (a), because we need to do the work for that anyway, and making a template isn't hard, and we can ship something.

Agreed. Do you think we should install sage into /opt/sage/sage-${version}; this is similar to the old texlive pkg. Since sagemath is not relocatable/installable we might have to compile it in-place. At least we can try to have a working system sagemath, then we move forward from there to improve (and towards option b if possible).

leahneukirchen commented 2 years ago

I don't think /opt should be used for things we compile ourselves, how about /usr/lib/sage-${version}? (and symlink selected sage binaries to /usr/bin)

leahneukirchen commented 2 years ago

I think DESTDIR is supported? It doesn't really matter if there is no special do_install step...

dkwo commented 2 years ago

I agree, let's try with (a) first.

tornaria commented 2 years ago

See #34030

tornaria commented 2 years ago

Pending PRs: #34182, #34186, #34189, #34191, #34193.

dkwo commented 2 years ago

I'm closing this, as the relevant pr #34030 is almost good to go.