Transition to build system for sage (the library)

8f46c63a-2596-4880-ba93-a09d8ba6c056 commented 11 years ago

Sage ("the library") needs a build system that works inside and outside of sage ("the distribution").

This ticket integrates the build systems for the parts of sagelib into the current toplevel build system.

Depends on #14726 Depends on #14834 Depends on #15102 Depends on #15112 Depends on #15117 Depends on #15123 Depends on #15126 Depends on #15176

CC: @nexttime @kiwifb @jondo

Component: distribution

Keywords: sagelib

Branch/Commit: u/felixs/sagelib @ fc097b0

Reviewer: Marc Mezzarobba

Issue created by migration from https://trac.sagemath.org/ticket/14807

8f46c63a-2596-4880-ba93-a09d8ba6c056 commented 11 years ago

Changed dependencies from #14726 to #14726 #14834

jdemeyer commented 11 years ago

Changed dependencies from #14726 #14834 to #14726, #14834

jdemeyer commented 11 years ago

comment:5

Why would this be a blocker?

8f46c63a-2596-4880-ba93-a09d8ba6c056 commented 11 years ago

comment:6

Replying to @jdemeyer:

Why would this be a blocker?

it's not, sorry.

i have created a dist tarball from my WIP branch. it is contained in http://tool.em.cs.uni-frankfurt.de/~felix/sagelib. please comment or help to improve it.

8f46c63a-2596-4880-ba93-a09d8ba6c056 commented 11 years ago

Commit: feff956

8f46c63a-2596-4880-ba93-a09d8ba6c056 commented 11 years ago

Changed dependencies from #14726, #14834 to #14726, #14834, #15117

8f46c63a-2596-4880-ba93-a09d8ba6c056 commented 11 years ago

Changed keywords from none to sagelib

8f46c63a-2596-4880-ba93-a09d8ba6c056 commented 11 years ago

Branch: u/felixs/sagelib

8f46c63a-2596-4880-ba93-a09d8ba6c056 commented 11 years ago

Description changed:

--- 
+++ 
@@ -1 +1 @@
-Sage ("the library") needs a build system that works outside of sage ("the distribution").
+Sage ("the library") needs a build system that works inside and outside of sage ("the distribution").

8f46c63a-2596-4880-ba93-a09d8ba6c056 commented 11 years ago

Changed dependencies from #14726, #14834, #15117 to #14726, #14834, #15102, #15112, #15117, #15123, #15126

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 11 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

`[changeset:afa7de7]`	`Merge branch 'python_sage_build' into sagelib`
`[changeset:848bef8]`	`Merge branch 'libgap_include' into sagelib`
`[changeset:f77e13c]`	`python-sage: env`
`[changeset:29f8f3e]`	`python-sage build system continued/cleanup`
`[changeset:f2d1c37]`	`integrate sagelib build systems into old toplevel`
`[changeset:f5261c6]`	`bump libgap package patchlevel`
`[changeset:318138e]`	`apply patches from patchdir`
`[changeset:28b0da1]`	`python-sage build system`

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 11 years ago

Changed commit from feff956 to afa7de7

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 11 years ago

Changed commit from afa7de7 to d4e1833

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 11 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

`[changeset:d4e1833]`	`Merge branches 'sage_doc', 'python_sage_build' and 'csage_build_system' into sagelib`
`[changeset:1f3f7e4]`	`Merge branches 'cython_dbg' and 'cython_deps' into python_sage_build`
`[changeset:42449c1]`	`python-sage doctest path tolerance`
`[changeset:ae8bb63]`	`python-sage: env`
`[changeset:5a16d99]`	`python-sage build system`
`[changeset:79cf599]`	`sage-bin env`
`[changeset:3ff5c49]`	`sagelib: environment for check target`
`[changeset:ffe9b9d]`	`sage-doc. use SAGE_DOC_SRC as doc source path`
`[changeset:2df5bfc]`	`sage-doc: build system. mostly dist`
`[changeset:510f8b7]`	`Merge branch 'gmp_mpir' into csage_build_system`
`[changeset:0d50108]`	`provide gmpxx_or_mpirxx.h (required by sage-python)`
`[changeset:1e4789f]`	`csage: gmp_or_mpir fixup`
`[changeset:926e3f7]`	`build autotools before csage`
`[changeset:d4e9997]`	`Merge branch 'autotools' into csage_build_system`
`[changeset:f82b422]`	`cython: patch: gdb output dir option`
`[changeset:72c4538]`	`add more autotools executables`
`[changeset:bd11c24]`	`cython: add -MF switch to write dependency files (#14728)`

8f46c63a-2596-4880-ba93-a09d8ba6c056 commented 11 years ago

Description changed:

--- 
+++ 
@@ -1 +1,3 @@
 Sage ("the library") needs a build system that works inside and outside of sage ("the distribution").
+
+This ticket integrates the build systems for the parts of `sagelib` into the current toplevel build system.

8f46c63a-2596-4880-ba93-a09d8ba6c056 commented 11 years ago

comment:11

A clean sagelib package that works outside of sage (the distribution) requires something like #14796. I will open another ticket...

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 11 years ago

Changed commit from d4e1833 to 2646f0d

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 11 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

`[changeset:2646f0d]`	`Merge branch 'runtime_paths' into sagelib`
`[changeset:0d3cfcd]`	`build/deps`
`[changeset:30fb726]`	`sagelib build system`
`[changeset:b2415da]`	`provide sage-sh within sage-the-distribution`
`[changeset:8de880c]`	`fix a typo`
`[changeset:39f7247]`	`ellcurve data dir fixup`
`[changeset:0b9cb04]`	`doctests in src/sage/plot/misc.py`
`[changeset:1ef62a1]`	`unhardwire cython.py completed`
`[changeset:390e28b]`	`qepcard doctest fix: singular might well be in /usr/bin, not .../local/bin`
`[changeset:0dcd721]`	`unhardwire ELLCURVES_DATA_DIR`
`[changeset:591471d]`	`python-sage: whitelist python2.7 multiprocessing in stackframes test`
`[changeset:c19a5f3]`	`unhardwire sandpile`
`[changeset:5746289]`	`unhardwire polymake path`

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 11 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

`[changeset:1c89552]`	`cython doctest`
`[changeset:ef8771d]`	`dependency chain in build/deps`
`[changeset:7aaf953]`	`Merge 'trac/u/ohanar/build_system' into sagelib`

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 11 years ago

Changed commit from 2646f0d to 1c89552

8f46c63a-2596-4880-ba93-a09d8ba6c056 commented 11 years ago

Changed dependencies from #14726, #14834, #15102, #15112, #15117, #15123, #15126 to #14726, #14834, #15102, #15112, #15117, #15123, #15126, #15176

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 11 years ago

Changed commit from 1c89552 to 0e91a83

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 11 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

`[changeset:0e91a83]`	`Merge branches 'sage-bin' and 'python_sage_build' into sagelib`
`[changeset:e429991]`	`inject proper paths into autotools calls`
`[changeset:41326b4]`	`sagelib: freeze (parts of) environment during configure`
`[changeset:ec442e5]`	`always use autotools within sage ("the distribution")`
`[changeset:26c67b2]`	`python-sage: make cython call more toolchain friendly`
`[changeset:69fda9a]`	`Merge remote-tracking branch 'trac/u/ohanar/build_system' into sage-bin`
`[changeset:f9f8d16]`	`sage-bin: remove PATH_REDUNDANCY cruft`
`[changeset:3e2aeb1]`	`transition from old toplevel. create symlinks, for now.`
`[changeset:e0064c9]`	`sage-bin expand SAGE_ETC correctly`
`[changeset:fcb670f]`	`simplify python byte compile rules`
`[changeset:2af9070]`	`GMP_VS_MPIR fixup`
`[changeset:45a30e1]`	`provide unhardwired SAGE_CFLAGS`
`[changeset:e195725]`	`python-sage: Makefiles`
`[changeset:c4562b3]`	`python_sage: build system continued`
`[changeset:6d074ee]`	`sage-bin: more config files`

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 11 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

`[changeset:fc097b0]`	`build/deps fixup`
`[changeset:88a4478]`	`Merge branch 'python_sage_build' into sagelib`
`[changeset:1dacfa5]`	`python-sage: Makefiles update for 512beta4`
`[changeset:cb27876]`	`python-sage: Makefiles fix`

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 11 years ago

Changed commit from 0e91a83 to fc097b0

jdemeyer commented 10 years ago

comment:17

Needs to be rebased.

embray commented 8 years ago

comment:21

I wonder if Felix could give an update on the status of this work. Obviously it hasn't been touched in a long while and needs to be rebased. But it seems like a lot of the heavy lifting has been done and this provides a good starting point for overhauling Sage's build.

What is the status of the Sage community's interest in this sort of work?

8f46c63a-2596-4880-ba93-a09d8ba6c056 commented 8 years ago

comment:22

Hi Erik.

maybe you can reycle the configure.ac (and most path related patches) for sagelib. the canonical transition would be to reuse the python build system for a start (templatize setup.py, call it from make...)

i could not get anywhere before rewriting the whole thing to automake, as setup.py did/does not work (until today?). mostly due to

lack of automatic dependency tracking (C/C++ compilers only write makefiles, setup.py does not use them. cython -MF is not ready yet)
missing VPATH builds (requirement for mere sanity, also for distcheck, cf. #21469),

rendering the approach useless for debugging/development on foreign distros.

eventually all this was of no use for sage (the distribution) until a functional top-level build system would be in place. it might still facilitate packaging and use...

i am not totally up to date with the recent (last ~3 years) changes, and YMMV.

jdemeyer commented 8 years ago

comment:23

@sagetrac-felixs: I think it is a bit strange that you're trying to use a autotools-based build system for a Python package. I don't know if that has ever been done. I agree in principle that autotools is better than whatever Python provides. However, you should weigh this against the fact that this would be very non-standard. I guess that distros would hate us for doing that.

8f46c63a-2596-4880-ba93-a09d8ba6c056 commented 8 years ago

comment:24

Replying to @jdemeyer:

@sagetrac-felixs: I think it is a bit strange that you're trying to use a autotools-based build system for a Python package.

hi Jeroen

the reason i pulled in autotools was initially just autoconf. if you can make autoconf+setup.py.in work, then you don't need automake. likewise, if you reimplement autoconf in python etcpp. there are many ways, most of which involve "implement a viable alternative to autotools", which in my understanding is beyond the scope of this project.

I don't know if that has ever been done.

automake supports python for a loong time. i am not "trying to use" rather than "suggesting" to use it. the cython rules i wrote for sagelib back then are now in use for another project (where, coincidentally, cython is used on some interface code).

I guess that distros would hate us for doing that.

that i don't understand. package maintainers usually keep saying things like "people, use auto{conf,tools,make}". cannot think of a way that would make packageing any easier. (care to explain?)

jdemeyer commented 8 years ago

comment:25

Replying to @sagetrac-felixs:

that i don't understand. package maintainers usually keep saying things like "people, use auto{conf,tools,make}".

...as alternative to home-brew build systems. But I have never seen a package maintainer say that as alternative to setup.py.

8f46c63a-2596-4880-ba93-a09d8ba6c056 commented 8 years ago

comment:26

But I have never seen a package maintainer say that as alternative to setup.py.

true/plausible. the typical python package has simple dependencies and does not use cython. no need for a chainsaw.

still, i see no evidence why someone might "hate us" for using auto* (for whichever sort of package). instead (let me repeat), the autotools implement

VPATH
(pre build) configuration
(compile time) dependencies
a make based build process all of which immediately facilitate development, testing, debugging and thus packaging.

jdemeyer commented 8 years ago

comment:27

Replying to @sagetrac-felixs:

instead (let me repeat), the autotools implement

VPATH

(pre build) configuration

(compile time) dependencies

a make based build process all of which immediately facilitate development, testing, debugging and thus packaging.

The autotools implement that for a typical C/C++ package. They don't really implement all that for Python packages. Which means that we have to implement that ourselves.

8f46c63a-2596-4880-ba93-a09d8ba6c056 commented 8 years ago

comment:28

The autotools implement that for a typical C/C++ package.

at least the four bullet points are intended to work for a lot more, including python packages [1]. arguably, to compile cython stuff, some custom (make) rules are required, but we had that. and it could be changed (=moved into automake).

They don't really implement all that

what exactly do you mean? please give an example.

thanks

[1] http://ftp.gnu.org/old-gnu/Manuals/automake-1.6.1/html_node/automake_59.html

jdemeyer commented 8 years ago

comment:29

OK, that looks interesting. I didn't know that automake could do that.

kiwifb commented 8 years ago

comment:30

Yes it wexist but it is not always implemented in a nice way for package maintainer. In most case I have seen that used, you have a main package using autoconf and then they have some python bindings. Typically if if the python bits support multiple versions of python, as a package maintainer I would prefer it in a separate package.

In a distro you will usually provide bindings for all of the versions of python you support when feasible. When it is integrated as part of a bigger package it is usually a pain to cycle through different versions of python. Once it is a separate package, autotools, cmake or setup.py, I don't really care any more, I'll easily cycle on both.

Admittedly software detection and so on is easier with something else than python...

83660e46-0051-498b-a8c1-f7a7bd232b5a commented 8 years ago

comment:31

Replying to @kiwifb:

Admittedly software detection and so on is easier with something else than python...

SConstruct! B)

kiwifb commented 8 years ago

comment:32

I will not feed the trolls. I will not feed the trolls......

83660e46-0051-498b-a8c1-f7a7bd232b5a commented 8 years ago

comment:33

Jeroen loves SC...

jdemeyer commented 8 years ago

comment:34

Replying to @kiwifb:

Admittedly software detection and so on is easier with something else than python...

For the record: in https://github.com/sagemath/cysignals I am using autoconf for the configuration but I am still using distutils to build and install.

8f46c63a-2596-4880-ba93-a09d8ba6c056 commented 8 years ago

comment:35

Replying to @jdemeyer:

For the record:

here's the autotoolized sage (dist [1] + lib [2]). this is where i gave up. i never touched the sage code since then... watch out for configure.ac and various m4 scripts. note that all subdirectories in src are standalone packages.

[1] https://github.com/felix-salfelder/sage/tree/autotools

[2] https://github.com/felix-salfelder/sage/tree/autotools/src

embray commented 8 years ago

comment:36

I have mixed feelings about the whole thing, but it's worth noting that it's getting closer and closer to the point where what's in setup.py won't matter as much as it does now. distutils is still the "easiest" way to build Python modules, so somewhere in there there will still likely be a setup.py for a while.

But there's been some progress on better supporting binary wheels on Linux (see for example https://www.python.org/dev/peps/pep-0513/, though that obviously doesn't cut it for Sage). There's also progress being made on standardizing a "source distribution" format for Python packages that isn't necessarily tied to any one build system. This means it will be fine, for example, to use autoconf to build Sage. It's fine now, even, it just doesn't fit in as as well (yet) with the Python ecosystem.

I think regardless, it will also help to further break up sage so that Python wrappers for its various dependencies are external projects themselves. I realize that's a much bigger problem though and not always practical.

embray commented 8 years ago

comment:37

Also, all that said, there are examples of Pythonic build configuration too. One of the best is matplotlib's: https://github.com/matplotlib/matplotlib/blob/master/setupext.py

Matplotlib has quite a few external, non-Python dependencies--nowhere close to Sage, but more than most Python packages, and it handles this nicely. Yes, it bothers me that it's "hand rolled", but most of that stuff in matplotlib can, and probably should, be factored out to an external package (with documentation!) that others can use. And I would rather be writing Python than m4 scriptlets >_<

Numpy has a similar system (used also by Scipy), but matplotlib's is a bit cleaner (even it has a few things I would do a little differently, but the concept is sound).

mkoeppe commented 4 years ago

comment:38

Outdated, should be closed

mkoeppe commented 4 years ago

Changed author from Felix Salfelder to none

mezzarobba commented 3 years ago

Reviewer: Marc Mezzarobba

sagemath / sage

Transition to build system for sage (the library) #14807