Enable VPATH builds (several independent build trees connected to one source tree)

[mkoeppe]

Executive summary Currently, --prefix selects the install tree (SAGE_LOCAL) of sage-the-distribution. But often we want several builds out of (almost) the same source tree.

The vpath mechanism proposed here selects the build tree (SAGE_ROOT) of sage-the-distribution. This includes the configuration and build artefacts such as src/build/....

---

VPATH builds are a powerful feature of autotools (and other) build systems. This feature allows the developer to build a package in a separate (initially empty) build directory tree. The source directory tree is only read from (and could even be mounted read-only) and is therefore always clean.

This is a powerful feature for the developer because from the same source tree many different configurations can be built and tested, without having to go through "make distclean" and reconfiguration. The source tree can also be shared, for example using a networked file system between different hosts, running different architectures. Another modern use case involves VMs. For example, Docker allows to mount the source directory from the host in the VM (see #21474).

For Sage, "different configurations" could mean different architectures (via VMs), different sets of installed packages, Py2 vs Py3, etc.

This is how it is used.

 cd SAGE_ROOT
 autoreconf

and then

 mkdir BUILDDIR
 cd BUILDDIR
 SRCDIR/configure --srcdir=SRCDIR
 make

The present patch implements this. BUILDDIR from the above example is reflected in the old environment variable $SAGE_ROOT; this is now to be distinguished from SRCDIR, which is reflected in the new environment variable $SAGE_VPATH. $SAGE_LOCAL defaults to BUILDDIR/local, but can of course be changed using configure --prefix.

The present patch changes the various places where $SAGE_ROOT is used. When "./configure" detects a VPATH build, it installs patched copies of sage, Makefile and build/make/install in BUILDDIR.

With the current set of patches, the compilation goes through and gives a working sage, except for the docbuild.

CC: @sagetrac-felixs @jdemeyer @kiwifb @embray @nexttime @vbraun @dimpase @fchapoton @jhpalmieri

Component: build: configure

Work Issues: merge conflicts

Author: Matthias Koeppe

Branch/Commit: u/mkoeppe/vpath_build @ 05f3a91

Issue created by migration from https://trac.sagemath.org/ticket/21469

[mkoeppe]

Branch: u/mkoeppe/vpath_build

[mkoeppe]

Commit: ac27cdc

[mkoeppe]

New commits:

ac27cdc

configure.ac: VPATH fixes

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

5698f8e	Distinguish SAGE_ROOT and SAGE_SRC_ROOT for VPATH build
a98cfa2	Use AC_CONFIG_COMMANDS
e9b1c0f	Symlink build/pkgs (temporary solution)

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from ac27cdc to e9b1c0f

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

e15ada6

Define SAGE_SRC_ROOT for VPATH build

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from e9b1c0f to e15ada6

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from e15ada6 to 9323c01

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

9323c01

Use SAGE_SRC_ROOT in package scripts

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

7a3913a

More SAGE_SRC_ROOT

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 9323c01 to 7a3913a

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

4cc3954

More SAGE_SRC_ROOT

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 7a3913a to 4cc3954

[mkoeppe]

comment:9

The current patch makes some high-level preparations for a VPATH build, in particular the package system.

"make" runs through until it gets to sagelib, which needs some more VPATH work (the current patch tries to get away by just making a symlink from $SAGE_SRC_ROOT/src to $SAGE_ROOT/src, which of course fails).

I've set this to 'needs review' in the hope of receiving some feedback regarding the approach of this patch.

[kiwifb]

comment:10

I am not sure what the objective of this ticket is. Considering the nature of sage's build system I am not sure what VPATH brings me.

This is most evident in the latest commit. As a distro maintainer of sage there are already build time things in env.py that are not needed at runtime and in a way are superfluous. You have just added one. Here the problem that I am forced to consider, is that going by the separation rules that I try to follow, there should be stuff in sage/env.py and some in a new file sage_setup/build_env.py but that's marginal to what you are doing.

Other people may have a different opinion but there is no value in this for me so far. Possibly more work in fact.

[mkoeppe]

comment:11

Replying to @kiwifb:

I am not sure what the objective of this ticket is. Considering the nature of sage's build system I am not sure what VPATH brings me.

A VPATH build makes it easy to build from the same source in many different configurations. Concretely, I've started to run some VMs (with Linux 64 bit, 32 bit) via Docker, which mount the Sage source directory from the host.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

7bb8718

Make sage script in build dir executable

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 4cc3954 to 7bb8718

[mkoeppe]

comment:13

Replying to @kiwifb:

As a distro maintainer of sage there are already build time things in env.py that are not needed at runtime and in a way are superfluous. You have just added one. Here the problem that I am forced to consider, is that going by the separation rules that I try to follow, there should be stuff in sage/env.py and some in a new file sage_setup/build_env.py but that's marginal to what you are doing.

Yes, this ticket does not address any of the issues that would arise in distribution packaging.

Instead it adds a standard feature expected from autotools build system, for those who use sage as a distribution.

[kiwifb]

comment:14

I should apologize for being so negative. Currently configure is run by the initial call to make, it would be nice to get to the stage where we run configure first :)

Seriously, yes VPATH as lots of advantages, since most people run sage from where they build it, being able to do several builds from the same source without make distclean or git clone is a good thing.

I guess, it mostly started my brain on the fact that the separation build and runtime is not clean in vanilla sage. Your VPATH work may help to clarify it in some areas. Let's try to minimise where it would mix things further.

[mkoeppe]

comment:15

Replying to @kiwifb:

I should apologize for being so negative.

No worries.

Currently configure is run by the initial call to make, it would be nice to get to the stage where we run configure first :)

That already works with this patch.

Seriously, yes VPATH as lots of advantages, since most people run sage from where they build it, being able to do several builds from the same source without make distclean or git clone is a good thing.

I guess, it mostly started my brain on the fact that the separation build and runtime is not clean in vanilla sage. Your VPATH work may help to clarify it in some areas. Let's try to minimise where it would mix things further.

Sounds good. Yes, I'm hoping too that it helps clarify some things as a side-effect.

[jdemeyer]

comment:16

Replying to @mkoeppe:

Instead it adds a standard feature expected from autotools build system, for those who use sage as a distribution.

I sort of agree with François.

Part of the problem is that Sage is a distribution (a collection of packages) as well as a Python library. The fact this this is currently mixed up in the sources (the sources of the distribution-part and the library-part are in one repo) makes it more difficult to understand what this ticket is about.

For the distribution part, there really is no such thing as a build directory. There is a a temporary directory in local/var/tmp/sage/build but I would not consider that a build directory in the usual sense.

For the library part, a VPATH build is meaningful. Although, I think that VPATH builds are unusual in the Python world. Luckily, the default build directories are platform-dependendent. Anyway, it seems like this ticket isn't about that.

[jdemeyer]

comment:17

I think the ticket description should have a lot more information about

1. which problem that this ticket is trying to solve (the why).

2. what the branch actually does (the what).

Note that this are 2 independent things (which are unfortunately often confused).

[mkoeppe]

comment:18

I've updated the description.

[mkoeppe]

Description changed:

--- 
+++ 
@@ -1,4 +1,10 @@
-It would be convenient for VPATH builds to work.
+VPATH builds are a powerful feature of autotools (and other) build systems. This feature allows the developer to build a package in a separate (initially empty) build directory tree. The source directory tree is only read from (and could even be mounted read-only) and is therefore always clean. 
+
+This is a powerful feature for the developer because from the same source tree many different configurations can be built and tested, without having to go through "make distclean" and reconfiguration. The source tree can also be shared, for example using a networked file system between different hosts, running different architectures. Another modern use case involves VMs. For example, Docker allows to mount the source directory from the host in the VM (see #21474).
+
+For Sage, "different configurations" could mean different architectures (via VMs), different sets of installed packages, Py2 vs Py3, etc.
+
+This is how it is used.

mkdir BUILDDIR @@ -7,6 +13,13 @@ make

-Probably #14796 has work into this direction, but it seems to be an abandoned effort.
+The present patch implements this. BUILDDIR from the above example is reflected in the old environment variable $SAGE_ROOT; this is now to be distinguished from SRCDIR, which is reflected in the new environment variable $SAGE_SRC_ROOT. `local` is still a fixed subdirectory of $SAGE_ROOT. 
+The present patch changes the various places where $SAGE_ROOT is used.
+When "./configure" detects a VPATH build, it installs patched copies of `sage`, `Makefile` and `build/make/install` in BUILDDIR. 
+
+With the current set of patches, the compilation goes through until `sagelib` is reached. This needs some more VPATH work (the current patch tries to get away by just making a symlink from $SAGE_SRC_ROOT/src to $SAGE_ROOT/src, which of course fails). For example, `src/build/cythonized` needs to be created in BUILDDIR, and include paths etc. need to be set up accordingly.
+(Probably #14796, what seems to be an abandoned effort, has work into this direction; I haven't used changes from there yet.)

+
+

[mkoeppe]

comment:19

Replying to @jdemeyer:

Replying to @mkoeppe: Part of the problem is that Sage is a distribution (a collection of packages) as well as a Python library. The fact this this is currently mixed up in the sources (the sources of the distribution-part and the library-part are in one repo) makes it more difficult to understand what this ticket is about.

I've now clarified the goal in the description. I have not touched sagelib yet; but without VPATH work on sagelib, the distribution VPATH changes won't work. I could use help with the work on sagelib.

For the distribution part, there really is no such thing as a build directory. There is a a temporary directory in local/var/tmp/sage/build but I would not consider that a build directory in the usual sense.

Right, local/var/tmp/sage/build falls short of being a real build directory. because for example src/build/ has a lot of more generated stuff.

For the library part, a VPATH build is meaningful. Although, I think that VPATH builds are unusual in the Python world. Luckily, the default build directories are platform-dependendent. Anyway, it seems like this ticket isn't about that.

Certainly there src/build/lib.macosx-10.9-x86_64-2.7 includes the platform name, but it does not account for other configuration differences that a VPATH build can take care of (neither does src/build/cythonized).

[mkoeppe]

Description changed:

--- 
+++ 
@@ -1,4 +1,4 @@
-VPATH builds are a powerful feature of autotools (and other) build systems. This feature allows the developer to build a package in a separate (initially empty) build directory tree. The source directory tree is only read from (and could even be mounted read-only) and is therefore always clean. 
+[VPATH builds](https://www.gnu.org/software/automake/manual/html_node/VPATH-Builds.html) are a powerful feature of autotools (and other) build systems. This feature allows the developer to build a package in a separate (initially empty) build directory tree. The source directory tree is only read from (and could even be mounted read-only) and is therefore always clean. 

 This is a powerful feature for the developer because from the same source tree many different configurations can be built and tested, without having to go through "make distclean" and reconfiguration. The source tree can also be shared, for example using a networked file system between different hosts, running different architectures. Another modern use case involves VMs. For example, Docker allows to mount the source directory from the host in the VM (see #21474).

[mkoeppe]

Description changed:

--- 
+++ 
@@ -13,7 +13,7 @@
  make

-The present patch implements this. BUILDDIR from the above example is reflected in the old environment variable $SAGE_ROOT; this is now to be distinguished from SRCDIR, which is reflected in the new environment variable $SAGE_SRC_ROOT. local is still a fixed subdirectory of $SAGE_ROOT. +The present patch implements this. BUILDDIR from the above example is reflected in the old environment variable $SAGE_ROOT; this is now to be distinguished from SRCDIR, which is reflected in the new environment variable $SAGE_SRC_ROOT. local is still a fixed subdirectory of $SAGE_ROOT (but see #21479). The present patch changes the various places where $SAGE_ROOT is used. When "./configure" detects a VPATH build, it installs patched copies of sage, Makefile and build/make/install in BUILDDIR.

[mkoeppe]

comment:22

Replying to @jdemeyer:

For the library part, a VPATH build is meaningful. Although, I think that VPATH builds are unusual in the Python world. Luckily, the default build directories are platform-dependendent. Anyway, it seems like this ticket isn't about that.

It seems that pip install keeps the source directory clean, building instead in a temporary directory. pip install also offers options --build to select a build directory (though it seems as if it does not work with all packages).

[mkoeppe]

comment:23

Replying to @mkoeppe:

It seems that pip install keeps the source directory clean, building instead in a temporary directory. pip install also offers options --build to select a build directory (though it seems as if it does not work with all packages).

However, there are some pip issues: https://github.com/pypa/pip/issues/2060 https://github.com/pypa/pip/issues/2053 https://github.com/pypa/pip/issues/804

[mkoeppe]

comment:24

Replying to @jdemeyer: For the library part, a VPATH build is meaningful. Although, I think that VPATH builds are unusual in the Python world. Luckily, the default build directories are platform-dependendent. Anyway, it seems like this ticket isn't about that.

Also, in the traditional distutils ways (https://docs.python.org/2/install/), it appears that one can use --build-base to do VPATH builds.

python setup.py build --build-base=BUILDDIR

[mkoeppe]

Dependencies: #21480

[mkoeppe]

comment:25

This is now #21480.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 7bb8718 to 81f4306

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

81f4306

Fix for unconfigured non-VPATH build

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 81f4306 to 7e11d55

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

7e11d55

VPATH build

[mkoeppe]

comment:30

squashed into 1 commit to simplify rebasing.

[mkoeppe]

Changed dependencies from #21480 to #21535

[mkoeppe]

comment:31

The next step towards VPATH builds should probably be #21535 (enabling src/setup.py build --build-base, to replace the hard-coded "build").

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

ca281d2	VPATH build
ae7bdf7	configure.ac: Quoting fix
23cc689	src/Makefile.in: Use setup.py --build-base

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 7e11d55 to 23cc689

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 23cc689 to 72818e5

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

72818e5

Some path fixes for VPATH

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

3ffb004	VPATH fixes for src/bin and src/ext installs
d0366b0	build/make/deps: Fix sage-env path

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 72818e5 to d0366b0

[mkoeppe]

Changed dependencies from #21535 to none

[mkoeppe]

comment:35

Removed #21535 dependency. setup.py --build-base actually works already.

With the current status of the branch, make builds a working sage, only the docbuild does not go through yet.

[mkoeppe]

Description changed:

--- 
+++ 
@@ -17,8 +17,7 @@
 The present patch changes the various places where $SAGE_ROOT is used.
 When "./configure" detects a VPATH build, it installs patched copies of `sage`, `Makefile` and `build/make/install` in BUILDDIR. 

-With the current set of patches, the compilation goes through until `sagelib` is reached. This needs some more VPATH work (the current patch tries to get away by just making a symlink from $SAGE_SRC_ROOT/src to $SAGE_ROOT/src, which of course fails). For example, `src/build/cythonized` needs to be created in BUILDDIR, and include paths etc. need to be set up accordingly.
-(Probably #14796, what seems to be an abandoned effort, has work into this direction; I haven't used changes from there yet.)
+With the current set of patches, the compilation goes through and gives a working `sage`, except for the docbuild.

[mkoeppe]

Description changed:

--- 
+++ 
@@ -13,12 +13,11 @@
  make

-The present patch implements this. BUILDDIR from the above example is reflected in the old environment variable $SAGE_ROOT; this is now to be distinguished from SRCDIR, which is reflected in the new environment variable $SAGE_SRC_ROOT. local is still a fixed subdirectory of $SAGE_ROOT (but see #21479). +The present patch implements this. BUILDDIR from the above example is reflected in the old environment variable $SAGE_ROOT; this is now to be distinguished from SRCDIR, which is reflected in the new environment variable $SAGE_SRC_ROOT. $SAGE_LOCAL defaults to BUILDDIR/local, but can of course be changed using configure --prefix. + The present patch changes the various places where $SAGE_ROOT is used. When "./configure" detects a VPATH build, it installs patched copies of sage, Makefile and build/make/install in BUILDDIR.

With the current set of patches, the compilation goes through and gives a working sage, except for the docbuild.

-