Make sagelib setup.py self-contained and independent of SAGE_ROOT

mkoeppe commented 8 years ago

This ticket changes the build process of sagelib in the following way:

src/Makefile delegates ALL building to src/setup.py
src/setup.py no longer depends on environment variables $SAGE_ROOT, $SAGE_SRC, $SAGE_DOC_SRC etc. (to demonstrate this, Makefile poisons these environment variables). It still depends on $SAGE_LOCAL and environment variables that point below it.

This ticket is meant as:

preparation for VPATH builds of sage-the-distribution (#21469)
working towards the goal of making sagelib pip-installable -- see #21507 for the eventual goal of having sagelib on PyPI
making the flow of directory information at build time clearer for developers

More specifically, the goal of this ticket is that only SAGE_LOCAL needs to be set when the user does 'pip install' of the sagelib. (This ticket almost achieves this, except it also needs SAGE_PKGS and SAGE_CYTHONIZED to be set. The hope is that #20382 and other future tickets will develop better mechanisms to communicate package and directory information to the build.)

. . . . . . .

Some possibly useful information:

Documentation on distutils (https://docs.python.org/2/install/), describing use of --build-base to do VPATH builds.
pip install keeps the source directory clean, building instead in a temporary directory, by copying the sources. pip install also offers options --build to select a build directory, but there are some pip issues: 2060, 2053, 804 that affect this
14807 has some tricks to making VPATH builds work without copying all python source files. But it uses automake instead of setup.sh; we will not do this in our ticket.

configure tarball: http://sage.ugent.be/www/jdemeyer/sage/configure-185.tar.gz

CC: @sagetrac-felixs @jdemeyer @kiwifb @embray @nexttime @vbraun @dimpase @jhpalmieri @videlec @saraedum @seblabbe @nthiery @mezzarobba

Component: build

Author: Matthias Koeppe

Branch/Commit: 0c2ac95

Reviewer: Jeroen Demeyer

Issue created by migration from https://trac.sagemath.org/ticket/21480

kiwifb commented 8 years ago

comment:1

Am interesting issue is that technically cython_debug has to be installed somewhere (preferably somewhere standard) to be accessible at runtime separately from the source.

I do something in sage-on-gentoo but that's not really satisfactory.

83660e46-0051-498b-a8c1-f7a7bd232b5a commented 8 years ago

Replying to @mkoeppe:

In preparation for VPATH builds of sage-the-distribution (#21469), let's keep src/ clean by using setup.py --build-base=$SAGE_ROOT/var/tmp/sage/build/sagelib

Please use --build-base=$SAGE_BUILD_DIR/sagelib.

83660e46-0051-498b-a8c1-f7a7bd232b5a commented 8 years ago

comment:3

... and sagelib perhaps with a version suffix.

mkoeppe commented 8 years ago

comment:4

Replying to @nexttime:

Replying to @mkoeppe:

In preparation for VPATH builds of sage-the-distribution (#21469), let's keep src/ clean by using setup.py --build-base=$SAGE_ROOT/var/tmp/sage/build/sagelib

Please use --build-base=$SAGE_BUILD_DIR/sagelib.

Yes, the plan after #21469 is to use the Sage builddir -- not just for sagelib, but also for other packages.

Before #21469 is merged, I want to use $SAGE_ROOT/var/tmp/sage/build/sagelib to match what other packages do.

mkoeppe commented 8 years ago

comment:5

Ah, I see what you meant, now I've found $SAGE_BUILD_DIR. Changed description accordingly.

mkoeppe commented 8 years ago

Description changed:

--- 
+++ 
@@ -1,4 +1,6 @@
 Currently, building sagelib creates the `src/build` directory, with subdirectories `cython_debug`, `cythonized`, `lib.UNAME`, `temp.UNAME`. 

-In preparation for VPATH builds of sage-the-distribution (#21469), let's keep `src/` clean by using `setup.py --build-base=$SAGE_ROOT/var/tmp/sage/build/sagelib`
+In preparation for VPATH builds of sage-the-distribution (#21469), let's keep `src/` clean by using `setup.py --build-base=$SAGE_BUILD_DIR/sagelib-VERSION`

+(`$SAGE_BUILD_DIR` defaults to `$SAGE_ROOT/var/tmp/sage/build/`)
+

83660e46-0051-498b-a8c1-f7a7bd232b5a commented 8 years ago

comment:6

Replying to @mkoeppe:

Replying to @nexttime:

Replying to @mkoeppe:

In preparation for VPATH builds of sage-the-distribution (#21469), let's keep src/ clean by using setup.py --build-base=$SAGE_ROOT/var/tmp/sage/build/sagelib

Please use --build-base=$SAGE_BUILD_DIR/sagelib.

Yes, the plan after #21469 is to use the Sage builddir -- not just for sagelib, but also for other packages.

??? SAGE_BUILD_DIR exists since years already... (Its default is $SAGE_ROOT/var/tmp/sage/build/.)

Before #21469 is merged, I want to use $SAGE_ROOT/var/tmp/sage/build/sagelib to match what other packages do.

mkoeppe commented 8 years ago

comment:7

Replying to @nexttime:

??? SAGE_BUILD_DIR exists since years already... (Its default is $SAGE_ROOT/var/tmp/sage/build/.)

Yes, thanks, see my other comment.

83660e46-0051-498b-a8c1-f7a7bd232b5a commented 8 years ago

comment:8

Ah ok, race condition.

mkoeppe commented 8 years ago

comment:9

By the way, help with implementing this change would be appreciated. I haven't looked at the sagelib build system at all so far.

mkoeppe commented 8 years ago

Branch: u/mkoeppe/keep_src__clean_by_using___build_base_when_building_sagelib

mkoeppe commented 8 years ago

Author: Matthias Koeppe

mkoeppe commented 8 years ago

New commits:

e399bf4 First, wishful step

mkoeppe commented 8 years ago

Commit: e399bf4

mkoeppe commented 8 years ago

Description changed:

--- 
+++ 
@@ -4,3 +4,8 @@

 (`$SAGE_BUILD_DIR` defaults to `$SAGE_ROOT/var/tmp/sage/build/`)

+Some possibly useful information:
+- Documentation on distutils (https://docs.python.org/2/install/), describing use of `--build-base` to do VPATH builds.
+- `pip install` keeps the source directory clean, building instead in a temporary directory, by copying the sources. 
+ `pip install` also offers options `--build` to select a build directory (though it seems as if it does not work with all packages). However, there are some pip issues: [2060](https://github.com/pypa/pip/issues/2060), [2053](https://github.com/pypa/pip/issues/2053), [804](https://github.com/pypa/pip/issues/804) that affect this
+- #14807 has some tricks to making VPATH builds work without copying all python source files. But it uses automake instead of setup.sh; we will not do this in our ticket.

jdemeyer commented 8 years ago

comment:13

Really sagelib-$SAGE_VERSION? Do you want to fill up everybody's hard disks with Sage build directories? Not to mention that this would require to rebuild everything whenever the Sage version changes.

jdemeyer commented 8 years ago

Description changed:

--- 
+++ 
@@ -1,6 +1,6 @@
 Currently, building sagelib creates the `src/build` directory, with subdirectories `cython_debug`, `cythonized`, `lib.UNAME`, `temp.UNAME`. 

-In preparation for VPATH builds of sage-the-distribution (#21469), let's keep `src/` clean by using `setup.py --build-base=$SAGE_BUILD_DIR/sagelib-VERSION`
+In preparation for VPATH builds of sage-the-distribution (#21469), let's keep `src/` clean by using `setup.py --build-base=$SAGE_BUILD_DIR/sagelib`

 (`$SAGE_BUILD_DIR` defaults to `$SAGE_ROOT/var/tmp/sage/build/`)

mkoeppe commented 8 years ago

comment:15

Fine with me without $SAGE_VERSION; I was just following leif's suggestion in comment 3.

8f46c63a-2596-4880-ba93-a09d8ba6c056 commented 8 years ago

comment:16

i understand that SAGE_BUILD_DIR is the srcdir for toplevel configure.

as long as sagelib is rooted in $(toplevel)/src, it might be less confusing to choose $SAGE_BUILD_DIR/src (not $SAGE_BUILD_DIR/sagelib) as the builddir for sagelib.

(future: replace src by sagelib, but on both ends)

i dont know exactly how the approach using setup.py will emulate VPATH builds. i think it should imitate "what automake would do", where applicable.

mkoeppe commented 8 years ago

comment:17

Replying to @sagetrac-felixs:

i understand that SAGE_BUILD_DIR is the srcdir for toplevel configure.

No, it's $SAGE_ROOT/var/tmp/sage/build/

8f46c63a-2596-4880-ba93-a09d8ba6c056 commented 8 years ago

comment:18

ok, nevermind (i meant to write "SAGE_BUILD_DIR is the builddir for toplevel"). that does not seem to be the case either.

still i am wondering why "src" does not (simply) translate to "src". (sure, i am slightly autotools biased).

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 8 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

`f3f275e`	`Put .cython_version next to cythonized files`
`94a58f1`	`Make SAGE_BUILD_DIR variable available in sage-env, not just in sage-spkg`
`a73fa06`	`Use SAGE_BUILD_DIR`

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 8 years ago

Changed commit from e399bf4 to a73fa06

mkoeppe commented 8 years ago

comment:20

Here's a first version for review. It seems to work for me.

There are still some to-do items (see comments in src/Makefile):

I am now using --build-base, but setup.sh also depends on SAGE_CYTHONIZED, defined in src/sage/env.py.

I think it would be better if setup.sh instead inferred that location from the build-base that was passed to it.

However, setup.sh already does a lot of stuff depending on SAGE_CYTHONIZED before distutils.core.setup is even called. Can this be fixed?

I think I could use some help from the Python experts in the cc list of this ticket on this.
sage/libs/pari/auto_gen.pxi and sage/ext/interpreters/__init__.py still need to be taken care of in preparation for the VPATH build.

mkoeppe commented 8 years ago

Description changed:

--- 
+++ 
@@ -7,5 +7,5 @@
 Some possibly useful information:
 - Documentation on distutils (https://docs.python.org/2/install/), describing use of `--build-base` to do VPATH builds.
 - `pip install` keeps the source directory clean, building instead in a temporary directory, by copying the sources. 
- `pip install` also offers options `--build` to select a build directory (though it seems as if it does not work with all packages). However, there are some pip issues: [2060](https://github.com/pypa/pip/issues/2060), [2053](https://github.com/pypa/pip/issues/2053), [804](https://github.com/pypa/pip/issues/804) that affect this
+ `pip install` also offers options `--build` to select a build directory, but there are some pip issues: [2060](https://github.com/pypa/pip/issues/2060), [2053](https://github.com/pypa/pip/issues/2053), [804](https://github.com/pypa/pip/issues/804) that affect this
 - #14807 has some tricks to making VPATH builds work without copying all python source files. But it uses automake instead of setup.sh; we will not do this in our ticket.

jdemeyer commented 8 years ago

comment:22

Thinking about it more, I disagree with building in $SAGE_BUILD_DIR/sagelib by default.

$SAGE_LOCAL (which contains $SAGE_BUILD_DIR) is meant as install directory, not as build directory.

Let's keep $SAGE_LOCAL to be exactly the install directory and nothing else (*).

I do agree with making the build directory configurable for VPATH builds. For typical packages however, if you do not do a VPATH build, the build directory is the same as the source directory. Sage should follow the same model. This means that the build directory should be $SAGE_SRC by default.

(*) One could argue that $SAGE_BUILD_DIR is currently used as build directory for packages. That is true, but they are only used temporarily, they are not meant to actually store stuff. So this isn't so bad.

8f46c63a-2596-4880-ba93-a09d8ba6c056 commented 8 years ago

comment:23

Let's keep $SAGE_LOCAL to be exactly the install directory and nothing else (*).

this is part of the problem with the current "build system". SAGE_LOCAL is not the install directory. it is where stuff ends up during the build process.

outside sage, an install directory is typically what you pass to --prefix, and where stuff is put into by "make install". you do that (if you are a sysadmin), after assuring that "make check" passes...

yes i know why SAGE_LOCAL exists, but it should be clear that it's not necessary and that it only has "evolved" that way. having simplified things in the past, now it is falling on your feet.

note how i tried to fix/work\ around that in my attempt to autotoolize sage (both the library and the distribution)... my point: the best approach will be to not use SAGE_LOCAL at all. here's the chance to cleanup sagelib, as a start.

that said: keep up the good and interesting work @mkoeppe. i hope i will have some more time later this year ...

jdemeyer commented 8 years ago

comment:24

Replying to @sagetrac-felixs:

this is part of the problem with the current "build system". SAGE_LOCAL is not the install directory.

Why do you think that $SAGE_LOCAL is not the install directory? It is the directory where everything is installed, which by definition makes it the install directory. The fact that Sage does not (yet) support --prefix doesn't change this fact.

the best approach will be to not use SAGE_LOCAL at all.

I like to know why you think that.

that said: keep up the good and interesting work @mkoeppe. i hope i will have some more time later this year ...

To be clear: I didn't say that this branch needs to be thrown out. I am just saying: keep the build directory configurable but keep the default what it currently is.

8f46c63a-2596-4880-ba93-a09d8ba6c056 commented 8 years ago

comment:25

Why do you think that $SAGE_LOCAL is not the install directory?

you wrote it. "--prefix" is not implemented/supported. (but it should be). there's no way to really "install" stuff in the usual sense. e.g., there is no way to install stuff after make check has passed. (i don't know if toplevel make check is currently implemented at all, just a thought).

where everything is installed, which by definition makes it the install directory

actually nothing gets installed. what is done is mostly overhead. working around the fact that some packages don't work right after the build alone. when i was done with the "package content lists" for spkg-install, i noticed that it was a huge waste of time... don't repeat that, better just skip the "install" step.

I like to know why you think that [SAGE_LOCAL should not be used].

every instance/use of SAGE_LOCAL breaks sagelib on (lets call it) foreign distros a bit. that's not helpful. it will as well interfere with any attempt on rewriting sage-the-distribution (be it autotools based, or pip or ebuild). the autotools approach (not a necessary step, but an example) provides a transition path to anything...

sage-the-distribution is a platform for sage (core) development. no more, no less. other platforms will come and go. what is needed is sagelib without the dependency on this (and on SAGE_LOCAL). why: because developers should be able to use sagelib and develop sage extensions on their own platforms, with their own tools and their own review policies.

so: please embrace contributions that reduce the use of SAGE_LOCAL.

(yes, its getting off-topic. but i hope, this answers the question.)

mkoeppe commented 8 years ago

comment:26

Replying to @jdemeyer:

Thinking about it more, I disagree with building in $SAGE_BUILD_DIR/sagelib by default.

$SAGE_LOCAL (which contains $SAGE_BUILD_DIR) is meant as install directory, not as build directory.

Let's keep $SAGE_LOCAL to be exactly the install directory and nothing else (*).

I agree with you; see #21479.

mkoeppe commented 8 years ago

comment:27

Replying to @sagetrac-felixs:

outside sage, an install directory is typically what you pass to --prefix, and where stuff is put into by "make install". you do that (if you are a sysadmin), after assuring that "make check" passes...

See #21479 for a discussion of --prefix and make vs. make install.

But let's keep the discussion of the present ticket focused on the task at hand.

mkoeppe commented 8 years ago

comment:28

Replying to @jdemeyer:

I do agree with making the build directory configurable for VPATH builds. For typical packages however, if you do not do a VPATH build, the build directory is the same as the source directory. Sage should follow the same model. This means that the build directory should be $SAGE_SRC by default.

I agree on this as well. #21469 (VPATH for distro) will do that.

In this ticket, I first want to make the --build-base work. In particular, get rid of SAGE_CYTHONIZED.

This also works towards the eventual goal of making 'sagelib' pip-installable.

The actual location used is a detail that will be easy to change later.

(*) One could argue that $SAGE_BUILD_DIR is currently used as build directory for packages. That is true, but they are only used temporarily, they are not meant to actually store stuff. So this isn't so bad.

Yes, the current patch follows this practice. But this is temporary until #21469 is done.

jdemeyer commented 8 years ago

comment:29

Replying to @sagetrac-felixs:

Why do you think that $SAGE_LOCAL is not the install directory?

you wrote it. "--prefix" is not implemented/supported.

The fact that the installation process does not implement ./configure --prefix does not mean that it's not an installation process...

every instance/use of SAGE_LOCAL breaks sagelib

But why?

Of course, it's all a matter of definition. I consider the process of copying files to $SAGE_LOCAL an "installation" and you do not (for reasons which are still unclear to me). I think things will become much simpler for you if you accept the fact that $SAGE_LOCAL is the installation directory and that copying files to $SAGE_LOCAL is an installation.

but i hope, this answers the question

I absolutely does not. You are just saying "it breaks stuff" but not explaining why.

dimpase commented 8 years ago

comment:30

Replying to @jdemeyer:

Replying to @sagetrac-felixs:

[...]

but i hope, this answers the question

I absolutely does not. You are just saying "it breaks stuff" but not explaining why.

One reason why is that in many deployment scenarios you will want to test what you have built before installing it. And Sage does not support this.

8f46c63a-2596-4880-ba93-a09d8ba6c056 commented 8 years ago

comment:31

indeed. in "build"-"check"-"install" there is not much room for definitions of "build" and "install".

this is drifting towards the question of whether or not sage should stick to common practices and terminology. you know i think it should.

apologies for being off-topic again, this may be more related to #15105 -- there is no ticket for just "practices and terminology".

jdemeyer commented 8 years ago

comment:32

Replying to @dimpase:

One reason why is that in many deployment scenarios you will want to test what you have built before installing it. And Sage does not support this.

How does this relate to the existence of $SAGE_LOCAL?

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 8 years ago

Changed commit from a73fa06 to a854831

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 8 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

a854831 Put setup.sh in charge of all sagelib building

jdemeyer commented 8 years ago

comment:34

Replying to @sagetrac-felixs:

this is drifting towards the question of whether or not sage should stick to common practices and terminology. you know i think it should.

If those "common" practices and terminology make sense, of course it should.

Anyway, I am completely not following what you are trying to say. I feel that you are always wandering around my questions instead of answering them.

mkoeppe commented 8 years ago

comment:35

I have created #21495 as a place for such discussions.

mkoeppe commented 8 years ago

Description changed:

--- 
+++ 
@@ -4,6 +4,8 @@

 (`$SAGE_BUILD_DIR` defaults to `$SAGE_ROOT/var/tmp/sage/build/`)

+As a second goal of this ticket, `setup.sh` is put in charge of ALL sagelib building. 
+
 Some possibly useful information:
 - Documentation on distutils (https://docs.python.org/2/install/), describing use of `--build-base` to do VPATH builds.
 - `pip install` keeps the source directory clean, building instead in a temporary directory, by copying the sources.

dimpase commented 8 years ago

comment:37

Replying to @jdemeyer:

Replying to @dimpase:

One reason why is that in many deployment scenarios you will want to test what you have built before installing it. And Sage does not support this.

How does this relate to the existence of $SAGE_LOCAL?

$SAGE_LOCAL is fine as long as you can also install things from there somewhere else. Currently you cannot do this in a proper way (yes, you can certainly create symbolic links etc, but this is often not enough).

jdemeyer commented 8 years ago

comment:38

Replying to @dimpase:

$SAGE_LOCAL is fine as long as you can also install things from there somewhere else. Currently you cannot do this in a proper way

Of course you can:

$ cd $package_source
$ sage --sh
$ ./configure --prefix="$SAGE_LOCAL"
$ make install

I have done this many times as first step of testing a new package or a package upgrade.

mkoeppe commented 8 years ago

comment:39

If I understand Dima correctly, he wants to install "from" $SAGE_LOCAL, not "to" $SAGE_LOCAL.

But really this discussion should go to #21495. Thanks!

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 8 years ago

Changed commit from a854831 to 1a027d5

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 8 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

1a027d5 Have setup.sh derive SAGE_CYTHONIZED from --build-base

mkoeppe commented 8 years ago

comment:41

Jeroen, how does this look to you now? (I haven't changed $SAGE_BUILD_DIR yet, but will in a moment.)

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 8 years ago

Changed commit from 1a027d5 to f6555ce

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 8 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

f6555ce Fix last change - handle --build-base earlier

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 8 years ago

Changed commit from f6555ce to d829eb4

sagemath / sage

Make sagelib setup.py self-contained and independent of SAGE_ROOT #21480

14807 has some tricks to making VPATH builds work without copying all python source files. But it uses automake instead of setup.sh; we will not do this in our ticket.