configure.ac: write build/make/Makefile within an AC_CONFIG_FILE, not during main configure

[mkoeppe]

build/make/Makefile should be written by config.status within an AC_CONFIG_FILE template, not at configure time.

Depends on #24729 Depends on #24703

Component: build: configure

Author: Erik Bray

Branch: 2a99185

Reviewer: Julian Rüth

Issue created by migration from https://trac.sagemath.org/ticket/21524

[jdemeyer]

The title "Get rid of file descriptor 7 hack" is different from the description "build/make/Makefile should be written within an AC_CONFIG_COMMANDS procedure". Do you want to fix both things here? Can you please clarify?

And honestly, I think there is nothing wrong with the "file descriptor 7 hack". autoconf itself uses similar hacks internally.

[mkoeppe]

comment:2

I've updated the title

[mkoeppe]

comment:3

I agree, there's nothing wrong with using a file descriptor to write to a file ;)

[mkoeppe]

Description changed:

--- 
+++ 
@@ -1,2 +1,3 @@
-build/make/Makefile should be written within an AC_CONFIG_COMMANDS procedure, not at configure time. 
+`build/make/Makefile` should be written within an AC_CONFIG_COMMANDS procedure, not at configure time. 

+Marking this as dependent on #21479 just because that that ticket changes the same part of `configure.ac`.

[mkoeppe]

Dependencies: #21479

[mkoeppe]

Description changed:

--- 
+++ 
@@ -1,3 +1,3 @@
 `build/make/Makefile` should be written within an AC_CONFIG_COMMANDS procedure, not at configure time. 

-Marking this as dependent on #21479 just because that that ticket changes the same part of `configure.ac`.
+Marking this as dependent on #21479 just because that ticket changes the same part of `configure.ac`.

[embray]

comment:6

Does it necessarily need to use AC_CONFIG_COMMANDS? Most of the stuff that writes build/make/Makefile could just as easily be moved into its own macro in a separate file in the m4/ directory.

As I see it the main goal here is to simplify the configure.ac by moving more functionality into separate files. But moving the existing code into an m4 macro would take less work, and has the advantage that it would have full access to any other variables set in other parts of the configure script, without having to explicitly pass them on to an external command.

[embray]

comment:7

On second thought, I can see why this way would be better, as it's more in spirit with how configure is supposed to work w.r.t. generating files. I'll have to give that some more thought. Maybe there aren't so many variables that need to be passed in.

[embray]

Branch: u/embray/build/makefile-in

[embray]

Description changed:

--- 
+++ 
@@ -1,3 +1,3 @@
-`build/make/Makefile` should be written within an AC_CONFIG_COMMANDS procedure, not at configure time. 
+`build/make/Makefile` should be written within an AC_CONFIG_FILE template, not at configure time. 

 Marking this as dependent on #21479 just because that ticket changes the same part of `configure.ac`.

[embray]

Commit: db428a3

[embray]

Author: Erik Bray

[embray]

comment:8

Here's my proposed solution to this (I updated the ticket to mention AC_CONFIG_FILE since that's what this uses).

This adds a build/make/Makefile.in template that is used to generated build/make/Makefile in config.status. The configure script now just collects values for some variables that are inserted into the Makefile template. I tried to keep as much actual Makefile syntax out of the output variables as possible, sticking mostly to just lists of package names that are operated over. Most of the rules for building packages are generated by the Makefile itself in $(eval ...) loops. The end result I think is actually cleaner and easier to understand.

If any of this should be controversial, it may be the use of some more advanced GNU make constructs, the portability of which I'm not sure. This was tested with GNU make 3.81 (about 11 years old), and 4.2.1 (very recent). That said, it's using some relatively advanced GNU make features that may not be available in other makes. I think it's still better, but if need be more of the Makefile generation could be pushed back into configure, but still provided via output variables, instead of writing directly to a file.

Increased the priority since I have other build system work that would likely depend on this, and also because it makes some non-trivial performance improvements, especially on Cygwin.

---

New commits:

b034b3a	Fix minor bug in the conway_polynomials dependencies; it should just list the bare package name
f88d3cb	Use AC_CONFIG_FILES + AC_OUTPUT to generate build/make/Makefile
db428a3	Further rewrite of configure.ac to do away with the multiple

[embray]

comment:10

Weird that the patchbot is failing due to Numpy of all things. I wonder if it has something more to do with the fact that it's building in a temp dir (since this is an "unsafe" ticket) than anything to do with the patch itself, but I'll have to see if I can reproduce...

[numpy-1.12.1] Traceback (most recent call last):
[numpy-1.12.1]   File "<stdin>", line 3, in <module>
[numpy-1.12.1]   File "/tmp/tmp4YcJD2-sage-git-temp-21524/local/lib/python2.7/site-packages/numpy/__init__.py", line 142, in <module>
[numpy-1.12.1]     from . import add_newdocs
[numpy-1.12.1]   File "/tmp/tmp4YcJD2-sage-git-temp-21524/local/lib/python2.7/site-packages/numpy/add_newdocs.py", line 13, in <module>
[numpy-1.12.1]     from numpy.lib import add_newdoc
[numpy-1.12.1]   File "/tmp/tmp4YcJD2-sage-git-temp-21524/local/lib/python2.7/site-packages/numpy/lib/__init__.py", line 18, in <module>
[numpy-1.12.1]     from .polynomial import *
[numpy-1.12.1]   File "/tmp/tmp4YcJD2-sage-git-temp-21524/local/lib/python2.7/site-packages/numpy/lib/polynomial.py", line 20, in <module>
[numpy-1.12.1]     from numpy.linalg import eigvals, lstsq, inv
[numpy-1.12.1]   File "/tmp/tmp4YcJD2-sage-git-temp-21524/local/lib/python2.7/site-packages/numpy/linalg/__init__.py", line 51, in <module>
[numpy-1.12.1]     from .linalg import *
[numpy-1.12.1]   File "/tmp/tmp4YcJD2-sage-git-temp-21524/local/lib/python2.7/site-packages/numpy/linalg/linalg.py", line 29, in <module>
[numpy-1.12.1]     from numpy.linalg import lapack_lite, _umath_linalg
[numpy-1.12.1] ImportError: /tmp/tmp4YcJD2-sage-git-temp-21524/local/lib/python2.7/site-packages/numpy/linalg/lapack_lite.so: undefined symbol: zungqr_

[saraedum]

Work Issues: merge conflicts

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

3f96a65	Use AC_CONFIG_FILES + AC_OUTPUT to generate build/make/Makefile
14c2577	Further rewrite of configure.ac to do away with the multiple

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from db428a3 to 14c2577

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 14c2577 to 3ba565f

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

3c16cc9	Use AC_CONFIG_FILES + AC_OUTPUT to generate build/make/Makefile
3ba565f	Further rewrite of configure.ac to do away with the multiple

[embray]

Changed work issues from merge conflicts to none

[embray]

Changed dependencies from #21479 to none

[jdemeyer]

comment:17

Why? This makes the Makefile much harder to understand for no obvious benefit...

[jdemeyer]

comment:18

Personally, I find this horrible. I highly value the debuggability of an explicit Makefile.

Of course, that's a very subjective thing so I'm keeping the needs_review status.

[jdemeyer]

Description changed:

--- 
+++ 
@@ -1,3 +1 @@
-`build/make/Makefile` should be written within an AC_CONFIG_FILE template, not at configure time. 
-
-Marking this as dependent on #21479 just because that ticket changes the same part of `configure.ac`.
+`build/make/Makefile` should be written by `config.status` within an `AC_CONFIG_FILE` template, not at `configure` time.

[embray]

comment:19

I explained all of this in my e-mail...

[jdemeyer]

comment:20

Allow me to reboot this discussion in a hopefully constructive way... what would you think of using some high-level programming like Python to write the build/make/Makefile? That would solve the problem that configure.ac is a slow mess. On the other hand, we could write actual Makefile rules instead of using macros.

[saraedum]

comment:21

I think the proposed changes are an improvement. I find the proposed setup easier to read, understand, and modify than the previous echos in configure. It also feels closer to the intended way of using autoconf. I am not sure if the resulting Makefile is harder to read, I don't really see why, in any case, I can see why we would like to get rid of the echo hack which makes it quite annoying to edit configure imho.

Some comments (apart from these minor issues, I would not mind setting this to positive review):

• I might be totally misunderstanding something but the following comment does not actually apply to the Makefile.in, does it?

+# This file has been automatically generated by
+#   /home/embray/src/sagemath/sage/config.status
+# You should not edit it by hand

I think that some general comment about sage-the-distribution would be nice here and some comment on what transforms this file to which other file.

• I don't really understand the following comment

+    # If $need_to_install_{PKG_NAME} is set to no, then set inst_<pkgname> to
+    # some dummy file to skip the installation.

So, I guess `pkgname` and `PKG_NAME` are the same? But where is `inst_<pkgname>` being set below? (Only indirectly in `Makefile.in`, right?)

• I don't see where you are obscuring anything here. (You copied it over and it should probably go into the .in file.)

+    # Below we have to obscure A""M_V_at, or autoconf would complain:
+    # "error: possibly undefined macro: A""M_V_at"

• I would like the configure.ac to contain a # comment at the top which explains how this all works, in particular how this differs from a standard autoconf/automake setup.

[saraedum]

Reviewer: Julian Rüth

[saraedum]

comment:22

I think this should be for followup ticket. It's taken about half a year to get from Erik's commits to somebody reviewing this. If we do a complete rewrite of this, I think this is going to take forever. (Btw, you agreed to fix up #20382 11 months ago. It's easy to get distracted and not move at all…. We should not strive for absolute perfection here but go with improvements that introduce good code. And then get better in the next iteration…)

Replying to @jdemeyer:

Allow me to reboot this discussion in a hopefully constructive way... what would you think of using some high-level programming like Python to write the build/make/Makefile? That would solve the problem that configure.ac is a slow mess. On the other hand, we could write actual Makefile rules instead of using macros.

[embray]

comment:23

Replying to @jdemeyer:

Allow me to reboot this discussion in a hopefully constructive way... what would you think of using some high-level programming like Python to write the build/make/Makefile? That would solve the problem that configure.ac is a slow mess. On the other hand, we could write actual Makefile rules instead of using macros.

It's funny you suggest that. I've actually been working off-and-on in my spare time on my own Python implementation of make. It has a name but until I feel ready to post the code it's a secret :-) It supports POSIX-compatible Makefiles but it also has its own pure-Python format that lets you write make rules in Python, including all the high-level constructs thereof.

In the meantime I don't really see how that would solve anything though.

[jdemeyer]

comment:24

Replying to @embray:

In the meantime I don't really see how that would solve anything though.

It would solve exactly the same issues as the ones that you are fixing in your branch, just in a different way.

[saraedum]

Work Issues: minor documentation issues

[jdemeyer]

comment:26

Replying to @embray:

It's funny you suggest that. I've actually been working off-and-on in my spare time on my own Python implementation of make.

I would be more interested in the converse: turn any Python setup.py into Makefile rules.

[embray]

comment:27

It would still be much slower just by involving Python. It would still be an (actual!) abuse of autoconf.

[embray]

comment:28

Replying to @saraedum:

Some comments (apart from these minor issues, I would not mind setting this to positive review):

• I might be totally misunderstanding something but the following comment does not actually apply to the Makefile.in, does it?

+# This file has been automatically generated by
+#   /home/embray/src/sagemath/sage/config.status
+# You should not edit it by hand

Oops, I'm not sure how that got there since I wrote this six months ago :) It might be that I started from editing a generated file exactly as it says not to do. But let me double-check that config.status isn't actually putting that there because that would be a bug, and a bad one at that.

I think that some general comment about sage-the-distribution would be nice here and some comment on what transforms this file to which other file.

That would help, though I'm not sure where to put it. It would be nice to have some documentation about though, as it took me a while to wrap my head around too. It's a bit hard to follow that the top-level Makefile calls build/make/install which calls make again but with build/make/Makefile, which in turn calls sage-spkg to actually build/install packages. I don't have too much of a problem with that aspect of the process but it's not obvious.

• I don't really understand the following comment

+    # If $need_to_install_{PKG_NAME} is set to no, then set inst_<pkgname> to
+    # some dummy file to skip the installation.

So, I guess `pkgname` and `PKG_NAME` are the same? But where is `inst_<pkgname>` being set below? (Only indirectly in `Makefile.in`, right?)

Yes, maybe this could be spelled more consistently.

This definitely deserves better documentation (most of how this works is carried over from the existing Makefile--I really didn't change much...) The actual physical (as in, a real file) targets associate with each package are the "stamp" files installed to $SAGE_LOCAL/var/lib/sage/installed/<pkg_name>-<pkg_version>. Within the Makefile the paths to these files are stored in variables named $(inst_<pkg_name>).

What this is referring to is that for packages which are not actually installed into Sage (e.g. we use the system version), instead of pointing to a real stamp file, the $(inst_<pkg_name>) just points to a dummy file indicating "yes, this dependency is satisfied". This way we don't need to remove that package as a dependency of any other package. It's just trivially satisfied.

Anyways, I'll see if I can make that comment clearer.

• I don't see where you are obscuring anything here. (You copied it over and it should probably go into the .in file.)

+    # Below we have to obscure A""M_V_at, or autoconf would complain:
+    # "error: possibly undefined macro: A""M_V_at"

Carried over from the original configure.ac. It's not relevant now anywhere so can just be deleted.

• I would like the configure.ac to contain a # comment at the top which explains how this all works, in particular how this differs from a standard autoconf/automake setup.

Okay, maybe that's a decent place to put it...

[orlitzky]

comment:30

While I agree that the templates are a bit incomprehensible, overall I like this change. The separation between configure.ac and Makefile.in makes it more clear what the build is trying to do, with less of a focus on how it's done. I think that in the future, it will be easier to replace chunks of either file, because we won't have to worry about unintended side effects buried in the shell code.

Other ideas:

1. Our Makefile.in is now GNU-Make-specific. Should we call it GNUMakefile.in to avoid insane errors on systems where "make" isn't GNU?
2. We should detect SHELL rather than setting it to /bin/bash, because there are systems where it lives in e.g. /usr/bin/bash (https://www.freedesktop.org/wiki/Software/systemd/TheCaseForTheUsrMerge/)
3. Is Bash needed for the makefile at all? If we replace "source" with ".", does anything else crash with POSIX sh?

[jdemeyer]

comment:31

Replying to @embray:

It would still be much slower just by involving Python.

Why do you think that? It just needs to read of bunch of files and do some string manipulation. It should be easy.

It would still be an (actual!) abuse of autoconf.

I'm not denying that...

Still, I think it would be fast, work well and produce readable output. So I think that you should at least seriously consider it.

[embray]

comment:32

Let's give this a try for now (modulo other review comments). I sympathize with your concerns but I strongly suspect that they will prove overstated. If it turns into a concrete problem somehow then I'll reconsider.

[embray]

comment:33

Replying to @orlitzky:

While I agree that the templates are a bit incomprehensible, overall I like this change. The separation between configure.ac and Makefile.in makes it more clear what the build is trying to do, with less of a focus on how it's done. I think that in the future, it will be easier to replace chunks of either file, because we won't have to worry about unintended side effects buried in the shell code.

I'm going to try to do a bit of reformatting to make it a little easier to read.

Other ideas:

1. Our Makefile.in is now GNU-Make-specific. Should we call it GNUMakefile.in to avoid insane errors on systems where "make" isn't GNU?

One thing I'll point out that I haven't explicitly mentioned yet: I asked a few people about this before doing the work (I forget who) but at the time everyone seemed to agree that it would be fine to require GNU make, as there are other packages in Sage that require it, so building the full Sage-the-distribution doesn't work with any other make implementation anyways.

I suppose we could do that though.

2. We should detect SHELL rather than setting it to /bin/bash, because there are systems where it lives in e.g. /usr/bin/bash (https://www.freedesktop.org/wiki/Software/systemd/TheCaseForTheUsrMerge/)

Right; originally we had SHELL = `command -v bash` here. I'll fix that.

3. Is Bash needed for the makefile at all? If we replace "source" with ".", does anything else crash with POSIX sh?

I'm not sure it's necessarily strictly needed for the makefile itself, but it in turn calls scripts that require bash, so I don't see any advantage in not just forcing bash in the makefile as well.

[embray]

comment:34

One other issue I noticed while working on cleaning this up and adding better debugging tools, is that this doesn't currently handle order-only prerequisites correctly. This was definitely obscured before, but will be easy to fix.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 3ba565f to 9ce84c8

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

0d12016	Remove some bogus comments
9ce84c8	Improved (?) documentation of the Makefile.in template.

[embray]

comment:36

Last commit made the templates look a bit less noisy. For example, before:

define NORMAL_PACKAGE_templ
$$(INST)/$(1)-$$(vers_$(1)): $$(foreach dep,$$(deps_$(1)),$$(inst_$$(dep)))
    +$(AM_V_at)sage-logger -p '$(SAGE_SPKG) $(1)-$$(vers_$(1))' '$(SAGE_LOGS)/$(1)-$$(vers_$(1)).log'

$(1): $$(INST)/$(1)-$$(vers_$(1))

$(1)-clean:
    rm -rf $$(INST)/$(1)-$$(vers_$(1))
endef

and after:

define NORMAL_PACKAGE_templ
$$(INST)/$(1)-$(2): $(3)
    +$(AM_V_at)sage-logger -p '$$(SAGE_SPKG) $(1)-$(2)' '$$(SAGE_LOGS)/$(1)-$(2).log'

$(1): $$(INST)/$(1)-$(2)

$(1)-clean:
    rm -rf $$(INST)/$(1)-$(2)
endef

I also added the DEBUG_RULES flag that I mentioned on sage-devel. If you run the generated makefile in dry-run mode with this flag defined you can see exactly what the templates are outputting:

$ make -f build/make/Makefile -n DEBUG_RULES=1
# Rules for standard packages
$(INST)/4ti2-1.6.7:   $(inst_zlib)  $(inst_mpir)  $(inst_glpk)
        +sage-logger -p '$(SAGE_SPKG) 4ti2-1.6.7' '$(SAGE_LOGS)/4ti2-1.6.7.log'

4ti2: $(INST)/4ti2-1.6.7

4ti2-clean:
        rm -rf $(INST)/4ti2-1.6.7

$(INST)/alabaster-0.7.10:   $(inst_python2)  |  $(inst_pip)
        +sage-logger -p '$(SAGE_SPKG) alabaster-0.7.10' '$(SAGE_LOGS)/alabaster-0.7.10.log'
...

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

fc063dc

Get SHELL from autoconf

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 9ce84c8 to fc063dc

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from fc063dc to d17e494

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

d796901	Delete redundant AC_CONFIG_MACRO_DIR
d17e494	Added a little bit more information about the build system and how it works, at a high level

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from d17e494 to 7668fcd

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

7668fcd

Fix sage -i

[jdemeyer]

comment:40

Replying to @embray:

I sympathize with your concerns but I strongly suspect that they will prove overstated.

I really hope that you are right and that this won't turn in a "I told you so" scenario.

I'll stop protesting for now. There is not much to discuss further anyway, we each know the other's opinions.