Make arb a standard package

[mezzarobba]

Arb has been an optional package for a few months. This is a proposal to fast-track it to standard, as suggested in <c02d759d-9a81-4c85-a110-a2e2ffd679dc@googlegroups.com>.

See also: #17218, #17194, #16747

Depends on #19280 Depends on #19443

CC: @cheuberg @fredrik-johansson @simon-king-jena

Component: packages: standard

Author: Marc Mezzarobba, Clemens Heuberger

Branch/Commit: ea5741f

Reviewer: Clemens Heuberger, Marc Mezzarobba

Issue created by migration from https://trac.sagemath.org/ticket/18546

[mezzarobba]

Changed commit from 4e1af06 to 4dbae98

[mezzarobba]

Changed branch from u/cheuberg/18546-arb-std to u/mmezzarobba/18546-arb-std

[vbraun]

comment:38

Merge conflict

[cheuberg]

comment:39

Merged 6.10.beta0 (conflict was with #19308). This introduces lots of blank space diff, so please diff with git diff -b to see the actual changes.

---

New commits:

03a7a4f

Merge tag '6.10.beta0' into t/18546/18546-arb-std

[cheuberg]

Changed commit from 4dbae98 to 03a7a4f

[cheuberg]

Changed branch from u/mmezzarobba/18546-arb-std to u/cheuberg/18546-arb-std-rebased

[mezzarobba]

comment:40

going back to my branch because I also rebased the follow-up tickets

---

New commits:

e2d393b

Merge 6.10.beta0 into arb-std

[mezzarobba]

Changed branch from u/cheuberg/18546-arb-std-rebased to u/mmezzarobba/18546-arb-std

[mezzarobba]

Changed commit from 03a7a4f to e2d393b

[cheuberg]

comment:41

I do not understand why you rebase at such a late stage with a chain of tickets depending on this here.

[cheuberg]

comment:42

anyhow, the two branches contain the same code, so I set this back to positive.

[mezzarobba]

comment:43

Replying to @cheuberg:

I do not understand why you rebase at such a late stage with a chain of tickets depending on this here.

I'm doing it because there are other tickets depending on this one, and IMO adding merges everywhere makes their history less readable. But I can do merges instead if you prefer...

[cheuberg]

comment:44

IMHO, rebasing destroys history and requires much more work, e.g., you have to merge the rebased ticket into all depending tickets, while a merge might just work without additional intervention.

Luckily, I never merged #18546 into #17220 and its dependencies; otherwise I'd have to remerge it again (although nothing relevant changed) and would have troubles in my private "develop" branch where I collect all tickets I authored in order to detect conflicts between apparently unrelated tickets early on.

Thus for me, rebasing something where somebody else might have based something (e.g., something pushed to a trac ticket) is something to avoid if at all possible.

[jdemeyer]

comment:45

Replying to @cheuberg:

Thus for me, rebasing something where somebody else might have based something (e.g., something pushed to a trac ticket) is something to avoid if at all possible.

I agree, except that I would replace the condition "something pushed to a trac ticket" by "something pushed to a trac ticket and set to needs_review". I often push work-in-progress to Trac and keep rewriting history until it's ready for review.

[mezzarobba]

comment:46

Replying to @jdemeyer:

Replying to @cheuberg:

Thus for me, rebasing something where somebody else might have based something (e.g., something pushed to a trac ticket) is something to avoid if at all possible.

I agree, except that I would replace the condition "something pushed to a trac ticket" by "something pushed to a trac ticket and set to needs_review".

For me, rebasing on top of the last release or beta is okay if not desirable even in this case. (In particular, please feel free to do it when you know I'm on the other end.) But I'll refrain from doing it if you don't like it.

[vbraun]

comment:47

Doesn't actually build arb, maybe you need to bump the patchlevel as well?

http://build.sagedev.org/release/builders/%20%20slow%20AIMS%20%20%28Debian%208%2064%20bit%29%20incremental/builds/143/steps/compile/logs/stdio

[cheuberg]

comment:48

Replying to @vbraun:

Doesn't actually build arb, maybe you need to bump the patchlevel as well?

http://build.sagedev.org/release/builders/%20%20slow%20AIMS%20%20%28Debian%208%2064%20bit%29%20incremental/builds/143/steps/compile/logs/stdio

Bumping the patchlevel does not help.

[jdemeyer]

comment:49

You need to add arb as dependency for the Sage library in build/make/deps.

[cheuberg]

comment:50

Replying to @jdemeyer:

You need to add arb as dependency for the Sage library in build/make/deps.

I'll try that soon (currently, I have no sage available due to recompilation).

What I tried up to now:

1. I checked out this ticket, ran make distclean and then make. This resulted in a version of sage with arb.
2. I checkout out plain 6.10.beta0, ran make distclean and then make, obviously resulting in a version without arb.
3. I checked out this ticket, ran ./configure and make. Now everything is recompiling (including arb), however, git diff does show empty output, so while this would solve the problem for me, I cannot push it to the ticket.

What I do not understand: why is it necessary to add arb to build/make/deps for an existing version while it is unnecessary for a fresh install or for people running ./configure?

[jdemeyer]

comment:51

Replying to @cheuberg:

What I do not understand: why is it necessary to add arb to build/make/deps for an existing version while it is unnecessary for a fresh install or for people running ./configure?

Probably the fact that arb starts with the letter a. Anyway, there is no point in analysing, the fix is obvious.

[cheuberg]

comment:52

Replying to @jdemeyer:

Probably the fact that arb starts with the letter a. Anyway, there is no point in analysing, the fix is obvious.

Unfortunately, it is not obvious to me:

Following Volker's recommendation, I increased the patchlevel, following your recommendation, I added to build/make/deps, but arb is still not built.

diff --git a/build/make/deps b/build/make/deps
index 3d0fb93..27f488d 100644
--- a/build/make/deps
+++ b/build/make/deps
@@ -100,6 +100,7 @@ base: $(INST)/$(BZIP2) $(INST)/$(PATCH) $(INST)/$(PKGCONF)
 # Sage library (e.g. CYTHON, JINJA2), and on the other hand all
 # dependencies for Cython files (e.g. PARI, NTL, SAGE_MP_LIBRARY).
 sagelib: \
+               $(INST)/$(ARB) \
                $(INST)/$(ATLAS) \
                $(INST)/$(CEPHES) \
                $(INST)/$(CLIQUER) \
diff --git a/build/pkgs/arb/package-version.txt b/build/pkgs/arb/package-version.txt
index 8f0f703..67b2238 100644
--- a/build/pkgs/arb/package-version.txt
+++ b/build/pkgs/arb/package-version.txt
@@ -1,2 +1,2 @@
-2.6.0
+2.6.0.p0

[jdemeyer]

comment:53

Did you remember to run ./configure first? If yes and it still doesn't work, please push your non-working branch (with the change to build/make/deps) and I'll have a look.

There is absolutely no need to change the patchlevel if you don't change the package.

[cheuberg]

comment:54

Replying to @jdemeyer:

Did you remember to run ./configure first?

Running ./configure solves the problem locally, but does not change any files under version control, thus will not solve the problem for anybody else.

If yes and it still doesn't work, please push your non-working branch (with the change to build/make/deps) and I'll have a look.

Pushed and thanks for having a look.

There is absolutely no need to change the patchlevel if you don't change the package.

ok, this is not part of the pushed branch.

---

Last 10 new commits:

0f4e373	Revert "Trac #19280: add fix to all x86 gmp-mparam"
f780291	Trac #19280: Use patch proposed in [#19280 comment:5](https://github.com/sagemath/sage/issues/19280#comment:5)
8cf7d18	Merge trac/u/cheuberg/mpir-bug into 6.9
ef257db	#18546 Make arb a standard package
a7a41bc	#18546 Remove 'optional - arb' doctest flags
0e2e8ea	#18546 Include modules depending on arb in the reference manual
76883ad	Trac #18546: remove optional - arb in real_mpfi
4dbae98	Trac #18546: make doctests 32-bit aware
e2d393b	Merge 6.10.beta0 into arb-std
ea5741f	Trac #18546: Add ARB as a dependency of sage

[cheuberg]

Changed branch from u/mmezzarobba/18546-arb-std to u/cheuberg/18546-arb-std-3

[cheuberg]

Changed commit from e2d393b to ea5741f

[jdemeyer]

comment:55

Replying to @cheuberg:

Running ./configure solves the problem locally, but does not change any files under version control, thus will not solve the problem for anybody else.

But ./configure is run automatically whenever the Sage version changes. So, if this ticket is merged in the next beta, there will not be problem since the Sage version will also change.

[vbraun]

comment:56

Doesn't build so I can't test it on the buildbot

http://build.sagedev.org/release/builders/%20%20fast%20Volker%20MiniMac%20%28OSX%2010.10%20x86_64%29%20incremental/builds/501/steps/compile/logs/stdio

[jdemeyer]

comment:57

@vbraun: would it be possible to either:

1. install autotools on the buildbot
2. build from an sdist instead of from git on the buildbot to avoid these kinds of problems.

[jdemeyer]

comment:58

Hang on, configure.ac didn't change, so autotools aren't needed here.

Do the buildbots run ./configure before building? If not, they should.

[vbraun]

comment:59

They don't run configure, and thats not in http://doc.sagemath.org/html/en/installation/source.html. I don't mind adding it but the docs should be updated first then.

[jdemeyer]

comment:60

It is documented

[vbraun]

comment:61

I'm against different instructions for build-from-source and build-from-source-with-ticket, thats unnecessary. And the buildbot can realistically only test one version anyways, so its save to assume that the other version is going to be broken. Also, not all buildbots have autotools (e.g. OSX doesn't).

[jdemeyer]

comment:62

So what would you propose then? This isn't going to be the last ticket with this problem.

See #19432 for a documentation improvement.

[vbraun]

comment:63

Basically we have to decide between

• Create the makefile rules at configure-time. Then running configure is mandatory; It may work without configure, but there can't be any guarantee
• Create the makefile rules from within make(*), using an include and auto-reload of changed includes. We previously used the latter, why did we have to change that? The rules aren't configurable, they just reflect the package types and dependencies.

(*) possibly by calling configure, though I'd prefer something more straightforward

[jdemeyer]

comment:65

Replying to @vbraun:

We previously used the latter, why did we have to change that?

1. the new "install packages with dependencies" means that ./sage -i PKGNAME needs a lot more time (running ./configure takes some time). In particular, Simon King complained about this.

2. ./configure && make is the standard way to build packages.

The rules aren't configurable, they just reflect the package types and dependencies.

That's something which can (and should) change in the future. It would be cool to run for example

./configure --with-system-python --with-package=valgrind --enable-check=yes

[jdemeyer]

comment:66

From now on, every ticket which tries to add a standard package will get set back to needs_work forever because the buildbot is broken?

[vbraun]

comment:67

Thats partly why I said that rerunning configure isn't the most straightforward way of updating the rules.

Configure switches can just override rules (e.g. using that explicit rules override pattern rules); that doesn't prevent us from writing a makefile that works by default.

I agree that ./configure && make is the standard way but this is not what Sage used to do so its more of a social problem to a) agree on changing it and b) communicating that change. Write to sage-devel.

I don't think the buildbot is broken, it just does what the docs say and our users have learned to rely on. The alternative to fixing the configure/make system is always updating the confball if you change the set of standard packages.

[jdemeyer]

comment:68

Replying to @vbraun:

Write to sage-devel.

That already happened when the change was made.

it just does what the docs say

Building Sage, then checking out a specific git branch and then rebuilding is not a normal build from source. So I don't think the installation manual applies here. What does apply is the developer's guide which does mention ./configure (see [comment:60]).

The alternative to fixing the configure/make system is always updating the confball if you change the set of standard packages.

Updating the confball is not going to help if you don't even run ./configure.

[jdemeyer]

comment:69

Volker, a different solution would be to change the Sage version number in the branch you test on the buildbots. That should force a reconfiguration and is closer to a user build.

[vbraun]

comment:70

Of course I can bump the version with each closed ticket but thats a bit silly....

[83660e46-0051-498b-a8c1-f7a7bd232b5a]

comment:71

Replying to @vbraun:

Of course I can bump the version with each closed ticket but thats a bit silly....

That's of course not what he meant.

Increasing the Sage version when testing with tickets not yet merged into a public release seems natural, as you're doing so in order to prepare the next release, aren't you?

[vbraun]

comment:72

Most of the work is figuring out which tickets merge cleanly and pass tests. Actually bumping the version is trivial and I only do that at the end of the release process. Its not always clear at the outset what the next version will be, another beta or rc or release?

[83660e46-0051-498b-a8c1-f7a7bd232b5a]

comment:73

Replying to @vbraun:

Most of the work is figuring out which tickets merge cleanly and pass tests. Actually bumping the version is trivial and I only do that at the end of the release process. Its not always clear at the outset what the next version will be, another beta or rc or release?

So it doesn't make much of a difference whether you change it at the beginning or at the end; in the rare cases where the next version won't be betaN+1 you could fix it again at the end, before releasing. What the temporary, internal version number for testing actually is doesn't really matter much.

[vbraun]

comment:74

You are proposing a more complicated release process without technical advantage, and which introduces additional differences between whats tested on the buildbot vs. what users end up doing. Sure it can be done, but why?

[83660e46-0051-498b-a8c1-f7a7bd232b5a]

comment:75

The problem here is that you're now actually testing in a different way or in a different setting than users will (or would) when the ticket got merged and a next version released.

[vbraun]

comment:76

At least I'm just running the same "make" command that users are going to use. How is only tesing "configure && make" ever going to give better test coverage?

No amount of testing can guard against misuses of the version stamp, but then onbody knows what the next version stamp is going to be beforehand.

[mezzarobba]

comment:77

I'm lost: could someone please explain what work this ticket needs?

[vbraun]

Changed dependencies from #19280 to #19280, #19443

[vbraun]

comment:78

I've opened #19443

[vbraun]

Changed branch from u/cheuberg/18546-arb-std-3 to ea5741f