Add sage-apply-patches helper script for use in spkg-install scripts

[embray]

This adds a new script to build/bin called sage-apply-patches which implements the same code that is copy/pasted through many of the spkg-install scripts (albeit often inconsistently). The patches in build/pkgs/PKGNAME/patches are now applied automatically.

I'm not attached to any of the details of how the script works--the implementation was designed mostly to make it easiest to automatically replace the existing repeated code snippet with minimal effort.

Component: build

Author: Erik Bray

Branch: b6b33cd

Reviewer: Jeroen Demeyer

Issue created by migration from https://trac.sagemath.org/ticket/20692

[jdemeyer]

comment:47

And since you rewrote git history anyway, you might as well squash everything to 1 commit.

[embray]

comment:48

The change in openblas is a mistake made in resolving a merge conflict; will revert.

[embray]

comment:49

Replying to @jdemeyer:

From reading the documentation in sage-apply-patches, I don't quite understand the meaning of the [-d patch-subdir] and [patch-dir] arguments. Are these arguments related? If yes, why is it not just 1 argument?

I think earlier, before #20837, it was more necessary to have separate arguments. But I still kind of like it as is.

The assumption here is that there is a standard root for patches to a package with may have subdirectories. The reason it's useful to have separate arguments is that spkg-install scripts don't need to care where the patch root is, but sometimes it's useful to select subdirectories within it from which to apply other patches.

I'm open to other ideas though.

[embray]

comment:50

Replying to @jdemeyer:

Also, you are deleting rubiks patches...

Looks like that's left over from before #21103--it wasn't intentional. What's odd is that everything worked fine without them so now I question whether they're even necessary but I won't think about that for now.

[jdemeyer]

comment:51

As you probably know, I absolutely hate needless complexity. I see no reason to have two arguments doing essentially the same thing. In your current branch, both kinds of arguments are used, but all use cases could be done with either the -d patch-subdir argument or the patch-dir argument. So just pick one.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

b058fae

Adds a new sage-apply-patches script.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 5343f3b to b058fae

[embray]

comment:53

I reverted the accidental changes and squashed. I also want to add documentation about this to the developer docs for maintaining packages.

[embray]

comment:54

Replying to @jdemeyer:

As you probably know, I absolutely hate needless complexity. I see no reason to have two arguments doing essentially the same thing. In your current branch, both kinds of arguments are used, but all use cases could be done with either the -d patch-subdir argument or the patch-dir argument. So just pick one.

I see what you mean about using it inconsistently--well really there's one case of that in singular. There's a reason for that though. The way the spkg-install for singular is written (talk about "needless complexity") is it has a bunch of sub-routine that executes in a loop. Before each subroutine it cd's in to the src/ directory (of the singular sources, so from the point of view of spkg-install that's src/src which is different from the default assumption made in sage-apply-patches.

Really if I were using sage-apply-patches as I intended here, it should call sage-apply-patches -d debug "$PATCH" :) (though it looks like I cd .. first, so I guess the $PATCH part can go away, maybe...)

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from b058fae to f92d0ef

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

f92d0ef

Use the -d argument to sage-apply-patches instead

[embray]

comment:56

As for the arguments I think I'll leave them as is. As of now the patch-dir argument is not used explicitly which is good, as it implies uniformity across the spkg-installs (previously there was less uniformity and I think both use cases were used).

That said, it would be more work now to remove it than to leave it as is, as it is completely harmless. And not having it limits the script's applicability to other problems that may arise in the future. I do not think it is "needless complexity".

[embray]

comment:58

Replying to @embray:

As for the arguments I think I'll leave them as is. As of now the patch-dir argument is not used explicitly which is good, as it implies uniformity across the spkg-installs (previously there was less uniformity and I think both use cases were used).

That said, it would be more work now to remove it than to leave it as is, as it is completely harmless. And not having it limits the script's applicability to other problems that may arise in the future. I do not think it is "needless complexity".

(FWIW if I were to choose one to remove it would be patch-dir since now it's not used.)

Also setting back to needs_work because I still need to update the documentation.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from f92d0ef to ed68c76

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

928528c	Adds a new sage-apply-patches script.
5f76324	Use the -d argument to sage-apply-patches instead
ed68c76	Added some updated documentation about how to apply patches

[embray]

comment:60

Rebased again, and updated the docs. Can we please move forward on merging this before I have to rebase again? If you really want to change something about sage-apply-patches we can do that separately, but this works now as-is.

[jdemeyer]

comment:61

Note to self: double-check fricas, frobby, gap3, gfan, scipoptsuite

[jdemeyer]

comment:62

This conflicts with #17254. I suggest that you merge that branch.

[jdemeyer]

Changed dependencies from #21103 to #17254

[jdemeyer]

comment:63

Also, since you have to change this branch anyway: could you remove the line echo "Applying Sage-specific patches" from sage-spkg? That is not needed since sage-apply-patches is sufficiently verbose. That message could actually be confusing if there are no patches to apply.

[jdemeyer]

comment:64

Another detail: the pushd/popd is also verbose:

[gsl-2.1] Finished extraction
[gsl-2.1] Applying Sage-specific patches
[gsl-2.1] /usr/local/src/sage-git/local/var/tmp/sage/build/gsl-2.1/src /usr/local/src/sage-git/local/var/tmp/sage/build/gsl-2.1
[gsl-2.1] No patch files found in ../patches
[gsl-2.1] /usr/local/src/sage-git/local/var/tmp/sage/build/gsl-2.1
[gsl-2.1] ****************************************************

This should be silenced

[jdemeyer]

Changed dependencies from #17254 to #17254, #21721

[jdemeyer]

comment:65

Some packages still use non-standard patching: #21721

[jdemeyer]

comment:66

In build/pkgs/scipoptsuite/spkg-install, you are still using both directory arguments:

sage-apply-patches -d scip $SPKGDIR/patches || exit 1

This could be

sage-apply-patches "$SPKGDIR/patches/scip" || exit $?

[jdemeyer]

Changed dependencies from #17254, #21721 to #17254, #21721, #21722

[jdemeyer]

comment:67

gap3 had more issues, so I created a separate ticket: #21722

[jdemeyer]

Description changed:

--- 
+++ 
@@ -1,3 +1,3 @@
-This adds a new script to `build/bin` called `sage-apply-patches` which implements the same code that is copy/pasted through many of the `spkg-install` scripts (albeit often inconsistently).  It also updates those `spkg-install`s to use it where appropriate.
+This adds a new script to `build/bin` called `sage-apply-patches` which implements the same code that is copy/pasted through many of the `spkg-install` scripts (albeit often inconsistently). The patches in `build/pkgs/PKGNAME/patches` are now applied automatically.

 I'm not attached to any of the details of how the script works--the implementation was designed mostly to make it easiest to automatically replace the existing repeated code snippet with minimal effort.

[jdemeyer]

Changed dependencies from #17254, #21721, #21722 to #21721, #21722

[jdemeyer]

comment:69

All dependencies are merged now, so please rebase to latest 7.5.beta1.

[jdemeyer]

Changed dependencies from #21721, #21722 to none

[jdemeyer]

Dependencies: #21676, #21792

[embray]

comment:71

All dependencies are not merged now....

[jdemeyer]

comment:72

Replying to @embray:

All dependencies are not merged now....

No, because I added two dependencies which would otherwise conflict.

[jdemeyer]

comment:73

You could just merge those two dependencies in this branch.

[mkoeppe]

comment:74

needs rebasing

[jdemeyer]

Changed dependencies from #21676, #21792 to none

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from ed68c76 to 0968ec0

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

f187f9c	Adds a new sage-apply-patches script.
0968ec0	Added some updated documentation about how to apply patches

[embray]

comment:77

Rebased again.

[embray]

comment:79

Sorry--misclicked.

[jdemeyer]

comment:80

Please fix [comment:63], [comment:64], [comment:66]. Those are trivial things which can easily be fixed.

[embray]

comment:81

The way sage-apply-patches is being invoked in scipoptsuite/spkg-install is actually exactly how those options are meant to be used but whatever.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 0968ec0 to addd3ee

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

e842689	Adds a new sage-apply-patches script.
02e8b04	Added some updated documentation about how to apply patches
addd3ee	Reduce verbosity--no need for the 'Applying' message, and no real need to use pushd/popd either since sage-apply-patches does not change directories.

[embray]

New commits:

e842689	Adds a new sage-apply-patches script.
02e8b04	Added some updated documentation about how to apply patches
addd3ee	Reduce verbosity--no need for the 'Applying' message, and no real need to use pushd/popd either since sage-apply-patches does not change directories.

[jdemeyer]

comment:84

I have no further objections. I'm currently building Sage from scratch with this branch. If this works fine, I will set this to positive_review (hoping that there are no conflicts).

[embray]

comment:85

Awesome, thanks!

[vbraun]

comment:87

Merge conflict

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

ad5dcd6	Adds a new sage-apply-patches script.
2858b5f	Added some updated documentation about how to apply patches
52a8ca0	Reduce verbosity--no need for the 'Applying' message, and no real need to use pushd/popd either since sage-apply-patches does not change directories.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from addd3ee to 52a8ca0