Put more files under revision control.

jhpalmieri commented 14 years ago

Put the text files in $SAGE_ROOT, and also the text files in spkg, under revision control. (See the discussion at the end of #9351.)

Here are the instructions:

apply attachment: trac_9433-sage-repo.2.patch
apply attachment: trac_9433-scripts.v9.patch to the scripts repository
move the attached file attachment: hgignore to $SAGE_ROOT/.hgignore (note that this is a new file)
move the attached file attachment: root-spkg-install.v4 to $SAGE_ROOT/spkg/root-spkg-install (note that this is a new file)
apply attachment: 9433_install.diff to $SAGE_ROOT/spkg/install
apply attachment: 9433_deps.diff to $SAGE_ROOT/spkg/standard/deps

Then from $SAGE_ROOT, run the attached script attachment: 9433_hg_script.sh to create the Mercurial repository.

Testing: see http://sage.math.washington.edu/home/release/sage-4.7.alpha0/

CC: @williamstein @dandrake @kcrisman @nexttime

Component: distribution

Author: John Palmieri

Reviewer: Leif Leonhardy, Volker Braun, Jeroen Demeyer

Merged: sage-4.7.alpha0

Issue created by migration from https://trac.sagemath.org/ticket/9433

jdemeyer commented 13 years ago

comment:127

The file spkg-install from the root repo is not under revision control. Is there a reason for this?

jdemeyer commented 13 years ago

The file $SAGE_ROOT/spkg/root-spkg-install

vbraun commented 13 years ago

Attachment: root-spkg-install.v3.gz

create root repository without dotencode

vbraun commented 13 years ago

comment:128

Attachment: 9433_hg_script.2.sh.gz

I'm pretty sure that the root-spkg-install ought to be under revision control. I'm travelling right now, so I can't test it out myself.

jhpalmieri commented 13 years ago

comment:129

I don't think there is anything in the repository which should require a new version of Mercurial. According to this page from the Mercurial wiki, if you create a repo with a newer version and then try to access it with an older version, you can see this message. So I don't think it's anything specific about the root repo.

I'll try to look into the root-spkg-install file issue.

jhpalmieri commented 13 years ago

comment:130

Okay, line 24 in 9433_hg_script.sh is

( cd spkg && hg add README.txt gen_html install pipestatus root-spkg-install )

So it looks like root-spkg-install should be tracked. Can you explain why you think it's not?

Also, I see that vbraun has a new version of this script which might avoid the dotencode issue...

vbraun commented 13 years ago

comment:131

Yes, I tripped over the dotencode thing before when working with mercurial. We don't really need that, so its ok to switch it off for backward compatibility, at least for the next year or so.

jhpalmieri commented 13 years ago

the file SAGE_ROOT/.hgignore, now including spkg-install

jhpalmieri commented 13 years ago

Description changed:

--- 
+++ 
@@ -9,6 +9,6 @@
 - apply [attachment: 9433_install.diff](https://github.com/sagemath/sage-prod/files/10649960/9433_install.diff.gz) to `$SAGE_ROOT/spkg/install`
 - apply [attachment: 9433_deps.diff](https://github.com/sagemath/sage-prod/files/10649966/9433_deps.diff.gz) to `$SAGE_ROOT/spkg/standard/deps`

-Then **from $SAGE_ROOT**, run the attached script [attachment: 9433_hg_script.sh](https://github.com/sagemath/sage-prod/files/10649963/9433_hg_script.sh.gz) to create the Mercurial repository.
+Then **from $SAGE_ROOT**, run the attached script [attachment: 9433_hg_script.2.sh](https://github.com/sagemath/sage-prod/files/10649968/9433_hg_script.2.sh.gz) to create the Mercurial repository.

 **Testing**: see [http://sage.math.washington.edu/home/release/sage-4.7.alpha0/](http://sage.math.washington.edu/home/release/sage-4.7.alpha0/)

jhpalmieri commented 13 years ago

comment:132

Attachment: hgignore.gz

Replying to @jdemeyer:

The file spkg-install from the root repo is not under revision control. Is there a reason for this?

I now realize that you're talking about the file spkg-install in the actual spkg file. This is just a copy of root-spkg-install made by sage-make_devel_packages, so we don't need to track it. I'm going to add it to the .hgignore file (by adding ^spkg-install$, so it only matches a file with exactly that name at the top level). This change requires review.

Meanwhile, I'm giving Volker's change to the hg_script a positive review: for me, it makes any errors about dotencode go away when I use an older version of Mercurial to access the repo.

jdemeyer commented 13 years ago

comment:133

About the dotencode requirement: the change in the shell script indeed "fixes" the actual root repository. However, the created sage-root spkg still has a mercurial repository which requires dotencode (so, after sdist and make, you need a new version of hg). Since none of the other spkgs have this behaviour, I think this still needs work.

jhpalmieri commented 13 years ago

comment:134

I don't really understand this complaint. After all, Sage is distributed with a version of Mercurial, and it is completely reasonable to require that version for repositories which are part of Sage. It's easy enough to type "sage -hg" instead of "hg", or make an alias, or put the Sage version of hg in your $PATH. I think this is more a problem with Mercurial -- why is it not backwards compatible? Who merged this version of Mercurial into Sage? Did the spkg maintainers or release manager notice or discuss this incompatibility issue?

Also, if you're going to start inventing rules about how an spkg should be prepared, you might consider putting those rules in the developer's guide, and perhaps discussing them and getting approval for them on sage-devel before imposing them.

We can add --config format.dotencode=0 at various places in sage-sdist and sage-make_devel_packages, and I'm attaching a new version of the scripts patch to do this, but this is not a viable long-term solution: the version requirements of Mercurial for anyone who wants to make a new spkg for Sage need to be made public, not imposed at anyone's whim.

jhpalmieri commented 13 years ago

Attachment: trac_9433-scripts.v8.patch.gz

patch for scripts repo

jhpalmieri commented 13 years ago

Description changed:

--- 
+++ 
@@ -3,7 +3,7 @@
 Here are the instructions:

 - apply [attachment: trac_9433-sage-repo.2.patch](https://github.com/sagemath/sage-prod/files/10649949/trac_9433-sage-repo.2.patch.gz)
-- apply [attachment: trac_9433-scripts.v7.patch](https://github.com/sagemath/sage-prod/files/10649964/trac_9433-scripts.v7.patch.gz) to the scripts repository
+- apply [attachment: trac_9433-scripts.v8.patch](https://github.com/sagemath/sage-prod/files/10649970/trac_9433-scripts.v8.patch.gz) to the scripts repository
 - move the attached file [attachment: hgignore](https://github.com/sagemath/sage-prod/files/10649969/hgignore.gz) to `$SAGE_ROOT/.hgignore` (note that this is a *new* file)
 - move the attached file [attachment: root-spkg-install.v3](https://github.com/sagemath/sage-prod/files/10649967/root-spkg-install.v3.gz) to `$SAGE_ROOT/spkg/root-spkg-install` (note that this is a *new* file)
 - apply [attachment: 9433_install.diff](https://github.com/sagemath/sage-prod/files/10649960/9433_install.diff.gz) to `$SAGE_ROOT/spkg/install`

jdemeyer commented 13 years ago

comment:136

Replying to @jhpalmieri:

Did the spkg maintainers or release manager notice or discuss this incompatibility issue?

I didn't know about this until looking at this ticket.

Why not simply avoid hg clone in sage-make_devel_packages? If the other spkgs can be made without using hg clone, surely the same should work for the sage_root spkg?

jdemeyer commented 13 years ago

comment:137

Version 1.7.3 of Mercurial was merged in sage-4.6.2.alpha2, ticket #10594. It can still be unmerged if you think that's a good thing to do. Then we would fall back to Mercurial 1.6.4.

jhpalmieri commented 13 years ago

comment:138

Replying to @jdemeyer:

Why not simply avoid hg clone in sage-make_devel_packages? If the other spkgs can be made without using hg clone, surely the same should work for the sage_root spkg?

It certainly could be done, but it makes the repo harder to maintain. Suppose you want to add a new file to the repo. If you clone, you just run "hg add" (or apply a patch which accomplishes this) and you're done. If you manually copy everything over, as is done for the other repos, then you also have to modify sage-make_devel_packages, maybe root-spkg-install, maybe sage-sdist. This is especially true for the root repo, where files need to be dealt with individually: it's not like the Sage repo where except for a handful of files, you just copy over an entire directory (devel/sage/sage/), and it's not like the scripts repo where except for a handful of files, you just copy over everything with a certain name ("sage-*").

Maybe instead it could run "hg manifest" and then manually copy over the listed files. But this seems really awkward when "hg clone" does exactly what is required.

Version 1.7.3 of Mercurial was merged in sage-4.6.2.alpha2, ticket #10594. It can still be unmerged if you think that's a good thing to do. Then we would fall back to Mercurial 1.6.4.

I'm really not sure about this. Perhaps it should be discussed on #10594. The lack of backwards compatibility seems problematic to me. I can try to post something there later today.

vbraun commented 13 years ago

comment:139

Anything that creates a new mercurial repository will by default require dotencode. In particular, $SAGE_LOCAL/bin/sage-clone which is unrelated to this ticket.

It would be easy to add --config format.dotencode=0 to all hg init, hg clone commands. But current distributions already picked up the new mercurial, so I don't see it as particularly pressing issue. Moreover, we have a suitable mercurial in Sage precisely because it is sometimes finky with old versions. I would be in favor of just using dotencode. If it causes a big problem then we can always revert the repository format, you just have to clone with dotencode=0.

jhpalmieri commented 13 years ago

Description changed:

--- 
+++ 
@@ -1,6 +1,10 @@
 Put the text files in `$SAGE_ROOT`, and also the text files in spkg, under revision control.  (See the discussion at the end of #9351.)

-Here are the instructions:
+Here are the instructions; the differences between the two versions are the scripts patch and the hg_script:
+
+---
+
+VERSION 1, trying to make compatible with older versions of Mercurial:

 - apply [attachment: trac_9433-sage-repo.2.patch](https://github.com/sagemath/sage-prod/files/10649949/trac_9433-sage-repo.2.patch.gz)
 - apply [attachment: trac_9433-scripts.v8.patch](https://github.com/sagemath/sage-prod/files/10649970/trac_9433-scripts.v8.patch.gz) to the scripts repository
@@ -11,4 +15,19 @@

 Then **from $SAGE_ROOT**, run the attached script [attachment: 9433_hg_script.2.sh](https://github.com/sagemath/sage-prod/files/10649968/9433_hg_script.2.sh.gz) to create the Mercurial repository.

+---
+
+VERSION 2, forcing the use of Mercurial 1.7 or later:
+
+- apply [attachment: trac_9433-sage-repo.2.patch](https://github.com/sagemath/sage-prod/files/10649949/trac_9433-sage-repo.2.patch.gz)
+- apply [attachment: trac_9433-scripts.v7.patch](https://github.com/sagemath/sage-prod/files/10649964/trac_9433-scripts.v7.patch.gz) to the scripts repository
+- move the attached file [attachment: hgignore](https://github.com/sagemath/sage-prod/files/10649969/hgignore.gz) to `$SAGE_ROOT/.hgignore` (note that this is a *new* file)
+- move the attached file [attachment: root-spkg-install.v3](https://github.com/sagemath/sage-prod/files/10649967/root-spkg-install.v3.gz) to `$SAGE_ROOT/spkg/root-spkg-install` (note that this is a *new* file)
+- apply [attachment: 9433_install.diff](https://github.com/sagemath/sage-prod/files/10649960/9433_install.diff.gz) to `$SAGE_ROOT/spkg/install`
+- apply [attachment: 9433_deps.diff](https://github.com/sagemath/sage-prod/files/10649966/9433_deps.diff.gz) to `$SAGE_ROOT/spkg/standard/deps`
+
+Then **from $SAGE_ROOT**, run the attached script [attachment: 9433_hg_script.sh](https://github.com/sagemath/sage-prod/files/10649963/9433_hg_script.sh.gz) to create the Mercurial repository.
+
+---
+
 **Testing**: see [http://sage.math.washington.edu/home/release/sage-4.7.alpha0/](http://sage.math.washington.edu/home/release/sage-4.7.alpha0/)

jdemeyer commented 13 years ago

comment:141

Testing distributions:

http://boxen.math.washington.edu/home/release/sage-4.7.alpha0/: sage-4.6.2.rc0 + this ticket
http://boxen.math.washington.edu/home/release/sage-4.7.alpha1/: sage-4.7.alpha0 + some random stuff (#10688 and attachment: 9433_testing.patch)

jdemeyer commented 13 years ago

comment:142

Upgrading sage-4.7.alpha0 -> sage-4.7.alpha1 fails because of uncommitted changes: during the install of sage_root-4.7.alpha1.spkg, I get an editor with

HG: Enter commit message.  Lines beginning with 'HG:' are removed.
HG: Leave message empty to abort commit.
HG: --
HG: user: Jeroen Demeyer <jdemeyer@cage.ugent.be>
HG: branch 'default'
HG: changed spkg/install
HG: changed spkg/standard/deps

hg diff gives:

diff -r 1c44cedc9957 spkg/install
--- a/spkg/install      Thu Feb 17 15:54:54 2011 +0000
+++ b/spkg/install      Sun Feb 20 16:06:09 2011 +0100
@@ -1,5 +1,7 @@
 #!/usr/bin/env bash

+# TESTING PATCH
+
 ###############################################################################
 # Check if pipestatus already exists, otherwise
 # create it to allow upgrade from Sage <4.5.  This is a temporary fix.
@@ -422,9 +424,6 @@
 TERMCAP=`$newest termcap`
 export TERMCAP

-WEAVE=`$newest weave`
-export WEAVE
-
 ZLIB=`$newest zlib`
 export ZLIB

and something similar for spkg/standard/deps.

It seems that spkg/install and spkg/standard/deps are changed by the upgrader before the Sage root repository is installed and that this causes trouble.

jdemeyer commented 13 years ago

Changed reviewer from Leif Leonhardy, Volker Braun to Leif Leonhardy, Volker Braun, Jeroen Demeyer

jdemeyer commented 13 years ago

comment:143

John (and/or Volker), do you have time to work on this in the coming week / 2 weeks? If yes, I would like to merge this in sage-4.7.alpha0. This ticket might require some more work to get it right, so it would be nice to know whether you have time. Otherwise we can postpone this to a later Sage release.

vbraun commented 13 years ago

comment:144

I'm around and am in favor of merging it asap.

jhpalmieri commented 13 years ago

comment:145

I have a fix to this problem which I am testing right now. It looks good so far, and I'll post it later if it continues to look good. (The fix is as follows: the script sage-update downloads new versions of several files, like install and deps; I'm now adding a command to commit the Sage root repo after doing this download.)

jhpalmieri commented 13 years ago

comment:146

Okay, I have two options; which is better? In sage-update, the files install, deps, and newest_version might get changed. We could:

commit the changes in sage-update, and then the changes will get committed again by root-spkg-install when the root repo spkg is installed.
or in root-spkg-install, we can overwrite any changes: do hg pull ... to pull from the new repo and then hg update --clean to discard any other changes.

The first of these adds a redundant commit message to the log file if the relevant files are changed. The second discards all changes, including any other ones that the user may have made. I don't know Mercurial well enough, but perhaps there is a better third choice?

vbraun commented 13 years ago

comment:147

I'm in favor of the following third option:

The first thing that sage-update should do is to commit any outstanding changes and, in particular, abort if there is a patch queue applied. So you are safe even if you forgot to commit your changes.
Then sage-update overwrites some critical files for its operation.
Finally, the sage_root spkg is installed and its spkg-install overwrites the remaining non-critical files in the root repository. I'd rather be guaranteed to have a clean slate after update than dying in the middle of the update because the attempted merge fails.

So if you had any uncommitted changes before updating, you'll end up with two commit messages. The more individual commits the better. The only drawback is that if you had made modifications to the root repository that you want to keep and you are not using patch queues, then you now have to extract your changes as a patch from the repository and apply that patch. I think thats acceptable since, I think, most developers use patch queues and I don't expect too frequent edits to the root repository.

If that doesn't convince you, I'd prefer option number 1 as my second choice :-)

jhpalmieri commented 13 years ago

comment:148

Here is a new version of the scripts patch. The only change is in sage-update. I updated the instructions, also: I'm not worrying about making the repo compatible with older versions of Mercurial right now.

jhpalmieri commented 13 years ago

Description changed:

--- 
+++ 
@@ -1,26 +1,11 @@
 Put the text files in `$SAGE_ROOT`, and also the text files in spkg, under revision control.  (See the discussion at the end of #9351.)

-Here are the instructions; the differences between the two versions are the scripts patch and the hg_script:
+Here are the instructions:

 ---

-VERSION 1, trying to make compatible with older versions of Mercurial:
-
 - apply [attachment: trac_9433-sage-repo.2.patch](https://github.com/sagemath/sage-prod/files/10649949/trac_9433-sage-repo.2.patch.gz)
-- apply [attachment: trac_9433-scripts.v8.patch](https://github.com/sagemath/sage-prod/files/10649970/trac_9433-scripts.v8.patch.gz) to the scripts repository
-- move the attached file [attachment: hgignore](https://github.com/sagemath/sage-prod/files/10649969/hgignore.gz) to `$SAGE_ROOT/.hgignore` (note that this is a *new* file)
-- move the attached file [attachment: root-spkg-install.v3](https://github.com/sagemath/sage-prod/files/10649967/root-spkg-install.v3.gz) to `$SAGE_ROOT/spkg/root-spkg-install` (note that this is a *new* file)
-- apply [attachment: 9433_install.diff](https://github.com/sagemath/sage-prod/files/10649960/9433_install.diff.gz) to `$SAGE_ROOT/spkg/install`
-- apply [attachment: 9433_deps.diff](https://github.com/sagemath/sage-prod/files/10649966/9433_deps.diff.gz) to `$SAGE_ROOT/spkg/standard/deps`
-
-Then **from $SAGE_ROOT**, run the attached script [attachment: 9433_hg_script.2.sh](https://github.com/sagemath/sage-prod/files/10649968/9433_hg_script.2.sh.gz) to create the Mercurial repository.
-
----
-
-VERSION 2, forcing the use of Mercurial 1.7 or later:
-
-- apply [attachment: trac_9433-sage-repo.2.patch](https://github.com/sagemath/sage-prod/files/10649949/trac_9433-sage-repo.2.patch.gz)
-- apply [attachment: trac_9433-scripts.v7.patch](https://github.com/sagemath/sage-prod/files/10649964/trac_9433-scripts.v7.patch.gz) to the scripts repository
+- apply [attachment: trac_9433-scripts.v9.patch](https://github.com/sagemath/sage-prod/files/10649971/trac_9433-scripts.v9.patch.gz) to the scripts repository
 - move the attached file [attachment: hgignore](https://github.com/sagemath/sage-prod/files/10649969/hgignore.gz) to `$SAGE_ROOT/.hgignore` (note that this is a *new* file)
 - move the attached file [attachment: root-spkg-install.v3](https://github.com/sagemath/sage-prod/files/10649967/root-spkg-install.v3.gz) to `$SAGE_ROOT/spkg/root-spkg-install` (note that this is a *new* file)
 - apply [attachment: 9433_install.diff](https://github.com/sagemath/sage-prod/files/10649960/9433_install.diff.gz) to `$SAGE_ROOT/spkg/install`

jhpalmieri commented 13 years ago

comment:149

Oh, also, I created some versions for testing upgrades:

http://sage.math.washington.edu/home/palmieri/misc/9433/sage-4.6.2.X0/
http://sage.math.washington.edu/home/palmieri/misc/9433/sage-4.6.2.X1/ (spkg/install changed)
http://sage.math.washington.edu/home/palmieri/misc/9433/sage-4.6.2.X2/ (Makefile changed)
http://sage.math.washington.edu/home/palmieri/misc/9433/sage-4.6.2.X3/ (Makefile and spkg/standard/deps changed)

The first of these is just 4.6.2.rc0 with the patches from this ticket applied. In each of the other ones, one or two files have been modified, as indicated. It's also worth testing by updating to the "X0" version, for example, then modifying a file tracked by the root repo (like README.txt) without committing the change, and then trying to update to "X1". The update should abort.

jhpalmieri commented 13 years ago

comment:150

I made a few minor cosmetic changes in the patch, and right now, the upgrade paths I mentioned use the old version, not the new one. I'll have them updated in a few minutes.

jhpalmieri commented 13 years ago

comment:151

Okay, everything's updated now.

jdemeyer commented 13 years ago

comment:152

I made new testing releases:

http://boxen.math.washington.edu/home/release/sage-4.7.alpha0/: sage-4.6.2.rc1 + this ticket
http://boxen.math.washington.edu/home/release/sage-4.7.alpha1/: sage-4.7.alpha0 + #10688 + attachment: 9433_testing.patch

Upgrading sage-4.7.alpha0 -> sage-4.7.alpha1 succeeded this time.

vbraun commented 13 years ago

comment:153

I've tried the update and it works beautifully.

I think there is still one problem: If the user has no .hgrc (like many non-developers trying to upgrade), then mercurial will fail to commit with

[vbraun@volker-desktop sage-4.7.alpha0]$ mv ~/.hgrc ~/backup.hgrc
[vbraun@volker-desktop sage-4.7.alpha0]$ hg commit -m "test"
abort: no username supplied (see "hg help config")

So whenever we commit to the root repo, we should specify a username explicitly. For example,

hg commit -u "committed by sage -upgrade" -m "test"

jhpalmieri commented 13 years ago

patch for scripts repo

jhpalmieri commented 13 years ago

Attachment: trac_9433-scripts.v9.patch.gz

The file $SAGE_ROOT/spkg/root-spkg-install

jhpalmieri commented 13 years ago

comment:154

Attachment: root-spkg-install.v4.gz

Replying to @vbraun:

I think there is still one problem: If the user has no .hgrc (like many non-developers trying to upgrade), then mercurial will fail

I wouldn't have spotted this. Good catch. I have new patches which add "-u ..." to various commit commands: in sage-update and in root-spkg-install. I haven't bothered with sage-sdist or sage-make_devel_packages, since these are done by the release manager who had better have a .hgrc file. (Besides, there are already other "hg commit" commands in those scripts.)

I've also updated my versions (4.6.2.X0 etc.) for testing upgrades with these changes.

jhpalmieri commented 13 years ago

Description changed:

--- 
+++ 
@@ -7,7 +7,7 @@
 - apply [attachment: trac_9433-sage-repo.2.patch](https://github.com/sagemath/sage-prod/files/10649949/trac_9433-sage-repo.2.patch.gz)
 - apply [attachment: trac_9433-scripts.v9.patch](https://github.com/sagemath/sage-prod/files/10649971/trac_9433-scripts.v9.patch.gz) to the scripts repository
 - move the attached file [attachment: hgignore](https://github.com/sagemath/sage-prod/files/10649969/hgignore.gz) to `$SAGE_ROOT/.hgignore` (note that this is a *new* file)
-- move the attached file [attachment: root-spkg-install.v3](https://github.com/sagemath/sage-prod/files/10649967/root-spkg-install.v3.gz) to `$SAGE_ROOT/spkg/root-spkg-install` (note that this is a *new* file)
+- move the attached file [attachment: root-spkg-install.v4](https://github.com/sagemath/sage-prod/files/10649972/root-spkg-install.v4.gz) to `$SAGE_ROOT/spkg/root-spkg-install` (note that this is a *new* file)
 - apply [attachment: 9433_install.diff](https://github.com/sagemath/sage-prod/files/10649960/9433_install.diff.gz) to `$SAGE_ROOT/spkg/install`
 - apply [attachment: 9433_deps.diff](https://github.com/sagemath/sage-prod/files/10649966/9433_deps.diff.gz) to `$SAGE_ROOT/spkg/standard/deps`

vbraun commented 13 years ago

comment:155

I'm happy with it now. I tested upgrades without ~/.hgrc and everything worked for me.

jdemeyer commented 13 years ago

comment:156

Agreed! This will get merged in sage-4.7.alpha0. I still expect breakage here and there, but that's what alpha versions are for.

jdemeyer commented 13 years ago

Merged: sage-4.7.alpha0

sagemath / sage

Put more files under revision control. #9433