jhpalmieri
commented
14 years ago comment:1
I'm marking this as "needs review", but consider this patch experimental.
Closed jhpalmieri closed 13 years ago
jhpalmieri
commented
14 years ago I'm marking this as "needs review", but consider this patch experimental.
jhpalmieri
commented
14 years ago A little explanation: this patch creates a directory "other-scripts" in SAGE_ROOT/local/bin. This new directory contains a brief README.txt and also subdirectories "root" and "spkg". "root" contains the text files from SAGE_ROOT. The only one with any changes is README.txt which explains how these files are under revision control. Similarly, "spkg" contains various text files from SAGE_ROOT/spkg, and the only one with any changes is README.txt.
jhpalmieri
commented
14 years ago This probably needs to be rebased. When people are ready to look at it, let me know and I'll see what I can do.
jhpalmieri
commented
14 years ago rebased against 4.5.alpha4 + #9456
jhpalmieri
commented
14 years ago Attachment: trac_9433-scripts.2.patch.gz
New approach, after a discussion on sage-devel: create a new repo at the top level tracking the appropriate files. I'm attaching a new version of the patch for the scripts repo. Someone -- the release manager, I guess -- also needs to create the top level repo, because I don't know how to do this in such a way that it can be posted on a ticket. Here are the instructions:
(I don't know if we really need these last two files, but this is probably not the ticket for making such decisions.) Finally, do
When you run "sage -sdist ..." it should add a tag for the new version of Sage.
This does not create a new spkg for the files in SAGE_ROOT, since those files have to be in place when you unpack the sage tar file. But it creates the repository so that people can post patches to the trac server, etc.
williamstein
commented
14 years ago Looking with my eyes, this looks good. I don't have time to test right now. The test would be to take a clean Sage, do the above, then do "sage -sdist ..." and make sure that in the sdist the above is all still there.
jhpalmieri
commented
14 years ago main repo: add "hg_root" command to Sage
jhpalmieri
commented
14 years ago Attachment: trac_9433-sage-repo.patch.gz
Replying to @williamstein:
The test would be to take a clean Sage, do the above, then do "sage -sdist ..." and make sure that in the sdist the above is all still there.
This works for me, but other people should definitely look at it carefully.
jhpalmieri
commented
14 years ago This probably needs work: how will it work with "sage -upgrade"?
jhpalmieri
commented
14 years ago Description changed:
---
+++
@@ -1,2 +1,19 @@
Put the text files in $SAGE_ROOT, and also the text files in spkg, under revision control. (See the discussion at the end of #9351.)
+Here are the instructions:
+
+- apply the patches trac_9433-sage-repo.patch and trac_9433-scripts.patch
+
+and create the Mercurial repository:
+
+- move the attached file "hgignore" to SAGE_ROOT/.hgignore
+- cd $SAGE_ROOT
+- hg init .
+- hg add .hgignore COPYING.txt README.txt makefile sage sage-python
+- cd spkg
+- hg add README.txt gen_html install pipestatus
+- cd standard
+- hg add README.txt deps libdist_filelist newest_version
+- hg add notes.txt numeric-24.2.txt
+- hg commit
+Here's a new version of the patch for the scripts repo. I think this should deal with upgrading: the script "sage-upgrade" now runs "sage --hg branch" from SAGE_ROOT, and if this fails, it assumes that there is no Mercurial repository and creates it.
jhpalmieri
commented
14 years ago This may need to be changed if #9609 is merged: the directions here, and also the modified "sage-upgrade" script, refer to files which would be deleted by #9609.
jhpalmieri
commented
14 years ago Description changed:
---
+++
@@ -14,6 +14,5 @@
- hg add README.txt gen_html install pipestatus
- cd standard
- hg add README.txt deps libdist_filelist newest_version
-- hg add notes.txt numeric-24.2.txt
- hg commit
New version, rebased against #9609.
jhpalmieri
commented
14 years ago new version, using "hg clone". apply to scripts repo.
jhpalmieri
commented
14 years ago Attachment: trac_9433-scripts.3.patch.gz
new version, using "hg clone". apply to scripts repo.
jhpalmieri
commented
14 years ago Attachment: trac_9433-scripts.patch.gz
Note that if you use "hg clone ..." to copy the repo, you have to do it earlier in the process: it apparently needs to be done with an empty directory, so in sage-sdist it is now done right after creating $TMP.
e14f4152-4982-4ace-8c95-73a0599b109b
commented
14 years ago If I
./sage -sdist 4.5.99hg log in the "root" repo now gives
changeset: 1:0fea58e94942
tag: tip
user: Mitesh Patel <qed777@gmail.com>
date: Fri Aug 06 21:40:23 2010 -0700
summary: Added tag 4.5.99 for changeset 4c1f4320f743
changeset: 0:4c1f4320f743
tag: 4.5.99
user: Mitesh Patel <qed777@gmail.com>
date: Fri Aug 06 21:33:45 2010 -0700
summary: Initial Sage "root" repositoryThe new 4.5.99 builds successfully from source and passes the long doctests. But hg log in the root repo for 4.5.99 lists just revision 0, and the root repo is missing from a binary distribution made with ./sage -bdist 4.5.99.
e14f4152-4982-4ace-8c95-73a0599b109b
commented
14 years ago Author: John Palmieri
e14f4152-4982-4ace-8c95-73a0599b109b
commented
14 years ago I also noticed
SAGE_ROOT/ipython and SAGE_ROOT/sage-README-osx.txt are missing from the new source and binary distributions.SAGE_ROOT/sage-python after unpacking sage-4.5.99.tar.jhpalmieri
commented
14 years ago Replying to @qed777:
The new 4.5.99 builds successfully from source and passes the long doctests. But
hg login the root repo for 4.5.99 lists just revision 0, and the root repo is missing from a binary distribution made with./sage -bdist 4.5.99.
Right, I didn't do this right. In the new patch, the root repo is not modified at all by sage-make_devel_packages; instead, sage-sdist clones it, tags it, and commits the new tag, while sage-bdist just clones it.
I also noticed
- SAGE_ROOT/ipython and SAGE_ROOT/sage-README-osx.txt are missing from the new source and binary distributions.
The missing ipython directory was an oversight. I think I've fixed it. The missing sage-README-osx.txt was intentional: this should only be included for binary distributions on OS X, and its presence there is taken care of by sage-bdist:
if [ "$UNAME" = "Darwin" ]; then
...
cp sage/local/bin/sage-README-osx.txt README.txt
...Perhaps we can close #6938 if this gets merged?
- An extra SAGE_ROOT/sage-python after unpacking sage-4.5.99.tar.
This was intentional. Before this, the file sage-python was stored in the scripts repo and then unpacked to the top level. I'm not sure why this was done, but I wanted to keep the end result as close as possible to what it was before.
jhpalmieri
commented
14 years ago Description changed:
---
+++
@@ -10,7 +10,9 @@
- cd $SAGE_ROOT
- hg init .
- hg add .hgignore COPYING.txt README.txt makefile sage sage-python
-- cd spkg
+- cd ipython
+- hg add *.py ipythonrc*
+- cd ../spkg
- hg add README.txt gen_html install pipestatus
- cd standard
- hg add README.txt deps libdist_filelist newest_version
kcrisman
commented
14 years ago I also noticed
- SAGE_ROOT/ipython and SAGE_ROOT/sage-README-osx.txt are missing from the new source and binary distributions.
The missing ipython directory was an oversight. I think I've fixed it. The missing sage-README-osx.txt was intentional: this should only be included for binary distributions on OS X, and its presence there is taken care of by sage-bdist:
if [ "$UNAME" = "Darwin" ]; then ... cp sage/local/bin/sage-README-osx.txt README.txt ...Perhaps we can close #6938 if this gets merged?
As one of the people involved on that ticket, that is fine. The problem is that #6938 does not currently have positive review! So I think that would be necessary first, or something else indicating that the solution proposed there is correct. Maybe 'merge' that ticket at the same time as this one, for whatever it's worth.
Sounds like you agree :) In fact, notice that once that is removed, that file will only appear ABOVE the SAGE_ROOT directory, in the place a normal README would occur in a dmg or bundle, so it does work properly (I've tested this numerous times
jhpalmieri
commented
14 years ago Description changed:
---
+++
@@ -7,13 +7,14 @@
and create the Mercurial repository:
- move the attached file "hgignore" to SAGE_ROOT/.hgignore
+- move the attached file "root-spkg-install" to SAGE_ROOT/spkg/root-spkg-install
- cd $SAGE_ROOT
- hg init .
- hg add .hgignore COPYING.txt README.txt makefile sage sage-python
- cd ipython
- hg add *.py ipythonrc*
- cd ../spkg
-- hg add README.txt gen_html install pipestatus
+- hg add README.txt gen_html install pipestatus root-spkg-install
- cd standard
- hg add README.txt deps libdist_filelist newest_version
- hg commitI realized there was another problem with the previous patch: while it would work when upgrading from a Sage build with no root repo, it wouldn't do anything when upgrading a Sage build with an existing root repo. The new patch constructs a sage_root spkg file, but this file only gets used during upgrading. So it is installed in the script "sage-upgrade", but it does not appear in spkg/standard/deps.
jhpalmieri
commented
14 years ago This still doesn't deal with upgrading very well. I need to think about how to do this and work on it for a while in a place where I have a better internet connection. I may have something in a few days. Meanwhile, if anyone has suggestions for how to deal with upgrades, I would like to hear them. The issues:
currently, I don't even know what happens with "sage -upgrade ..." if SAGE_ROOT/makefile is changed, for example. Does anything happen?
does it matter whether we install the new sage_root spkg before or after the sage_scripts spkg? The patch here affects the sage-upgrade script, but the new version won't get run during an upgrade anyway.
is it good enough to just install the new sage_root spkg? I think it might be, but I might be confused. Anyway, I think I need time to play with it and test it out.
jhpalmieri
commented
14 years ago Description changed:
---
+++
@@ -3,19 +3,8 @@
Here are the instructions:
- apply the patches trac_9433-sage-repo.patch and trac_9433-scripts.patch
-
-and create the Mercurial repository:
-
- move the attached file "hgignore" to SAGE_ROOT/.hgignore
- move the attached file "root-spkg-install" to SAGE_ROOT/spkg/root-spkg-install
-- cd $SAGE_ROOT
-- hg init .
-- hg add .hgignore COPYING.txt README.txt makefile sage sage-python
-- cd ipython
-- hg add *.py ipythonrc*
-- cd ../spkg
-- hg add README.txt gen_html install pipestatus root-spkg-install
-- cd standard
-- hg add README.txt deps libdist_filelist newest_version
-- hg commit
+Then **from $SAGE_ROOT**, run the attached script "hg_script" to create the Mercurial repository.
+This seems to work for me, with one slight glitch: if I run "sage -upgrade" on a copy of sage which does not yet include the root repo, it lists the spkg's to be upgraded and asks me "do you really want to continue", then it does some stuff, and then it lists just the root_repo spkg and asks me again if I want to continue. The issue is that, before it has installed some of the upgraded packages, it doesn't know what to do with the root_repo spkg, so it doesn't get installed the first time through. I don't see any way around this, but it's a one-time problem.
83660e46-0051-498b-a8c1-f7a7bd232b5a
commented
14 years ago Ooops, I once had been aware of this ticket, but forgot to cc me...
Before doing this, can we rename makefile to Makefile?
jhpalmieri
commented
14 years ago Before doing this, can we rename makefile to Makefile?
I have no problems with that. Are there ever any (good) reasons for using makefile instead of Makefile?
83660e46-0051-498b-a8c1-f7a7bd232b5a
commented
14 years ago Should sage-README-osx.txt be ignored (i.e. not be under revision control)?
Also a candidate for renaming (README.MacOS_X.txt or alike).
From the attached hg_script:
...
$hg add .hgignore .hgtags COPYING.txt README.txt makefile sage sage-python
cd ipython
$hg add *.py ipythonrc*
...sage-python and ipython/ should IMHO be in .hgignore, since these are not in $SAGE_ROOT until Sage is built.
jhpalmieri
commented
14 years ago Replying to @nexttime:
Should
sage-README-osx.txtbe ignored (i.e. not be under revision control)?
It shouldn't be present at all, and with the patches here, it isn't. See #6938 and also my comment above. (It should only appear in the parent directory of SAGE_ROOT when you make a dmg file on a Mac.)
sage-pythonandipython/should IMHO be in.hgignore, since these are not in$SAGE_ROOTuntil Sage is built.
This is another change on this ticket: these files are now tracked in this repo, not in sage_scripts anymore. Actually, I don't know if ipython was tracked anywhere, but now it is. I don't know who chose to include sage-python in the top-level directory, but since it's there, it should be tracked properly. (It's a simple enough script I don't mind having two copies of it.) I personally don't see the point of it and think it should be removed, but that should probably happen on another ticket. Or maybe it should be a link to local/bin/sage-python?
kcrisman
commented
14 years ago Replying to @jhpalmieri:
Replying to @nexttime:
Should
sage-README-osx.txtbe ignored (i.e. not be under revision control)?It shouldn't be present at all, and with the patches here, it isn't. See #6938 and also my comment above. (It should only appear in the parent directory of SAGE_ROOT when you make a dmg file on a Mac.)
Correct.
83660e46-0051-498b-a8c1-f7a7bd232b5a
commented
14 years ago Replying to @jhpalmieri:
Replying to @nexttime:
Should
sage-README-osx.txtbe ignored (i.e. not be under revision control)?It shouldn't be present at all, and with the patches here, it isn't. See #6938 and also my comment above. (It should only appear in the parent directory of SAGE_ROOT when you make a dmg file on a Mac.)
Missed that.
sage-pythonandipython/should IMHO be in.hgignore, since these are not in$SAGE_ROOTuntil Sage is built.This is another change on this ticket: these files are now tracked in this repo, not in sage_scripts anymore.
That, too... :/
Actually, I don't know if ipython was tracked anywhere, but now it is. I don't know who chose to include sage-python in the top-level directory, but since it's there, it should be tracked properly. (It's a simple enough script I don't mind having two copies of it.) I personally don't see the point of it and think it should be removed, but that should probably happen on another ticket. Or maybe it should be a link to local/bin/sage-python?
Perhaps...
Sorry for the noise.
83660e46-0051-498b-a8c1-f7a7bd232b5a
commented
14 years ago I'd also like to have a VERSION.txt in SAGE_ROOT/, best containing just the "plain" version number (perhaps only with also the release date added); cf. #9434 (the number is a coincidence!).
What about release notes (in the top-level directory)?
kcrisman
commented
14 years ago Replying to @nexttime:
I'd also like to have a
VERSION.txtinSAGE_ROOT/, best containing just the "plain" version number (perhaps only with also the release date added); cf. #9434 (the number is a coincidence!).
That's a good idea. Be sure not to hold this one up too long ;)
What about release notes (in the top-level directory)?
That definitely used to be in this directory, filename HISTORY.txt. Maybe it was getting long? It certainly got out of date quickly. It might be an onus on the release manager to create the notes before the release is made, though - usually that takes a little time, and then it somehow gets added to the official version at sagemath.org (and a few other places?). The problem is that this isn't fully automated yet.
83660e46-0051-498b-a8c1-f7a7bd232b5a
commented
14 years ago Replying to @kcrisman:
Replying to @nexttime:
I'd also like to have a
VERSION.txtinSAGE_ROOT/, best containing just the "plain" version number (perhaps only with also the release date added); cf. #9434 (the number is a coincidence!).That's a good idea. Be sure not to hold this one up too long ;)
I don't think adding this needs great effort... We already have devel/sage/sage/version.py, which looks like this:
"""nodoctests"""
version='4.6.alpha1'; date='2010-09-15'What about release notes (in the top-level directory)?
That definitely used to be in this directory, filename
HISTORY.txt. Maybe it was getting long? It certainly got out of date quickly. It might be an onus on the release manager to create the notes before the release is made, though
Doesn't have to be under revision control (i.e. could be in .hgignore); it IMHO shouldn't be too lengthy, perhaps just contain the most recent changes (e.g. tickets merged since last final, with a reference to a complete version elsewhere).
- usually that takes a little time, and then it somehow gets added to the official version at sagemath.org (and a few other places?). The problem is that this isn't fully automated yet.
I think the release notes as in the announcements are meanwhile fully automatically generated by a script. I just wonder if these are really "human"-(means user-)readable, if we intend that.
kcrisman
commented
14 years ago Those ideas seem reasonable, but probably I'm not the one to make the call, since I'm not doing the work.
- usually that takes a little time, and then it somehow gets added to the official version at sagemath.org (and a few other places?). The problem is that this isn't fully automated yet.
I think the release notes as in the announcements are meanwhile fully automatically generated by a script. I just wonder if these are really "human"-(means user-)readable, if we intend that.
The hard part is, but at least in theory the "known issues" and "new features" sections and things like that are supposed to be human-generated. mvngu used to make great categorized ones, but likely hasn't had the time lately. I think they are more human-readable than some I've seen in other programs, though :)
jhpalmieri
commented
14 years ago I think that for release notes, we could just have a document which says something like "Please see http://sagemath.org/mirror/src/changelogs/sage-$VERSION.txt for a list of recent changes." Or we could use the link "http://sagemath.org/mirror/src/changelogs/" and then we wouldn't have to update it. This also provides easy access to older changelogs (which the $VERSION.txt doesn't, and notice it's just a text file -- no links to the parent directory or older changelogs). Opinions? We could also add the link "http://wiki.sagemath.org/ReleaseTours/", although this hasn't been updated in a while.
I like the generic option better. But notice that this file won't be auto-generated by "sage -sdist" or any other script, and it should probably be under revision control. This is not the right ticket for adding new files to the top directory, or for modifying existing files, so I think this should go elsewhere.
I also agree that a VERSION.txt file is a good idea. Since we can automatically generate this, and since it shouldn't be under revision control, I think we can do it on this ticket. It should just require modifying sage-sdist and sage-bdist. I'll try towork on this some time soon.
83660e46-0051-498b-a8c1-f7a7bd232b5a
commented
14 years ago As a first step, just add VERSION.txt to .hgignore (here). ;-)
My intention (perhaps as a developer) regarding "release notes" (whatever the file is called) also was not to have to search trac or some web page(s) (or query Mercurial) just to see which tickets have recently been merged...
For users, of course a more abstract description would be better (bugs fixed, packages newly included or upgraded, new features etc.)
jhpalmieri
commented
14 years ago I've opened up #9922 for adding release notes. For VERSION.txt, I think we (meaning me) should do the whole thing on this ticket: add it to .hgignore, create it in sage-sdist, and make sure it gets copied by sage-bdist, or else the whole thing should go on another ticket. If I don't get to it on this one, then I'll open a ticket describing the steps (including adding it to .hgignore) -- I don't really see the point in doing one piece of it here.
e14f4152-4982-4ace-8c95-73a0599b109b
commented
14 years ago Should we update VERSION.txt after successful upgrades? Or maybe have separate fields for the original version (and whether it was a binary) and the current version? Or even keep a brief upgrade history?
jhpalmieri
commented
14 years ago I've attached a new version of hg_script which rename 'makefile' to 'Makefile' if it hasn't already been done. (I thought I also had to modify root-spkg-install, but I don't think I do: once the repo has been made via hg_script, it will have 'Makefile' in it, not 'makefile'.)
I also think mpatel has a good point about VERSION.txt. I think there are possible design decisions to be made about how to create that file, what should go in it, how to modify it, etc., so I think it should go another ticket. It could piggy-back on #9922, I think. I'm going to change the description of that ticket to include this.
jhpalmieri
commented
14 years ago fixes problems found by mpatel. apply to scripts repo
jhpalmieri
commented
14 years ago Attachment: trac_9433-scripts.v2.patch.gz
jhpalmieri
commented
14 years ago Description changed:
---
+++
@@ -2,7 +2,7 @@
Here are the instructions:
-- apply the patches trac_9433-sage-repo.patch and trac_9433-scripts.patch
+- apply the patches trac_9433-sage-repo.patch and trac_9433-scripts.v2.patch
- move the attached file "hgignore" to SAGE_ROOT/.hgignore
- move the attached file "root-spkg-install" to SAGE_ROOT/spkg/root-spkg-install
A lot of environment variables should be quoted ($SAGE_ROOT, $TMP, $OPT etc. in most commands).
Needs rebasing in case #9896 gets merged... (or the other way around)
83660e46-0051-498b-a8c1-f7a7bd232b5a
commented
14 years ago root-spkg-install also needs work:
...
if [ -e makefile ]; then
cp makefile "$TARGET"/Makefile
else
cp Makefile "$TARGET"
fi
if [ ! -d "$TARGET/skpg" ]; then
mkdir "$TARGET/spkg"
fi
if [ ! -d "$TARGET/skpg/standard" ]; then
mkdir "$TARGET/spkg/standard"
...We'd better use -f than -e, s/skpg/spkg/g.
83660e46-0051-498b-a8c1-f7a7bd232b5a
commented
14 years ago P.S.: Instead of
if [ ! -d foo ]; then
mkdir foo
fiyou can simply do
mkdir -p fooDescription changed:
---
+++
@@ -2,7 +2,7 @@
Here are the instructions:
-- apply the patches trac_9433-sage-repo.patch and trac_9433-scripts.v2.patch
+- apply the patches trac_9433-sage-repo.patch and trac_9433-scripts.v3.patch
- move the attached file "hgignore" to SAGE_ROOT/.hgignore
- move the attached file "root-spkg-install" to SAGE_ROOT/spkg/root-spkg-install
Put the text files in
$SAGE_ROOT, and also the text files in spkg, under revision control. (See the discussion at the end of #9351.)Here are the instructions:
$SAGE_ROOT/.hgignore(note that this is a new file)$SAGE_ROOT/spkg/root-spkg-install(note that this is a new file)$SAGE_ROOT/spkg/install$SAGE_ROOT/spkg/standard/depsThen from $SAGE_ROOT, run the attached script attachment: 9433_hg_script.sh to create the Mercurial repository.
Testing: see http://sage.math.washington.edu/home/release/sage-4.7.alpha0/
CC: @williamstein @dandrake @kcrisman @nexttime
Component: distribution
Author: John Palmieri
Reviewer: Leif Leonhardy, Volker Braun, Jeroen Demeyer
Merged: sage-4.7.alpha0
Issue created by migration from https://trac.sagemath.org/ticket/9433