Closed jhpalmieri closed 15 years ago
Oh, I forgot: you need to apply the patch at #6585 for the reference manual to build. This version of Sphinx also prints a lot of new warnings, too.
I installed the updated SPKG with
./sage -f http://sage.math.washington.edu/home/palmieri/SPKG/sphinx-0.6.2.p0.spkg
and applied the patch at #6585. When doing
./sage -docbuild reference html
I got the following errors:
mvngu@sage sage-4.1.1.alpha0-sage.math.washignton.edu-x86_64-Linux]$ ./sage -docbuild reference html
Traceback (most recent call last):
File "/scratch/mvngu/sage-4.1.1.alpha0-sage.math.washignton.edu-x86_64-Linux/devel/sage/doc/common/builder.py", line 667, in <module>
getattr(get_builder(name), type)()
File "/scratch/mvngu/sage-4.1.1.alpha0-sage.math.washignton.edu-x86_64-Linux/devel/sage/doc/common/builder.py", line 348, in _wrapper
for module_name in self.get_modified_modules():
File "/scratch/mvngu/sage-4.1.1.alpha0-sage.math.washignton.edu-x86_64-Linux/devel/sage/doc/common/builder.py", line 412, in get_modified_modules
env = self.get_sphinx_environment()
File "/scratch/mvngu/sage-4.1.1.alpha0-sage.math.washignton.edu-x86_64-Linux/devel/sage/doc/common/builder.py", line 403, in get_sphinx_environment
return BuildEnvironment.frompickle(config, env_pickle)
File "/scratch/mvngu/sage-4.1.1.alpha0-sage.math.washignton.edu-x86_64-Linux/local/lib/python2.6/site-packages/Sphinx-0.6.2-py2.6.egg/sphinx/environment.py", line 210, in frompickle
env = pickle.load(picklefile)
AttributeError: 'module' object has no attribute 'RedirStream'
I think I got this error, too. Once I deleted doc/output, things worked for me. (I think there is something stored in the old version of the documentation which confuses the new version of Sphinx.)
The upgraded Sphinx SPKG can't handle binary encodings in the file:
sage/combinat/partition.py
In particular, the following sections from that source file:
line:2233
sage: p = loads('x\x9c\x85\x8e\xbd\xaa\xc2@\x10\x85\x89\xff.>\xc4\x94\xda\x04\x15|\x04\xb1\xb1\x90\x0b[\x87I\x18\x935\xc9\xae\xb33\xda\t\xd7\xc2\xf76"biw\x0e\x9c\x9f\xef\xbfW\x08\x96\x94\x16\xa1\xcd\x9dGM\xcf\x18\xd5\xa9\x0b\xde\x1c>Jv\x91PIt\xbf\xcd|m8Y\xdc\xb9w\xe3\xfe\xdc&\xf5\xbb\x1d\x9d/%u^\xa9\xa4hZ\xac)\xfb\x18\x1e\xd8d\xfd\xf8\xe3\xa1\x1df\x1e[\xe2\x91\xdd|\x97!\x1ca\xb5\x84\n\xaf\xdd\x02\xbc\xbe\x05\x1a\x12\x01\xad\xd0C\x88@|\xc1\x064\xc0\x9a\xc7v\x16\xf2\x13\x15\x9a\x15\r\x8a\xf0\xe47\xf9;ixj\x13_u \xd8\x81\x98K\x9e>\x01\x13iVH')
line:2294
sage: p = loads('x\x9c\x85\x8e\xbd\x0e\x820\x14\x85\x03\x8a?\x8d\x0f\xd1Q\x97\x06y\x07\xe3\xaa&\x9d\xc9\xa5\xb9\x96\n\xb4^Z\xdcLt\xf0\xbd\xc5 qt;\'9?\xdf#V\x1e4\n\xe5\x9a\xc2X\x08\xe2\nm0\xc18\xcb\x0e\xa3\xf2\xfb\x16!\xa0\x0f\xbbcn+F\xd1\xe6I\xf1\x9d&k\x19UC\xbb5V{al@\x8d-k\xa0\xc2|44\x95Q\xf6:Q"\x93\xdcB\x834\x93\xe9o\x99\xbb3\xdf\xa6\xbc\x84[\xbf\xc0\xf5\xf7\x87\x7f 8R\x075\x0f\x8eg4\x97+W\\P\x85\\\xd5\xe0=-\xfeC\x0fIFK\x19\xd9\xb2g\x80\x9e\x81u\x85x\x03w\x0eT\xb1')
line:2839
sage: dmp = 'x\x9ck`J.NLO\xd5K\xce\xcfM\xca\xccK,\xd1+H,*\xc9,\xc9\xcc\xcf\xe3\n\x80\xb1\x8a\xe3\x93\x81DIQbf^I1W!\xa3fc!Sm!\xb3F(7\x92x!Km!k(GnbE<\xc8\x88B6\x88\xb9E\x99y\xe9\xc5z@\x05\xa9\xe9\xa9E\\\xb9\x89\xd9\xa9\xf10N!{(\xa3QkP!Gq(c^\x06\x90c\x0c\xe4p\x96&\xe9\x01\x00\xc2\xe53\xfd'
When these lines are removed, the HTML version of the reference manual builds OK. So a possible fix for line 2233 is to replace everything in the TEST block starting from that line by:
sage: p = PartitionsGreatestLE(10,2)
sage: p == loads(dumps(p))
True
We can do something similar for line 2294; replace everything in the TEST block starting from line 2294 by:
sage: p = PartitionsGreatestEQ(10,2)
sage: p == loads(dumps(p))
True
As for the TEST block starting from line 2839, I have no idea how to fix it so that the reference manual would successfully builds. Someone on the sage-combinat team to the rescue?
Replying to @jhpalmieri:
Oh, I forgot: you need to apply the patch at #6585 for the reference manual to build. This version of Sphinx also prints a lot of new warnings, too.
I think that this patch fixes the issue with binary encodings; at least it did for me.
With the patch at #6585 and the SPKG, building the reference manual on sage.math takes much longer than previously.
Replying to @sagetrac-mvngu:
With the patch at #6585 and the SPKG, building the reference manual on sage.math takes much longer than previously.
I agree: on my mac, it took 28 minutes to build the html version with the old one, and 43 minutes with the new one. With the --jsmath option on, the old one took 6 minutes and the new one took 15 minutes. I was getting similar comparisons on sage.math. (This is with me deleting devel/sage/doc/output between each build, to level the playing field.)
It's strange: for the tutorial and developer's guide, building is about the same or slightly faster with the new version. I also made a modified version of the developer's guide by duplicating some sections (so that the resulting PDF file was 559 pages); the new version was slightly faster there, too. Maybe it's all of the math in the reference manual? Maybe it's the large number of files? Maybe it's the autogeneration of files?
Cursory profiling with #6187's "testreference" target indicates that the rewritten autodoc
extension is significantly slower than before. I think autodoc
now also uses far more memory. Attempting to build the full reference manual on my machine, Sphinx "stalled" at 97% of modules read. I think the kernel was too busy swapping everything but Sphinx to disk.
Perhaps there's a more efficient, parallelizable representation. Can we subclass and/or Cythonize sphinx.ext.autodoc.Documenter?
Minor progress: It seems that a custom autodoc-skip-member method now overrides Sphinx's default behavior, which is to skip private methods (__init__
, etc.). To keep these out of the reference manual, we might use, e.g.,
def skip_NestedClass(app, what, name, obj, skip, options):
"""docstring"""
skip_nested = str(obj).find("sage.misc.misc") != -1 and name.find("MainClass.NestedClass") != -1
return skip or skip_nested
in conf.py
. Unfortunately, this does not significantly reduce the build time or memory usage.
Apparently, the culprit is sphinx.pycode
's Python source analyzer, which sphinx.ext.autodoc
uses to extract attribute docstrings. From Sphinx-0.6.2-py2.6.egg/sphinx/pycode/__init__.py
:
The docstrings can either be in special '#:' comments before the assignment
or in a docstring after it.
According to the analyzer, there are just two places to find such docstrings in the Sage library:
combinat/designs/block_design.py
schemes/elliptic_curves/ell_field.py
Still, resource use is high, because there is a lot of caching going on. We can reduce it with weak references:--- __init__.py.orig 2009-08-01 09:58:27.303927239 -0700
+++ __init__.py 2009-08-01 11:35:17.477865562 -0700
@@ -120,10 +120,11 @@ class PycodeError(Exception):
res += ' (exception was: %r)' % self.args[1]
return res
-
+import weakref
class ModuleAnalyzer(object):
# cache for analyzer objects -- caches both by module and file name
- cache = {}
+# cache = {}
+ cache = weakref.WeakValueDictionary()
@classmethod
def for_string(cls, string, modname, srcname='<string>'):
This does reduce build time and memory usage, but I don't know if the output is identical.
For sage -docbuild reference html --jsmath
, at least, the results are close:
$ du -s output_v0*
95550 output_v051
97392 output_v062
97388 output_v062_mod
The differences are in environmental.pickle
or stem from using a function as a default argument in
combinat/words/word_content.py
rings/polynomial/groebner_fan.py
Clearly, it's time for a break.
Nice observation. Making the change with weak_references does indeed speed things up:
0.5.1 28 minutes
0.6.2 43 minutes
0.6.2 33 minutes (version using weakref)
With no other changes, it includes all of the private methods, as already pointed out (and I assume it did before using weakref -- this may account for some of the increased time, since it's producing more output), but it also includes (for some classes, anyway), an entry for __weakref)_
. See the documentation for sage/algebras/steenrod_algebra.py, for instance: it lists as an attribute of SteenrodAlgebraFactory
__weakref__
list of weak references to the object (if defined)
Actually, noticing __weakref__
for that module is what led to the comment about autodoc-skip-member
. I think this was before I "patched" Sphinx, but I may be mistaken. I'm also not sure about why __weakref__
appears spontaneously. Perhaps a super class?
See #6664 for the autodoc-skip-member
issue.
Also, I've asked about the merits of the weakref
workaround on sphinx-dev.
Example inheritance diagram for "The Steenrod algebra". This is just for fun.
Attachment: inheritance_example.patch.gz
The new inheritance diagram extension has potential. I've attached a minimal example. Please do not merge it.
The options are limited, however. For example, I could not find a directive that auto-generates a diagram for each module. Also, it seems there's no way to color-code the nodes/classes, e.g., to indicate their modular provenance. Maybe it's possible to do one or more of these ourselves, in builder.py
, conf.py
, or by patching Sphinx.
I should add that this requires a local installation of GraphViz.
Replying to @qed777:
Also, I've asked about the merits of the
weakref
workaround on sphinx-dev.
The thread. Preliminary tests indicate that the proposed alternative does not reduce memory use significantly.
Replying to @qed777:
Replying to @qed777:
Also, I've asked about the merits of the
weakref
workaround on sphinx-dev.The thread. Preliminary tests indicate that the proposed alternative does not reduce memory use significantly.
Here, actually.
There's been no further activity about the workaround on sphinx-dev. Since weakrefs appear to work for us, should we add a patch to the spkg?
Replying to @qed777:
There's been no further activity about the workaround on sphinx-dev. Since weakrefs appear to work for us, should we add a patch to the spkg?
Hmm... could you make an spkg with that workaround?
Replying to @sagetrac-mvngu:
Replying to @qed777:
There's been no further activity about the workaround on sphinx-dev. Since weakrefs appear to work for us, should we add a patch to the spkg?
Hmm... could you make an spkg with that workaround?
I can certainly do it, but not until Monday.
Does the problem described here recur with the new spkg?
The new spkg is at
http://sage.math.washington.edu/home/palmieri/SPKG/sphinx-0.6.2.p1.spkg
(The old one is still there; it's "...p0.spkg", as listed in the description above.)
By the way, note that since this includes the private methods in the reference manual, there are a lot of new warnings when building that document. (People haven't written all of the docstrings for their __init__
methods, etc., to conform with proper reST format, since they didn't need to before.) These should be dealt with in a separate ticket: otherwise, we might feel obliged to keep rebasing this ticket until it was merged.
Does #6664 suppress the private methods?
Does #6664 suppress the private methods?
Yes, but do we want to suppress the private methods? I actually prefer that they be included in the reference manual.
Uh oh, it seems as though with this spkg, changes to pyx files are not detected. That is, build the documentation (say, with the jsmath option). Then edit a .py file and a .pyx file (I chose sage/misc/latex.py and sage/matrix/matrix0.pyx). Run 'sage -docbuild reference print_modified_modules': the .py file will be listed, but not the .pyx file. I can't figure out why this is happening :(
I also don't know why. builder.py
uses Sphinx's get_outdated_files()
method to get an iterable of modified modules. However, this method's code is the same in v0.5.1 and v0.6.2. I did find that the newer Sphinx does not include .so files for .pyx files in environment.pickle
's dependencies
dictionary. This may "explain" the observed behavior.
I played around with the Sphinx code for a while, and I could not figure out what's going on. Well, I believe that "matrix0.so" was listed as a dependency of "matrix0.pyx" in version 0.5.1, so changing the .pyx file resulted in the modification time for the .so file to be wrong, triggering a rebuild of the docs, but in 0.6.2, the .so file no longer appears to be a dependency of the .pyx file. I've posted a message to sphinx-dev about how dependencies are determined; we'll see if there are any helpful responses.
Okay, I now have a patch which seems to recognize changes to pyx files. The dependency information is stored in the pickle, so to test this, first delete SAGEDOC/output/doctrees/
. The new spkg is available from
http://sage.math.washington.edu/home/palmieri/SPKG/sphinx-0.6.2.p2.spkg.
Please test thoroughly. (One thing to watch out for: I thought it wasn't working because I was changing sage/matrix/action.pyx and seeing no notification when the reference manual was built, but then I realized that action.pyx isn't even included in the reference manual. Doh.)
In case anyone downloaded the version posted yesterday, there is a slightly modified version today: the author of Sphinx posted a patch for the Cython problem to the Sphinx mercurial site, and so the version today uses that patch instead. (Mine was similar to his, but why not use the official one?)
I've replaced the old one with the new one, so the link is the same: http://sage.math.washington.edu/home/palmieri/SPKG/sphinx-0.6.2.p2.spkg.
Okay, here's another one. Sphinx was just updated to version 0.6.3, and this is supposed to fix our problems. In my testing, it doesn't have the Cython problem that the original version of 0.6.2 had; it incorporates the patch in the most recent 0.6.2 spkg. The 0.6.3 spkg is a bit slower than 0.5.1 and also than our version of 0.6.2 with the unauthorized weakref patch. It's not nearly as slow and doesn't seem to use nearly as much memory as the unpatched 0.6.2. Here's the spkg: http://sage.math.washington.edu/home/palmieri/SPKG/sphinx-0.6.3.p0.spkg.
Some timings in minutes for "sage -docbuild reference html --jsmath", with the patch at #6664 applied:
Mac sage.math ubuntu
Sphinx 0.5.1 6 5 10
Sphinx 0.6.2.p2 6 6 10
Sphinx 0.6.3 8 9 11
"Mac" is my Intel iMac running OS X 10.5; "sage.math" is sage.math, and "ubuntu" is an old ubuntu box I have access to.
As I said above, the memory usage seems to stay within reason for all of these, but I'm not using anything sophisticated to test this: with the terrible version, my computer would become very sluggish, and with the versions tested here, that doesn't happen.
For Sage 4.1.2.alpha0's HTML reference manual, at least, the new spkg seems to work just as Georg indicated. I'll try to do more testing soon.
By the way, it might be useful to update spkg-install along the lines of #6598 and this example. For example,
#!/usr/bin/env bash
if [ -z "$SAGE_LOCAL" ]; then
echo "SAGE_LOCAL undefined ... exiting";
echo "Maybe run 'sage -sh'?"
exit 1
fi
success() {
if [ $? -ne 0 ]; then
echo "Error building Sphinx: '$1'"
exit 1
fi
}
set -e
echo "Removing old version..."
rm -rf "$SAGE_LOCAL"/lib/python/site-packages/Sphinx-*
success 'deleting previous version'
[...]
Replying to @qed777:
By the way, it might be useful to update spkg-install along the lines of #6598 and this example. For example,
Okay, try this one: http://sage.math.washington.edu/home/palmieri/SPKG/sphinx-0.6.3.p1.spkg. (Putting "set -e" in other places made the script exit without printing the custom error message. With this one, if, for example, I make the directory "$SAGE_LOCAL"/lib/python/site-packages/Sphinx-*
undeletable, I get the appropriate message.)
Thanks very much for pointing that out. I may need to update the spkgs at #5447...
Forgive me if I'm mistaken, but doesn't Sphinx>=0.6 depend on Jinja2?
Incompatible changes:
http://sphinx.pocoo.org/changes.html#release-0-6-mar-24-2009
Yes. I think site-packages/Jinja2-2.1.1-py2.6-linux-x86_64.egg
, say, is installed automatically (via network) by python setup.py install
in spkg-install
.
Should we make it a separate spkg, to supplement or replace jinja-1.2.p0.spkg
?
Author: John Palmieri
I checked that Sphinx now incorporates changes to included .py and .pyx files.
For the HTML builder, both jsmath
and pngmath
modes appear to work as expected.
The PDF reference manual's table of contents is missing. This may stem from #6885 and my invoking LaTeX's batch mode with b
.
Reviewer: Mitesh Patel
Replying to @qed777:
Yes. I think
site-packages/Jinja2-2.1.1-py2.6-linux-x86_64.egg
, say, is installed automatically (via network) bypython setup.py install
inspkg-install
.Should we make it a separate spkg, to supplement or replace
jinja-1.2.p0.spkg
?
I think it would probably best to make a separate spkg so that installation will not require network access. I've made an spkg that I can post if you want.
Also, the spkg for this states that it depends on Jinja>=1.1
, which is no longer the case.
Okay, I've updated the file SPKG.txt (in http://sage.math.washington.edu/home/palmieri/SPKG/sphinx-0.6.3.p1.spkg) to say that it depends on Jinja >= 2. If you want to post a link to an spkg for jinja, that would be great.
Here you go: http://drop.io/jinja2_spkg
A few issues with the Jinja spkg: the source is under version control but it shouldn't be. When I install it, I get jinja2-2.1.1, not jinja2-2.2, so the name is wrong. Also, the part of the spkg-install file which is supposed to remove the old version wasn't removing the right thing (it said "Jinja-2", but the directory is actually "Jinja2-2.1.1").
Anyway, here's a modified spkg, much smaller since src is not in the mercurial repository. This is an updated version of the version 1 jinja spkg, with the new src dropped in, replacing the old src: http://sage.math.washington.edu/home/palmieri/SPKG/jinja2-2.1.1.p0.spkg.
Should this have its own ticket? Will it screw up anything from #6568? I don't know anything about Jinja, so I don't know what to do to test it; all I see is that the docs are built correctly with the new versions of Sphinx and Jinja.
Thanks for the fixes.
Jinja and Jinja2 are totally different, and so it shouldn't affect any current templating routines. In any case, once Sphinx 0.6.3 is included, we can remove the dependency on Jinja by replacing all instances of import jinja
with import jinja2
, etc. The API's are pretty much unchanged, and from what I know, none of the changed interfaces are currently used.
I'm not at all sure what you people are doing here, so perhaps I should have kept away, but (in trying to help with #6820) I tried installing the new sphinx spkg using
./sage -f http://sage.math.washington.edu/home/palmieri/SPKG/sphinx-0.6.2.p0.spkg
and now
john@ubuntu%sage -docbuild reference html
Traceback (most recent call last):
File "/home/john/sage-4.1.1/devel/sage/doc/common/builder.py", line 667, in <module>
getattr(get_builder(name), type)()
File "/home/john/sage-4.1.1/devel/sage/doc/common/builder.py", line 348, in _wrapper
for module_name in self.get_modified_modules():
File "/home/john/sage-4.1.1/devel/sage/doc/common/builder.py", line 412, in get_modified_modules
env = self.get_sphinx_environment()
File "/home/john/sage-4.1.1/devel/sage/doc/common/builder.py", line 403, in get_sphinx_environment
return BuildEnvironment.frompickle(config, env_pickle)
File "/home/john/sage-4.1.1/local/lib/python2.6/site-packages/Sphinx-0.6.2-py2.6.egg/sphinx/environment.py", line 210, in frompickle
env = pickle.load(picklefile)
AttributeError: 'module' object has no attribute 'RedirStream'
This is in the main branch, so a vanilla 4.1.1 build. Should that be 4.1.2.alpha0?
Hey.
The first spkg had problems. There are newer spkgs. http://sage.math.washington.edu/home/palmieri/SPKG/sphinx-0.6.3.p1.spkg is the latest.
As the summary says. The spkg can be found at
http://sage.math.washington.edu/home/palmieri/SPKG/sphinx-0.6.3.p1.spkg
After installing it and before building any docs, you will probably need to delete the directory SAGE_ROOT/devel/sage/doc/output -- this holds cached information which may not be compatible between different versions of Sphinx.
Sphinx 0.6.3 also uses Jinja2. Sage includes Jinja, not Jinja2. As long as you're connected to a network, the installation of Sphinx 0.6.3 downloads Jinja2 and installs it -- it can coexist with Jinja, so it shouldn't mess anything up. If you're not connected to a network, and for putting in the Sage tar ball, there is an spkg for Jinja2:
http://sage.math.washington.edu/home/palmieri/SPKG/jinja2-2.1.1.p0.spkg
For producing a new Sage tarball, since we're adding a new spkg, I think we need to modify the files
spkg/install
andspkg/standard/deps
; I've attached new versions of these.Finally, the patch below ("inheritance_example.patch") is not required. It provides an illustration of a new capability of Sphinx 0.6.3 (and I think it's quite nice), but it's not something that needs to be installed or reviewed for this ticket.
CC: @mwhansen @sagetrac-sage-combinat
Component: packages: standard
Author: John Palmieri
Reviewer: John Cremona, Tim Dumol, Mitesh Patel
Merged: sage-4.2.alpha0
Issue created by migration from https://trac.sagemath.org/ticket/6586