search

sagemath / sage

Main repository of SageMath
https://www.sagemath.org
Other
1.38k stars 469 forks source link

Build the reference manual incrementally #6495

Closed e14f4152-4982-4ace-8c95-73a0599b109b closed 11 years ago

e14f4152-4982-4ace-8c95-73a0599b109b commented 15 years ago

Building the Sage reference manual can take a significant amount of time. Decreasing this time could speed up Sage development.

The patch is large, but most of it consists of moving files from one location to another, as described below. A summary of the changes:

Changes in doc/en/reference — this is where the size of the patch comes from, although the changes are pretty simple:

Changes to doc/common/builder.py:

Other changes:

The html docs for Sage 5.4.rc2, html without MathJax, html with MathJax, and pdf, built after applying the patches here.

Apply:

Before building the docs, you should delete the documentation output directory: rm -rf SAGE_ROOT/devel/sage/doc/output. To test this, you should run sage --docbuild all html and sage --docbuild all pdf. (Note: just running sage --docbuild reference html will probably produce many warnings. If you run it a second time, the warnings should go away.)

Depends on #13064 Depends on #8327 Depends on #13891

Dependencies: 5.7.beta2 + #13064, #8327, #13891

CC: @jhpalmieri @nexttime @nilesjohnson @hivert @mguaypaq @mwhansen

Component: documentation

Keywords: days38

Author: Mitesh Patel, John Palmieri, Florent Hivert

Reviewer: Volker Braun, Florent Hivert

Merged: sage-5.8.beta0

Issue created by migration from https://trac.sagemath.org/ticket/6495

jhpalmieri commented 11 years ago
comment:123

Attachment: trac_6495-part4-interrupts.patch.gz

Rebased to #13143 and Sage 5.4.rc2 (because of changes to builder.py in #13579).

jhpalmieri commented 11 years ago

Description changed:

--- 
+++ 
@@ -27,9 +27,11 @@

 ---

-The html docs for Sage 5.4.beta0, built
-[without MathJax](http://sage.math.washington.edu/home/palmieri/misc/6495/html/en/) and
-[with MathJax](http://sage.math.washington.edu/home/palmieri/misc/6495-jsmath/html/en/), built after applying the patches here.
+The html docs for Sage 5.4.rc2,
+[html without MathJax](http://sage.math.washington.edu/home/palmieri/misc/6495/html/),
+[html with MathJax](http://sage.math.washington.edu/home/palmieri/misc/6495-jsmath/html/), and 
+[pdf](http://sage.math.washington.edu/home/palmieri/misc/6495/pdf/),
+built after applying the patches here.

 ---
kcrisman commented 11 years ago

Work Issues: rebase

kcrisman commented 11 years ago
comment:125

John, I've tried to use this on an existing Sage installation (well, previous versions of this) a few different times, and it always ends up causing problems, even when I delete the whole output directory. Is there something to do other than apply the all-in-one patch, delete output, do sage -b, and start building docs? For instance, is there an env var (e.g. SAGE_NUM_THREADS) to set or something? I feel like I must have done something obvious wrong...

On a related note, this no longer applies :-(

jhpalmieri commented 11 years ago
comment:126

What sort of problems do you see? As far as environment variables, setting MAKE is the right thing to do: export MAKE='make -j2' for example.

I just rebased the patches on top of 5.5.rc0.

jhpalmieri commented 11 years ago

Changed work issues from rebase to none

kcrisman commented 11 years ago
comment:129

What sort of problems do you see? As far as environment variables, setting MAKE is the right thing to do: export MAKE='make -j2' for example.

Lots of 

WARNING: intersphinx inventory '/Users/.../sage-5.5.rc0/devel/sage/doc/output/html/en/reference/monoids/objects.inv' not fetchable due to <type 'exceptions.IOError'>: [Errno 2] No such file or directory: '/Users/.../sage-5.5.rc0/devel/sage/doc/output/html/en/reference/monoids/objects.inv'

stuff.

build succeeded, 1074 warnings.

That's while building reference html, and now I see upon careful reading of the ticket description that perhaps that's implicitly expected. When I build the whole documentation as a target, I don't get this problem, though I do get one warning:

WARNING: search index couldn't be loaded, but not all documents will be built: the index will be incomplete.
jhpalmieri commented 11 years ago

Description changed:

--- 
+++ 
@@ -46,4 +46,4 @@
 - [attachment: trac_6495-part3-the-remaining-vs-5.4.patch](https://github.com/sagemath/sage-prod/files/10645451/trac_6495-part3-the-remaining-vs-5.4.patch.gz) — this patch takes care of merging the indexes and the bibliography.
 - [attachment: trac_6495-part4-interrupts.patch](https://github.com/sagemath/sage-prod/files/10645450/trac_6495-part4-interrupts.patch.gz) — handle ctrl-c.

-Before building the docs, you should delete the documentation output directory: `rm -rf SAGE_ROOT/devel/sage/doc/output`.
+Before building the docs, you should delete the documentation output directory: `rm -rf SAGE_ROOT/devel/sage/doc/output`. To test this, you should run `sage --docbuild all html` and `sage --docbuild all pdf`. (Note: just running `sage --docbuild reference html` will probably produce many warnings. If you run it a second time, the warnings should go away.)
jhpalmieri commented 11 years ago
comment:130

Yes, the warnings are somewhat expected. Those are like what happens when you run LaTeX on a file for the first time: all of the unknown reference warnings which get resolved the second time through. I think typically people will do the following:

That is, I hope that most users/developers won't see all of these warnings. I've updated the ticket description so it mentions the warnings. (We could turn off all of the warnings all of the time, but I don't think that's a good idea.)

jdemeyer commented 11 years ago
comment:131

I consider it a requirement that there are no visible warnings/errors when building the documentation from scratch. Otherwise, it becomes difficult to differentiate warnings caused by this ticket from true warnings.

jdemeyer commented 11 years ago
comment:132

Does this patch enable links like :func:`plot()` from other manuals (e.g. the developer manual) to the reference manual?

kcrisman commented 11 years ago
comment:133

I consider it a requirement that there are no visible warnings/errors when building the documentation from scratch. Otherwise, it becomes difficult to differentiate warnings caused by this ticket from true warnings.

John, is this even possible under this setup? It would really be a shame to hold this up but Jeroen makes a good point as well.

jhpalmieri commented 11 years ago
comment:134

Replying to @jdemeyer:

Does this patch enable links like :func:`plot()` from other manuals (e.g. the developer manual) to the reference manual?

I think so, but it ought to be tested. You might need to specify the cross-reference more explicitly, the way you might if you want to reference the Python docs, for instance.

Replying to @kcrisman:

I consider it a requirement that there are no visible warnings/errors when building the documentation from scratch. Otherwise, it becomes difficult to differentiate warnings caused by this ticket from true warnings.

John, is this even possible under this setup? It would really be a shame to hold this up but Jeroen makes a good point as well.

I thought that Florent had ideas about how to get rid of the one remaining warning. (As I said before, the intersphinx warnings are turned off on the first pass through the reference manual when you do sage --docbuild all html, and I think this is appropriate.)

I'll try to look into both of these. Anyone else is welcome, also. Florent, do you have any time these days?

dimpase commented 11 years ago
comment:135

please rebase for 5.6.beta3:

$sage-5.6.beta3/devel/sage$ ../../sage -hg qpushapplying trac_6495-all-in-one.patch
patching file doc/en/reference/combinat/index.rst
Hunk #1 FAILED at 3
1 out of 3 hunks FAILED -- saving rejects to file doc/en/reference/combinat/index.rst.rej
patch failed, unable to continue (try -v)
patch failed, rejects left in working dir
errors during apply, please fix and refresh trac_6495-all-in-one.patch
hivert commented 11 years ago
comment:136

Replying to @jhpalmieri:

I thought that Florent had ideas about how to get rid of the one remaining warning. (As I said before, the intersphinx warnings are turned off on the first pass through the reference manual when you do sage --docbuild all html, and I think this is appropriate.)

I'll try to look into both of these. Anyone else is welcome, also. Florent, do you have any time these days?

I'm deeply sorry for letting time pass and not finishing this one... I'll be in Sage-Days in Edimbourg in two weeks so I hopefully will finish this one.

Florent

6bfcfeed-779b-4b16-82e6-63808dde0af0 commented 11 years ago
comment:137

I did the following:

Does that help?

jhpalmieri commented 11 years ago
comment:138

Snark: is that specific to this ticket, or does sage --docbuild ru/tutorial pdf fail before applying these patches?

jhpalmieri commented 11 years ago
comment:139

Rebased for 5.6.beta3.

6bfcfeed-779b-4b16-82e6-63808dde0af0 commented 11 years ago
comment:140

I installed the needed cyrillic support and it went through without problem.

jhpalmieri commented 11 years ago
comment:141

Another approach to parallel building (not that we want to throw away all of this work): a proposed patch to Sphinx. I don't know if using this proposal would affect Sphinx's memory usage, though, which is another issue that is helped by breaking up the reference manual.

jdemeyer commented 11 years ago
comment:142

Replying to @jhpalmieri:

Another approach to parallel building (not that we want to throw away all of this work): a proposed patch to Sphinx.

It only parallelizes the "writing output" phase, not the "reading" phase, so it cannot be that useful.

I don't know if using this proposal would affect Sphinx's memory usage, though

Most likely not.

jdemeyer commented 11 years ago
comment:143

Just for fun, I packaged it: http://boxen.math.washington.edu/home/jdemeyer/spkg/sphinx-parallel.spkg and made it support the Sage standard MAKE="-j20" mechanism.

hivert commented 11 years ago
comment:144

Replying to @jdemeyer:

It only parallelizes the "writing output" phase, not the "reading" phase, so it cannot be that useful.

This is more or less what I did when experimenting on #6255. At that time, since I was building the whole documentation it increased the memory usage. By the way, we probably should close #6255 at some point as a duplicate of this one.

Florent

jdemeyer commented 11 years ago
comment:145

With the parallel Sphinx package I get lots of errors, some might be due to the "upgrade" to the devleopment version of Sphinx 1.2, but some also to the parallel code.

jhpalmieri commented 11 years ago
comment:146

I think our Sphinx package needs some work. In particular, the file SAGE_ROOT/devel/sage/common/sage-autodoc.py should be part of the Sphinx package, built by patching Sphinx's autodoc.py. This would make it easier to incorporate modifications to autodoc.py. I don't know if this has to do with the problems with using Sphinx 1.2, but it might.

It's probably also not a good idea to use Sphinx's parallel building at the same time as ours, or else we'll be using more threads than we intended: maybe building 8 pieces of documentation at the same time, and then writing each of those using 8 processes. That sounds like a bad idea to me.

Having said all of that, do the errors you get suggest problems with our parallel code?

jdemeyer commented 11 years ago
comment:147

Replying to @jhpalmieri:

Having said all of that, do the errors you get suggest problems with our parallel code?

I didn't use any code from this ticket, I only used the patch you linked to for parallel Sphinx writing.

jhpalmieri commented 11 years ago

Attachment: trac_6495-part3-the-remaining-vs-5.4.patch.gz

apply third

jhpalmieri commented 11 years ago
comment:148

Rebased to 5.6.rc0 (among other things, dealing with the addition of doc/en/reference/sat.rst).

hivert commented 11 years ago
comment:149

This is an answer to Mike Hansen:

He asked if the page url are kept or not. The answer is that they moved: For example:

doc/output/html/en/reference/sage/symbolic/expression.html

is now

doc/output/html/en/reference/calculus/sage/symbolic/expression.html

So the location is not kept.

Florent

kcrisman commented 11 years ago
comment:150

Replying to @hivert:

This is an answer to Mike Hansen:

He asked if the page url are kept or not. The answer is that they moved:

Hmm, I was dimly aware of this but didn't think it through. That will break a loooot of links (e.g., on ask.sagemath.org). Is there any way to auto-generate forwarding or 'mirrors' from the old pages?

hivert commented 11 years ago

Description changed:

--- 
+++ 
@@ -38,6 +38,7 @@
 Apply:

 - [attachment: trac_6495-all-in-one.patch](https://github.com/sagemath/sage-prod/files/10645461/trac_6495-all-in-one.patch.gz).
+- [attachment: trac-6495_silence_warning-fh.patch](https://github.com/sagemath/sage-prod/files/10645453/trac-6495_silence_warning-fh.patch.gz).

 For ease of review, you can instead look at the following four patches:

@@ -47,3 +48,4 @@
 - [attachment: trac_6495-part4-interrupts.patch](https://github.com/sagemath/sage-prod/files/10645450/trac_6495-part4-interrupts.patch.gz) — handle ctrl-c.

 Before building the docs, you should delete the documentation output directory: `rm -rf SAGE_ROOT/devel/sage/doc/output`. To test this, you should run `sage --docbuild all html` and `sage --docbuild all pdf`. (Note: just running `sage --docbuild reference html` will probably produce many warnings. If you run it a second time, the warnings should go away.)
+
dimpase commented 11 years ago
comment:152

Replying to @kcrisman:

Replying to @hivert:

This is an answer to Mike Hansen:

He asked if the page url are kept or not. The answer is that they moved:

Hmm, I was dimly aware of this but didn't think it through. That will break a loooot of links (e.g., on ask.sagemath.org). Is there any way to auto-generate forwarding or 'mirrors' from the old pages?

there are several options:

  1. use .htaccess thing to fix this on any particular website;

  2. write a script to look through all href targets in each file with changed location, and create for it a file with forwards;

  3. maybe sphinx can help in this business, I don't know.

In any event we do not want to keep the old links around forever, I think, so there should be some kind of deprecation being turned on.

hivert commented 11 years ago
comment:153

He asked if the page url are kept or not. The answer is that they moved:

Hmm, I was dimly aware of this but didn't think it through. That will break a loooot of links (e.g., on ask.sagemath.org). Is there any way to auto-generate forwarding or 'mirrors' from the old pages?

there are several options:

  1. use .htaccess thing to fix this on any particular website;

  2. write a script to look through all href targets in each file with changed location, and create for it a file with forwards;

  3. maybe sphinx can help in this business, I don't know.

I don't thinks it does. On the contrary, it makes it a nightmare (at least to me). Sphinx made the assumption that creating a new document needs creating a new directory (storing html, markup, indexes...). Since we splitting the reference manual in different documents we had to move files around.

Mike Hansen was considering implementing something which is close to option 2, because it it seems that option 1 is not easily doable (whereas I'm starting to now more than a little about Sphinx, I shamefully admitting that I don't know how to write/configure a website).

Florent

vbraun commented 11 years ago
comment:154

We should just have versioned documentation on sagemath.org, just like Singular:

Having said that, Apache mod_rewrite could be used to dull some of the pain and automatically guess the new location.

mwhansen commented 11 years ago
comment:156

Replying to @vbraun:

Having said that, Apache mod_rewrite could be used to dull some of the pain and automatically guess the new location.

Can you really use mod_rewrite to do this? You have to turn a request like reference/sage/symbolic/expression.html to something like reference/calculus/sage/symbolic/expression.html . So, you'd have to walk to filesystem in order to find out the analog of "calculus".

vbraun commented 11 years ago
comment:157

Yes and no... its not beautiful but you can just add a rule for each sage/ subdirectory that does the equivalent of 

sed s-sage/symbolic-calculus/sage/symbolic-
mwhansen commented 11 years ago

Attachment: trac_6495-redirect_html.patch.gz

mwhansen commented 11 years ago
comment:158

Replying to @vbraun:

Yes and no... its not beautiful but you can just add a rule for each sage/ subdirectory that does the equivalent of 

sed s-sage/symbolic-calculus/sage/symbolic-

I don't think that you can really do this since it isn't guaranteed that all of the files that are in say sage/misc/ will all go to the same document.

I wrote a patch which adds small HTML files to do the redirect. I've looked through Florent's code as well, and things seem fine to me -- trac_6495-all-in-one.patch and trac-6495_silence_warning-fh.patch .

If someone reviews trac_6495-redirect_html.patch , we can mark this as positive_review.

mwhansen commented 11 years ago

Description changed:

--- 
+++ 
@@ -39,6 +39,7 @@

 - [attachment: trac_6495-all-in-one.patch](https://github.com/sagemath/sage-prod/files/10645461/trac_6495-all-in-one.patch.gz).
 - [attachment: trac-6495_silence_warning-fh.patch](https://github.com/sagemath/sage-prod/files/10645453/trac-6495_silence_warning-fh.patch.gz).
+- [attachment: trac_6495-redirect_html.patch](https://github.com/sagemath/sage-prod/files/10645452/trac_6495-redirect_html.patch.gz).

 For ease of review, you can instead look at the following four patches:

@@ -48,4 +49,3 @@
 - [attachment: trac_6495-part4-interrupts.patch](https://github.com/sagemath/sage-prod/files/10645450/trac_6495-part4-interrupts.patch.gz) — handle ctrl-c.

 Before building the docs, you should delete the documentation output directory: `rm -rf SAGE_ROOT/devel/sage/doc/output`. To test this, you should run `sage --docbuild all html` and `sage --docbuild all pdf`. (Note: just running `sage --docbuild reference html` will probably produce many warnings. If you run it a second time, the warnings should go away.)
-
kcrisman commented 11 years ago
comment:160

I wrote a patch which adds small HTML files to do the redirect. I've looked through Florent's code as well, and things seem fine to me -- trac_6495-all-in-one.patch and trac-6495_silence_warning-fh.patch .

If someone reviews trac_6495-redirect_html.patch , we can mark this as positive_review.

Thank you so much for adding this, Mike - I hope to try it out tonight and see if it works fine, but I doubt I'll be competent to give positive review.

Here is a question about all the patches. There are a lot of functions (esp. underscore ones) in doc/... that have no doctests. Problem? (This includes Mike's new function, but is not limited to it.) Hope this isn't evil to ask.

jhpalmieri commented 11 years ago
comment:161

I have yet to test attachment: trac_6495-redirect_html.patch, but the docstring for html ends rather abruptly.

jhpalmieri commented 11 years ago
comment:162

If I run sage --docbuild all html, at the end of the second pass for the reference manual, I see

*******************SKIPPING Load indexer call*******************
WARNING: search index couldn't be loaded, but not all documents will be built: the index will be incomplete.
Build finished.  The built documents can be found in /scratch/palmieri/6495/sage-5.7.beta0/devel/sage/doc/output/html/en/reference

So the warning isn't gone.

jhpalmieri commented 11 years ago
comment:163

Addendum: if I delete documentation output and run sage --docbuild reference html twice, I don't see the warning message. So perhaps something can be tweaked to make it go away on the second pass when running sage --docbuild all html.

jhpalmieri commented 11 years ago
comment:164

Rebased to 5.7.beta0.

jhpalmieri commented 11 years ago
comment:165

attachment: trac_6495-redirect_html.patch doesn't quite work, because os.path.split(path) always returns a list of length 2. With this change, it works for me:

diff --git a/doc/common/builder.py b/doc/common/builder.py
--- a/doc/common/builder.py
+++ b/doc/common/builder.py
@@ -391,7 +391,7 @@ class WebsiteBuilder(DocBuilder):
                     redirect_filename = os.path.join(reference_dir, shorter_path, filename)

                     # the number of levels up we need to use in the relative url
-                    levels_up = len(os.path.split(shorter_path))
+                    levels_up = len(shorter_path.split(os.sep))

                     # the relative url that we will redirect to
                     redirect_url = "/".join(['..']*levels_up + [document_name, shorter_path, filename])

Fix that, and fix the docstring for the html method, and I think this part is good.

The warning is still an issue, and I don't know why it still shows up during the second pass for the reference manual when doing sage --docbuild all html but not when running sage --docbuild reference html twice. Florent, any ideas? Does anyone else see this?

kcrisman, I don't know about doctests for all of these methods.

hivert commented 11 years ago
comment:166

Strange, It's working for me. Can you post the log of the docbuild which shows the warning, please

Florent

hivert commented 11 years ago

Attachment: trac-6495_silence_warning-fh.patch.gz

hivert commented 11 years ago
comment:167

Sorry, I ended up uploading the wrong version of my patch. I uploaded a new version.

Florent

hivert commented 11 years ago
comment:168

Hi there

I'm at the point where everything seems to work for me. I'd like to test this automatically it I can't see how. So provided someone testify that the new version of attachment: trac-6495_silence_warning-fh.patch works for him, I'm ready to put a positive review here.

Cheers,

Florent

jdemeyer commented 11 years ago

Changed reviewer from Volker Braun to Volker Braun, Florent Hivert