search

sagemath / sage

Main repository of SageMath
https://www.sagemath.org
Other
1.38k stars 469 forks source link

Build the reference manual incrementally #6495

Closed e14f4152-4982-4ace-8c95-73a0599b109b closed 11 years ago

e14f4152-4982-4ace-8c95-73a0599b109b commented 15 years ago

Building the Sage reference manual can take a significant amount of time. Decreasing this time could speed up Sage development.

The patch is large, but most of it consists of moving files from one location to another, as described below. A summary of the changes:

Changes in doc/en/reference — this is where the size of the patch comes from, although the changes are pretty simple:

Changes to doc/common/builder.py:

Other changes:

The html docs for Sage 5.4.rc2, html without MathJax, html with MathJax, and pdf, built after applying the patches here.

Apply:

Before building the docs, you should delete the documentation output directory: rm -rf SAGE_ROOT/devel/sage/doc/output. To test this, you should run sage --docbuild all html and sage --docbuild all pdf. (Note: just running sage --docbuild reference html will probably produce many warnings. If you run it a second time, the warnings should go away.)

Depends on #13064 Depends on #8327 Depends on #13891

Dependencies: 5.7.beta2 + #13064, #8327, #13891

CC: @jhpalmieri @nexttime @nilesjohnson @hivert @mguaypaq @mwhansen

Component: documentation

Keywords: days38

Author: Mitesh Patel, John Palmieri, Florent Hivert

Reviewer: Volker Braun, Florent Hivert

Merged: sage-5.8.beta0

Issue created by migration from https://trac.sagemath.org/ticket/6495

jhpalmieri commented 12 years ago
comment:38

Replying to @williamstein:

Why not exactly one environment variable "MAKE" which gets set to "make -jN" for some N, and that is it?

That's an interesting idea. See #12016.

jhpalmieri commented 12 years ago

Dependencies: #12016

jhpalmieri commented 12 years ago
comment:39

Here's a new patch which uses the setting of MAKE to build docs in parallel (or not). It's very similar to the code in sage-ptest from #12016, except that when you run sage-ptest, the assumption should be that you want to work in parallel, so the default number of threads (if MAKE is not set) is min(8, #cpus). For doc building, we shouldn't assume parallel building by default, I guess, so the default number of threads is 1.

jdemeyer commented 12 years ago
comment:40

Replying to @jhpalmieri:

For doc building, we shouldn't assume parallel building by default, I guess, so the default number of threads is 1.

Indeed!

jdemeyer commented 12 years ago
comment:41

Replying to @jhpalmieri:

Here's a new patch which uses the setting of MAKE to build docs in parallel (or not). It's very similar to the code in sage-ptest from #12016, except that when you run sage-ptest, the assumption should be that you want to work in parallel, so the default number of threads (if MAKE is not set) is min(8, #cpus). For doc building, we shouldn't assume parallel building by default, I guess, so the default number of threads is 1.

So then we don't need the environment variable SAGE_PARALLEL_DOCBUILD, nor the Makefile patch, anymore.

jdemeyer commented 12 years ago
comment:42

With sage-4.8.alpha1, I get

Building reference manual, first pass.

sphinx-build -b html -d /mnt/usb1/scratch/jdemeyer/merger/sage-4.8.alpha2/devel/sage/doc/output/doctrees/en/reference   -A hide_pdf_links=1 -Q  /mnt/usb1/scratch/jdemeyer/merger/sage-4.8.alpha2/devel/sage/doc/en/reference /mnt/usb1/scratch/jdemeyer/merger/sage-4.8.alpha2/devel/sage/doc/output/html/en/reference
Build finished.  The built documents can be found in /mnt/usb1/scratch/jdemeyer/merger/sage-4.8.alpha2/devel/sage/doc/output/html/en/reference
Traceback (most recent call last):
  File "/mnt/usb1/scratch/jdemeyer/merger/sage-4.8.alpha2/devel/sage/doc/common/builder.py", line 1332, in <module>
    getattr(get_builder(name), type)()
  File "/mnt/usb1/scratch/jdemeyer/merger/sage-4.8.alpha2/devel/sage/doc/common/builder.py", line 288, in _wrapper
    getattr(get_builder(document), 'html')(*args, **kwds)
  File "/mnt/usb1/scratch/jdemeyer/merger/sage-4.8.alpha2/devel/sage/doc/common/builder.py", line 405, in _wrapper
    N = re.search('(-j|--jobs[=]?)\s*([0-9]*)', MAKE).groups()[1]
UnboundLocalError: local variable 're' referenced before assignment
make: *** [doc-html] Error 1
jhpalmieri commented 12 years ago

Description changed:

--- 
+++ 
@@ -32,6 +32,5 @@

 - [attachment: trac_6495-part1-moving-files.patch](https://github.com/sagemath/sage-prod/files/10645464/trac_6495-part1-moving-files.patch.gz) — this moves 'algebras.rst' to 'algebras/index.rst', and similarly for all other files.  It adds `.. include:: ../footer.txt` to the end of each of these files, and it removes any cross-referencing information like `.. _ch:groups:`, since that doesn't work anymore with the new structure.  It also creates identical files 'DIR/conf.py' in each of the new subdirectories of doc/en/reference, except for doc/en/algebras/conf.py.  That file is created in the next patch so that you can focus on reviewing just the second patch.
 - [attachment: trac_6495-part2-everything-else.patch](https://github.com/sagemath/sage-prod/files/10645471/trac_6495-part2-everything-else.patch.gz) — this does everything else; in other words, all of the important content is in this patch.
-- [attachment: trac_6495-root.patch](https://github.com/sagemath/sage-prod/files/10645446/trac_6495-root.patch.gz) — root repo.  Add "doc-parallel" and "doc-pdf-parallel" targets to the main Makefile.

 Before building the docs, you should delete the documentation output directory: `rm -rf SAGE_ROOT/devel/sage/doc/output`.
jhpalmieri commented 12 years ago
comment:43

So then we don't need the environment variable SAGE_PARALLEL_DOCBUILD, nor the Makefile patch, anymore.

Right, fixed. Also, I added import re into builder.py; this should fix the other problem as well.

jhpalmieri commented 12 years ago
comment:44

This now uses SAGE_NUM_THREADS from #12016.

jdemeyer commented 12 years ago
comment:45

Regarding #12016: you should simply use the value of SAGE_NUM_THREADS, nothing fancier than that.

Regarding spkg-dist: this is essentially a duplicate of #12081 and #12086, so this patch should be removed. In any case, removing the files from MANIFEST.in is the proper way of dealing with this, as opposed to removing the files before packaging the repository. Ideally, sage-sdist should not change the current Sage source tree at all.

What's the rationale for adding all these files to doc/common/themes/sageref?

Instead of always building twice, would it be possible to detect whether the manual has already been built once. For example, if I want both the HTML and PDF documentation, the current patch would do 4 passes, even if 3 would be sufficient.

jhpalmieri commented 12 years ago

Changed keywords from sd32 to none

jhpalmieri commented 12 years ago
comment:47

Rebased to 5.0.beta13, but the intersphinx stuff needs fixing (the use of intersphinx here conflicts with the changes in #9128, and I haven't fixed this). Part 1 of the patch was mostly generated automatically using the attached script. After running the script, I removed lines like .. _ch:algebras: (from algebras/index.rst, in this case) by hand.

Replying to @jdemeyer:

Regarding #12016: you should simply use the value of SAGE_NUM_THREADS, nothing fancier than that.

Okay, done.

Regarding spkg-dist: this is essentially a duplicate of #12081 and #12086, so this patch should be removed. In any case, removing the files from MANIFEST.in is the proper way of dealing with this, as opposed to removing the files before packaging the repository. Ideally, sage-sdist should not change the current Sage source tree at all.

I removed that part of the patch.

What's the rationale for adding all these files to doc/common/themes/sageref?

The new structure of the reference manual, in particular the new directory structure, means we need new templates for the files coming from reference/algebras/index.rst, as opposed to the old templates, which work for the main file reference/index.rst.

Instead of always building twice, would it be possible to detect whether the manual has already been built once. For example, if I want both the HTML and PDF documentation, the current patch would do 4 passes, even if 3 would be sufficient.

I don't know how to do this.

jhpalmieri commented 12 years ago

Work Issues: intersphinx

jhpalmieri commented 12 years ago

Description changed:

--- 
+++ 
@@ -23,7 +23,6 @@
 - `doc/common/themes/sage/layout.html`: fix a bug where clicking the Sage logo in the upper left corner of the docs wouldn't take you to the right place, at least in the local documentation.
 - `doc/common/themes/sageref/`: add a new theme for the pieces of the reference manual.  This is almost identical to the "sage" theme, except for a little tinkering to the links along the top and bottom lines.
 - in the main Sage library, change a few pathnames to media files in the reference manual, since those files have been moved.
-- `spkg-dist`: when building the main sage spkg file, delete all of the autogenerated files from the doc directory.  This is important because if some autogenerated files from before the patch are still present after the patch, the docbuild process can occasionally get confused.  It also saves some space, making a smaller spkg file.
 - make the necessary changes to .hgignore and MANIFEST.in to deal with the relocated files.

 ---
jdemeyer commented 12 years ago
comment:49

This patch would allow building the reference manual with less memory usage. 1.5GB (still a lot) is sufficient to build the manual, as opposed to more than 2GB without the patch.

jdemeyer commented 12 years ago
comment:50

I do get several warnings when building the HTML manual. The following is repeated many times:

sphinx-build -b html -d /fast_scratch/jdemeyer/sage-5.0.beta13/devel/sage/doc/output/doctrees/de/tutorial   -A hide_pdf_links=1  /fast_scr
atch/jdemeyer/sage-5.0.beta13/devel/sage/doc/de/tutorial /fast_scratch/jdemeyer/sage-5.0.beta13/devel/sage/doc/output/html/de/tutorial
Running Sphinx v1.1.2
loading translations [de]... done
loading pickled environment... not yet created
loading intersphinx inventory from /fast_scratch/jdemeyer/sage-5.0.beta13/devel/sage/doc/common/python.inv...
loading intersphinx inventory from /fast_scratch/jdemeyer/sage-5.0.beta13/devel/sage/doc/output/html/en/reference/objects.inv...
WARNING: intersphinx inventory '/fast_scratch/jdemeyer/sage-5.0.beta13/devel/sage/doc/output/html/en/reference/objects.inv' not fetchable
due to <type 'exceptions.IOError'>: [Errno 2] No such file or directory: '/fast_scratch/jdemeyer/sage-5.0.beta13/devel/sage/doc/output/htm
l/en/reference/objects.inv'
building [html]: targets for 22 source files that are out of date
updating environment: 22 added, 0 changed, 0 removed
[...]
Build finished.  The built documents can be found in /fast_scratch/jdemeyer/sage-5.0.beta13/devel/sage/doc/output/html/de/tutorial

There are several of the form:

sphinx-build -b html -d /fast_scratch/jdemeyer/sage-5.0.beta13/devel/sage/doc/output/doctrees/en/reference/schemes   -A hide_pdf_links=1  -q  -a  /fast_scratch/jdemeyer/sage-5.0.beta13/devel/sage/doc/en/reference/schemes /fast_scratch/jdemeyer/sage-5.0.beta13/devel/sage/doc/output/html/en/reference/schemes
None:37: WARNING: citation not found: Fulton
Build finished.  The built documents can be found in /fast_scratch/jdemeyer/sage-5.0.beta13/devel/sage/doc/output/html/en/reference/schemes
jhpalmieri commented 12 years ago
comment:51

The intersphinx stuff actually seems to be working, despite my earlier comment. I'll keep testing it to make sure, but I think it's okay.

Regarding the warnings: the missing intersphinx inventories are expected. That's why we have to build everything twice: once to create all of the inventory files, and then a second time to use them. At least for me, if I run sage --docbuild reference html twice in a row, I don't get these warnings the second time through. The citation warnings need to be fixed, though.

jhpalmieri commented 12 years ago

Changed work issues from intersphinx to citations

hivert commented 12 years ago
comment:52

Hi John,

What is the status of this one ? It this robust ?

I'm quite glad that I didn't stomp on your foot with #9128, but I may be right now in the process on jumping high to land on your foot ;-/: I just dived deep inside Sphinx and using @parallel, I managed to have the "writing output..." part of the doc compilation in parallel. I've right now no idea how robust it is. I'll put a log of my experiment on #6255. Also, it seems possible that the read part could also be made parallel.

Florent

jhpalmieri commented 12 years ago
comment:53

Hi Florent,

This is in pretty good shape, but it's not perfect. It undoes some of what you did in #9128 (mainly because I haven't tried to rewrite the patch to do it differently), and in particular, I'm not sure that the other parts of the Sage documentation can use intersphinx to access information from the reference manual. Citations may be an issue, in particular if the same reference is cited twice in two different parts of the reference manual: we may just need to add another copy of the citation, or perhaps a master list of citations that gets used by everything. I don't know if that's practical.

There are also issues with having to build the reference manual twice so that all of the references are resolved. This is not ideal.

I think that doing the reading and/or writing in parallel would be good, but given the size of the reference manual, breaking it into pieces seems worthwhile as well. If the parallel reading and writing help to cut down on the memory usage, which seems to be getting out of hand, then maybe that is good enough for now. (At least on sage.math, the writing part seems to take way too long, so doing that in parallel might help significantly.)

So if you have a workable solution which accomplishes some of what is done here, and perhaps does it more simply, go right ahead. I'll take a look at your comments at #6255.

John

hivert commented 12 years ago
comment:54

Hi John,

Thanks for the quick answer.

This is in pretty good shape, but it's not perfect. It undoes some of what you did in #9128 (mainly because I haven't tried to rewrite the patch to do it differently), and in particular, I'm not sure that the other parts of the Sage documentation can use intersphinx to access information from the reference manual.

I'll have a look at it. Please do not hesitate to ask for some more explanation on the hack I did with intersphinx. Is there a specific reason why you doubt intersphinx will work for the other part of the doc ?

There are also issues with having to build the reference manual twice so that all of the references are resolved. This is not ideal.

It doesn't seem to be a huge problem with LaTeX, since this never has been solved since years... Though I never seen a LaTeX compilation as long as Sage's doc.

I think that doing the reading and/or writing in parallel would be good, but given the size of the reference manual, breaking it into pieces seems worthwhile as well.

I agree.

If the parallel reading and writing help to cut down on the memory usage, which seems to be getting out of hand, then maybe that is good enough for now. (At least on sage.math, the writing part seems to take way too long, so doing that in parallel might help significantly.)

I don't think it will cut down memory usage in any way. I'd rather expect the contrary. My solution seems to be working but since I currently for a sage for every single file, a lot of time is wasted in forking. I'll try to improve it tomorrow.

So if you have a workable solution which accomplishes some of what is done here, and perhaps does it more simply, go right ahead. I'll take a look at your comments at #6255.

I don't think I really will. As I said I'll probably trade speed against memory usage... I'll keep you in touch.

Cheers,

Florent

jdemeyer commented 12 years ago
comment:55

Replying to @hivert:

I don't think it will cut down memory usage in any way. I'd rather expect the contrary.

Memory usage is already too much, so anything that further increases memory usage is very bad.

hivert commented 12 years ago
comment:56

Replying to @jdemeyer:

Replying to @hivert:

I don't think it will cut down memory usage in any way. I'd rather expect the contrary.

Memory usage is already too much, so anything that further increases memory usage is very bad.

I respectfully disagree if

83660e46-0051-498b-a8c1-f7a7bd232b5a commented 12 years ago
comment:57

Replying to @hivert:

Replying to @jdemeyer:

Replying to @hivert:

I don't think it will cut down memory usage in any way. I'd rather expect the contrary.

Memory usage is already too much, so anything that further increases memory usage is very bad.

Like adding more modules and functions to the Sage library, and rising the "doc[test] coverage"... ;-)

I respectfully disagree if

  • it is optional

That would be fine.

  • it divides by more than 10 the documentation compile time

Wow... :P

hivert commented 12 years ago
comment:58

This is an extremely useful patch which should go in Sage ASAP. It effectively reduce the compilation time to 10min on my I7 laptop. This is brilliant. It will certainly prove more maintainable in the long term than my parallelization of Sphinx expertiment. Though, I think we should discuss of is with Georges Brandl (the maintainer of Sphinx).

I'm now reviewing the code itself. You'll find below some comments.

Florent

Suggestion of improvement:

Some remaining problems:

hivert commented 12 years ago
comment:59

Hi John,

Using your patch I realized that for the first stage, you don't need to get the html file only the pickle and object.inv file. Did you try to do that ? It seems that this is easily feasible with a custom Builder. I'm currently testing this workflow.

Florent

hivert commented 12 years ago
comment:60

Another comment: You should start by building the reference manual and then the other component. Currently in class AllBuilder method _wrapper you start by building the other documents.

Florent

hivert commented 12 years ago
comment:61

I just attached a new patch which defines a new builder for creating the pickle and object.inv file. It can be called by 

sage -docbuild DOCUMENT invpickle

It is automatically called for the first pass when building DOCUMENT=all. As a consequence on my laptop the first pass in only 2 min and a half long.

Florent

hivert commented 12 years ago
comment:62

The invpickle builder seem quite efficient here are the timing on my I7 laptop:

Florent

hivert commented 12 years ago
comment:63

There is also a failure:

sage -t  builder.py
File "/home/data/Sage-Install/sage-5.0.beta13/devel/sage-doc/doc/common/builder.py", line 862:
    [...]
    sage: builder.ReferenceBuilder("reference").auto_rest_filename("sage.combinat.partition")
      File "/home/data/Sage-Install/sage-5.0.beta13/devel/sage/doc/common/builder.py", line 409, in _wrapper
        getattr(DocBuilder(self.name, lang), format)(*args, **kwds)
    AttributeError: 'DocBuilder' object has no attribute 'auto_rest_filename'
jhpalmieri commented 12 years ago
comment:64

Hi Florent,

Your ideas look great; I hope you can keep working on them. I am happy with all of your suggestions. In particular:

By the way, mpatel should get a lot of the credit for this; he wrote the original patches, which I've just cleaned up and reorganized.

jhpalmieri commented 12 years ago
comment:65

I got two doctest failures (after applying your patch), which can be fixed with this patch:

diff --git a/doc/common/builder.py b/doc/common/builder.py
--- a/doc/common/builder.py
+++ b/doc/common/builder.py
@@ -200,7 +200,7 @@ class DocBuilder(object):
             sage: import os, sys; sys.path.append(os.environ['SAGE_DOC']+'/common/'); import builder
             sage: b = builder.DocBuilder('tutorial')
             sage: b._output_formats()
-            ['changes', 'html', 'htmlhelp', 'json', 'latex', 'linkcheck', 'pickle', 'web']
+            ['changes', 'html', 'htmlhelp', 'invpickle', 'json', 'latex', 'linkcheck', 'pickle', 'web']

         """
         #Go through all the attributes of self and check to
@@ -859,7 +859,7 @@ class ReferenceSubBuilder(DocBuilder):

             sage: import os, sys; sys.path.append(os.environ['SAGE_DOC']+'/common/'); import builder
             sage: import builder
-            sage: builder.ReferenceBuilder("reference").auto_rest_filename("sage.combinat.partition")
+            sage: builder.ReferenceSubBuilder("reference").auto_rest_filename("sage.combinat.partition")
             '.../devel/sage/doc/en/reference/sage/combinat/partition.rst'
         """
         return self.dir + os.path.sep + module_name.replace('.',os.path.sep) + '.rst'
jhpalmieri commented 12 years ago
comment:66

By the way, searching is broken with these patches. The files searchindex.js (both in reference/ and in its various subdirectories) are all essentially empty. Hmm.

It looks like combining the indices should be doable, but a little annoying: read in the html from each, extract the entries, sort them, and rewrite to a new index file. The same should work for the module index. I think talking to Georg Brandl makes sense: if we wanted Sphinx to support this kind of thing out of the box, what exactly would we be looking for? I guess at the least we want separate Sphinx documents which share indices, searches, and inventories. It would be nice to have a specific "Sphinx Enhancement Proposal", or even some code to contribute.

hivert commented 12 years ago
comment:67

Hi John,

Another reason to have this in Sage ASAP: #12878

I'm willing to help on this one but we need to coordinate. I'll be in Montreal for the next Sage days 38 so hopefully will have so time for Sphinx. Do you think you'll have so time on it in the forthcoming weeks ? Please tell me what I can do to help. Here are a few suggestions:

Cheers,

Florent

jhpalmieri commented 12 years ago
comment:68

I hope I have time to work on it next week, although right now I have other things I need to be doing (like reading my student's PhD thesis).

Also:

I will try to post any progress that I make, and you should do likewise, so we don't end up duplicating each other's work.

jhpalmieri commented 12 years ago
comment:69

As far as handling ctrl-c, I found this question. It seems we can replace pool.apply_async(ARGS) with pool.apply_async(ARGS).get(999999). This apparently adds a (very long) timeout to the process, and it apparently works around a bug in Python. In practice, it seems to work for me.

diff --git a/doc/common/builder.py b/doc/common/builder.py
--- a/doc/common/builder.py
+++ b/doc/common/builder.py
@@ -415,7 +415,7 @@ class ReferenceBuilder(AllBuilder):
             for doc in self.get_all_documents(refdir):
                 pool.apply_async(build_ref_doc,
                                  (doc, lang, format,
-                                  os.path.split(output_dir)[0]) + args, kwds)
+                                  os.path.split(output_dir)[0]) + args, kwds).get(999999)
             pool.close()
             pool.join()
             if format == 'html':
83660e46-0051-498b-a8c1-f7a7bd232b5a commented 12 years ago
comment:70

Replying to @jhpalmieri:

  • I really like the idea of the invpickle builder, so please continue with that. I don't have any better suggestions for the name.

One should perhaps just expand the name to contain "inventory"; "pickle" is IMHO minor (or irrelevant) and won't tell most(?) people much.

  • The verbosity of intersphinx is a minor issue. If we can reduce it, that would be fine, but it's not the highest priority, in my opinion.

Mine, too. Although I hate Sphinx's / Sage's current messages already, especially since "Build succeeded. The built documents can be found in ..." is always printed, so is plain wrong in case of an error. (We tried to fix that once, but then I think Jeroen decided to keep it as is.)

  • I think it shouldn't be too hard to parallelize building the rest of the docs. We want to do it so that hitting ctrl-c will quit the build. I think we should try using tools from sage.parallel rather than the Python multiprocessing module. If you want to work on that, that would be great.

Why not just use make? Either add (and change the) targets in the top-level Makefile, or add one to devel/sage/doc/. To me seems cleanest (preferably the latter), and docbuilding IMHO shouldn't depend [more] on the Sage library [as needed / it already does].

jhpalmieri commented 12 years ago
comment:71

The problem with using make is that I want to be able to run sage --docbuild all html, and I don't think that running this should call make. The fix I mentioned above for interrupts (adding a timeout) doesn't make use of the sage.parallel module, just plain Python, by the way. So we should be able to do this for building all of the documents in parallel.

hivert commented 12 years ago
comment:72

Replying to @jhpalmieri:

As far as handling ctrl-c, I found this question. It seems we can replace pool.apply_async(ARGS) with pool.apply_async(ARGS).get(999999). This apparently adds a (very long) timeout to the process, and it apparently works around a bug in Python. In practice, it seems to work for me.

Does it really work ? For me is seems that ctrl-c is now working but the doc is no more compiling in parallel.

Florent

hivert commented 12 years ago
comment:73

Concerning todos (and maybe indexes too), I think I've found an alternative: they are pickled in the files environment.pickle. Here is an example, in the directory doc/output/doctrees/en/reference/modules:

sage: import cPickle
sage: f = open('environment.pickle', 'rb')
sage: env = cPickle.load(f)
sage: f.close()
sage: env.todo_all_todos
[{'docname': 'sage/modules/free_module', 'source': u'/home/data/Sage-Install/sage-5.0.beta14/local/lib/python2.7/site-packages/sage/modules/free_module.py:docstring of sage.modules.free_module.FreeModuleFactory', 'todo': <todo_node: <title...><paragraph...>>, 'lineno': 122, 'target': <target: >}]
sage: env.indexentries.keys()
['index', 'sage/modules/free_module_element', 'sage/modules/real_double_vector', 'sage/modules/matrix_morphism', 'sage/modules/fg_pid/fgp_module', 'sage/modules/free_module_homspace', 'sage/modules/vector_space_homspace', 'sage/modules/fg_pid/fgp_element', 'sage/modules/complex_double_vector', 'sage/modules/fg_pid/fgp_morphism', 'sage/modules/vector_callable_symbolic_dense', 'sage/modules/module', 'sage/modules/free_module_morphism', 'sage/modules/vector_space_morphism', 'sage/modules/free_module']

So it seems that we can get them from there from the pickle concatenate them and let sphinx output the todo list and the index.

Florent

hivert commented 12 years ago
comment:74

Replying to @hivert:

So it seems that we can get them from there from the pickle concatenate them and let sphinx output the todo list and the index.

I've a proof of concept which seems to be working upto two problems:

I think both problem can be fixed.

hivert commented 12 years ago
comment:75

Hi there,

I just uploaded a new attachment: invbuilder.patch; you need to apply it after the two mentionned patch in the header of this ticket. It seems to solve the problems of

It remains to merge the javascript indexes.

Please tests.

Florent

hivert commented 12 years ago
comment:76

I just uploaded a new patch which merge the javascript indexes as well !!! It was a little more tricky but it seems to work now. The only things that remains to be merged is the list of modules. I'll write a proposal to Sphinx for my merging tool.

Florent

hivert commented 12 years ago
comment:77

The new version now also merge modules indexes !!! The obtained doc seems to be in quite a good shape.

hivert commented 12 years ago

inventory builder + merge todo list & html / js indexes

hivert commented 12 years ago
comment:78

Attachment: invbuilder.patch.gz

Citations may be an issue, in particular if the same reference is cited twice in two different parts of the reference manual: we may just need to add another copy of the citation, or perhaps a master list of citations that gets used by everything. I don't know if that's practical.

I've probably a way to handle cross-citation as well. They are stored in the environement and I can get them to gather the link in the main reference manual, exactly the way I gather TODO and indexes. It is just a little more tricky because I need to redispatch them to the other documents. I'm experimenting...

Florent

hivert commented 12 years ago
comment:79

Attachment: trac_6495-part3-the-remaining.patch.gz

Hi there,

I just uploaded which seems to solve most of the problems we had here, including merging of

I put a need review though I'm quite not sure everything is ready for inclusion into Sage. I do need feedback and people shaking my code to see if it's robust.

Florent

hivert commented 12 years ago

Changed author from Mitesh Patel, John Palmieri to Mitesh Patel, John Palmieri, Florent Hivert

hivert commented 12 years ago

Description changed:

--- 
+++ 
@@ -31,5 +31,7 @@

 - [attachment: trac_6495-part1-moving-files.patch](https://github.com/sagemath/sage-prod/files/10645464/trac_6495-part1-moving-files.patch.gz) — this moves 'algebras.rst' to 'algebras/index.rst', and similarly for all other files.  It adds `.. include:: ../footer.txt` to the end of each of these files, and it removes any cross-referencing information like `.. _ch:groups:`, since that doesn't work anymore with the new structure.  It also creates identical files 'DIR/conf.py' in each of the new subdirectories of doc/en/reference, except for doc/en/algebras/conf.py.  That file is created in the next patch so that you can focus on reviewing just the second patch.
 - [attachment: trac_6495-part2-everything-else.patch](https://github.com/sagemath/sage-prod/files/10645471/trac_6495-part2-everything-else.patch.gz) — this does everything else; in other words, all of the important content is in this patch.
+- [attachment: trac_6495-part3-the-remaining.patch](https://github.com/sagemath/sage-prod/files/10645448/trac_6495-part3-the-remaining.patch.gz) — this patch takes care of merging the indexes and the bibliography.

 Before building the docs, you should delete the documentation output directory: `rm -rf SAGE_ROOT/devel/sage/doc/output`.
+
hivert commented 12 years ago

Changed work issues from citations to none