building the doc

[kiwifb]

It would be nice to have at least an experimental way of building the doc again. Plenty of work has been done upstream that we could try to leverage.

[kiwifb]

Well the doc can be built in parallel now. I have an experimental sage-doc ebuild but it mysteriously stale after the first pass. It does all en/reference and hangs. Very frustrating.

[kiwifb]

It is bizarre, it looks like the workers don't stop and and return they just hang around after they are finished and they are not terminated by a parent process. Possibly they cannot communicate with it. Things seem to work from vanilla sage even after upgrading docutils and sphinx. There are errors/warnings of course but no hangs.

[kiwifb]

While for a while sage-doc-9999-r1 worked it is now officially broken, sage's sphinx formatted documentation is officially too old for being built in sage-on-gentoo.

[strogdon]

I have a few patches that may assist in building sage-doc-9999-r1. It seems to work for available sphinx and docutils-0.9.1-r1, possibly all docutils except 0.10. That version needs a little TLC. I believe the changes that allow using docutils-0.9.1-r1 will also be needed with 0.10. I will send patches privately. Using docutils-0.9.1-r1 and <sphinx-1.2 is not optimal since one has a python_targets_python3_2/python_targets_python3_3 useflag nightmare. Anyway, one doesn't want to build the docs too many times, it requires an incredible amount of cpu cycles and this may be too much to maintain.

[kiwifb]

It may be. I presume you are not up to use github to make pull requests? I would make including your work much easier.

[strogdon]

I didn't think I could do that, but will investigate.

[strogdon]

The patches I had prepared required that the to-be-installed sage sources be patched. I don't think one should have to do this. The sources should be patched in the appropriate PORTAGE directory during building of sage-doc. In the ebuild SAGE_DOC and SAGE_SRC are exported as pointing to appropriate PORTAGE directories, however in common/builder.py we have from sage.env import SAGE_DOC, SAGE_SRC and in common/conf.py there is from sage.env import SAGE_DOC. And SAGE_DOC and SAGE_SRC point to locations in the system tree. If I remove these import lines from conf.py and builder.py then during building it becomes clear that the exports in the ebuild aren't doing the intended job; SAGE_DOC and SAGE_SRC are undefined inside builder.py.

[kiwifb]

I may look at that tonight defining those variables is an integral part of building things, I thought I had given correct definitions in the ebuild.

[kiwifb]

OK I checked the ebuild and what you said. env.py checks whether the variables are defined before assigning them the default value. In short you can always override the default system value with your own. I made sure that feature was preserved. If, when builder.py is run you get the system value, something wrong is happening that needs further debugging.

[strogdon]

OK, not only is there from sage.env import SAGE_DOC, SAGE_SRC in common/builder.py but there is also import sage.all around line 1466 and in sage.all there is from sage.env import SAGE_ROOT, SAGE_DOC, SAGE_LOCAL, DOT_SAGE, SAGE_ENV. It seems that both of these change what was initially exported for SAGE_DOC in the ebuild.

[strogdon]

I didn't see that you had posted before I posted above. If sage.py does check whether variables are set then something is changing things.

[kiwifb]

Hum, plenty of silly stuff in build_option.py but it should still work. I probably should open a ticket to remove the from sage.env line from all.py. I don't think anything import these variables from sage.all and if they do they should be fixed. But I don't see how mechanically we lose the values of SAGE_DOC and SAGE_SRC. I will have to spend some time with the ebuild tomorrow. Nice set of patches by the way. It is easier than I thought and yes they should go in sage-doc only. So according to you SAGE_SRC is getting lost. If SAGE_DOC was getting set to default you'd get sandbox violation.

[strogdon]

Sandbox, I should have thought of that. Yes, the problem is from where things are being read not where they are being written. Here is the symptom when no patches are applied to the in-tree Sage:

[combinat ] reading sources... [ 27%] sage/combinat/finite_state_machine
Error building the documentation.

Note: incremental documentation builds sometimes cause spurious
error messages. To be certain that these are real errors, run
"make doc-clean" first and try again.
Traceback (most recent call last):
  File "common/builder.py", line 1474, in <module>
    getattr(get_builder(name), type)()
  File "common/builder.py", line 269, in _wrapper
    getattr(get_builder(document), 'inventory')(*args, **kwds)
  File "common/builder.py", line 485, in _wrapper
    x.get(99999)
  File "/usr/lib64/python2.7/multiprocessing/pool.py", line 554, in get
    raise self._value
OSError: [graphs   ] /usr/lib64/python2.7/site-packages/sage/graphs/generic_graph.py:docstring of sage.graphs.generic_graph.GenericGraph.breadth_first_search:24: ERROR: Content block expected for the "SEEALSO" directive; none found.

So SAGE_SRC is now /usr/lib64/python2.7/site-packages. Or at the very least files are being looked for under site-packages.

[kiwifb]

I think the documentation framework may be calling sage to get some of its info which may why it doesn't look like it respect SAGE_SRC. There is still some work to do to check that claim.

[kiwifb]

Will add stuff to sage-9999 to see how it behaves. But I need to update gap to 4.7.4 first for beta3 it may take a couple of days because things are pretty hectic for me right now.

[kiwifb]

Waiting for Volker to release a tarball for libgap 4.7.4. Could host one on lmona.de but I'd rather wait for it. I have many commits queued or in progress.

[kiwifb]

Massive push. Things may break.

[strogdon]

With sphinx-1.1.3-r7 and docutils-0.9.1-r1 the html docs seem to build. However there is an issue with the pdf-docs. There are several warnings:

[dynamics ] reading sources... [ 70%] sage/dynamics/interval_exchanges/iet
[cmd      ] /var/tmp/portage/sci-mathematics/sage-doc-9999-r1/work/sage-9999/src/doc/en/reference/cmd/options.rst:: WARNING: unusable reference target found: ../../../../html/en/reference/misc/s
age/misc/sagedoc.html#sage.misc.sagedoc.search_src
[cmd      ] /var/tmp/portage/sci-mathematics/sage-doc-9999-r1/work/sage-9999/src/doc/en/reference/cmd/options.rst:: WARNING: unusable reference target found: ../../../../html/en/reference/misc/s
age/misc/sagedoc.html#sage.misc.sagedoc.search_doc
[cmd      ] /var/tmp/portage/sci-mathematics/sage-doc-9999-r1/work/sage-9999/src/doc/en/reference/cmd/index.rst:: WARNING: unusable reference target found: ../genindex.html
[cmd      ] /var/tmp/portage/sci-mathematics/sage-doc-9999-r1/work/sage-9999/src/doc/en/reference/cmd/index.rst:: WARNING: unusable reference target found: ../py-modindex.html
[cmd      ] /var/tmp/portage/sci-mathematics/sage-doc-9999-r1/work/sage-9999/src/doc/en/reference/cmd/index.rst:: WARNING: unusable reference target found: ../search.html
[cmd      ] writing... done

The unfound targets are referenced in src/doc/en/reference/footer.txt. I don't know about this. The major failure is due to:

/bin/sh: all-pdf: command not found
...
RuntimeError: failed to run $MAKE all-pdf in /var/tmp/portage/sci-mathematics/sage-doc-9999-r1/work/sage-9999/src/doc/output/latex/en/reference/cmd

I've seen this before and it would appear that MAKE is not defined.

[kiwifb]

Yes a clear symptom since it tries to execute all-pdf as a command. It expects to be called from a parent makefile. Will fix that later tonight. Hadn't tested the pdf.

[kiwifb]

Now that singular is out of the way I will look at this.

[kiwifb]

beurk in builder.py:

    def pdf(self):
        """
        Builds the PDF files for this document.  This is done by first
        (re)-building the LaTeX output, going into that LaTeX
        directory, and running 'make all-pdf' there.

        EXAMPLES::

            sage: import os, sys; sys.path.append(os.environ['SAGE_DOC']+'/common/'); import builder
            sage: b = builder.DocBuilder('tutorial')
            sage: b.pdf() #not tested
        """
        self.latex()
        tex_dir = self._output_dir('latex')
        pdf_dir = self._output_dir('pdf')
        if subprocess.call("cd '%s' && $MAKE all-pdf && mv -f *.pdf '%s'"%(tex_dir, pdf_dir), shell=True):
            raise RuntimeError("failed to run $MAKE all-pdf in %s"%tex_dir)

        logger.warning("Build finished.  The built documents can be found in %s", pdf_dir)

Have to inspect what exactly it wants to do but I think there will be surgery. But that explains the calll to make.

[strogdon]

From a vanilla Sage perspective it seems that MAKE has to be exported in some fashion to get parallel building to work. Perhaps this is in effect when building the pdf-docs. So I thought, perhaps, MAKE could be exported in the sage-doc ebuild. But perhaps there are reasons not to do that. And of course the html docs seem to build in parallel without exporting MAKE?

[kiwifb]

Was about to write something. Not sure how things have broken down yet. but sphinx is supposed to copy a makefile from /usr/lib64/python2.7/site-packages/sphinx/texinputs/Makefile into src/doc/output/latex/en/reference/cmd (or somewhere in that hierarchy) and then make is run. self.latex() should do that I think.

[kiwifb]

Doesn't look like I have even a html output at the moment. Too many errors. undefined labels kills everything. I don't get the error with pdf. IT fails but keep going. I have docutils 0.10/sphinx 1.1.3-r7, are you still at docutils 0.9/sphinx 1.2.2?

[strogdon]

Things will not work with docutils-0.10. docutils-0.9.1-r1 is okay. I've tried sphinx-1.1.3-r7 and sphinx-1.2.1 with success. I'm beginning to think there is a bug in docutils-0.10. I can get rid of the undefined labels but the resulting html is missing links to the "linked" figure. I'm assuming your problem is in tutorial.py and with figure-examples-catalan-trees. The patch to remove the undefined label also works with <docutils-0.10 where it's not needed. Here is the patch to get things to build with docutils-0.10 if downgrading is not possible

--- sage/combinat/tutorial.py.orig      2014-03-10 16:04:48.595069424 -0500
+++ sage/combinat/tutorial.py   2014-03-10 17:21:39.171373854 -0500
@@ -205,7 +205,7 @@

 A *complete binary tree* is either a leaf `\mathrm{L}`, or a
 node to which two complete binary trees are attached (see
-:ref:`figure-examples-catalan-trees`).
+:ref:`Figure: The five complete binary trees with four leaves <figure-examples-catalan-tr
ees>`).

 .. _figure-examples-catalan-trees:

@@ -1412,7 +1412,7 @@
         sage: function('Node', nargs=2)
         Node

-    The second tree in :ref:`figure-examples-catalan-trees`
+    The second tree in :ref:`Figure: The five complete binary trees with four leaves <fig
ure-examples-catalan-trees>`
     can be represented by the expression::

         sage: tr = Node(Node(Leaf, Node(Leaf, Leaf)), Leaf)
@@ -1893,4 +1893,4 @@
    children to explore, and to reduce the cost of each test of
    canonicity.

-"""
\ No newline at end of file
+"""

[kiwifb]

Yes I thought you said things about docutils earlier. I don't think there is anything forcing the upgrade to docutils 0.10. I will put a dependency to <0.10 in sage-doc

[kiwifb]

Hum, only 0.9.1-r1 and 0.10 are python-r1 the rest will probably be dropped, we may soon have to live with 0.10.

[kiwifb]

OK now that I have the stuff I think exporting make will be enough, Not special way of setting make though which I assume may be bothersome on exotic target like AIX.

[kiwifb]

I got pdf doc with the latest commit :)

[kiwifb]

I got quite a number of missing html files according to my doctest run. I am not sure your patch will be enough to solve that problem but we'll try once i completed the work on beta4. baselayout need a change but I know what to do. It is just finding the time to do it. Hopefully that's the only bit missing coming from beta4.

[kiwifb]

Looks like my earlier problems may have been due to inconsistencies between the versions of sage (6.2.beta3) and sage-doc (may have been 6.2.beta4). Now that everything is consitent to 6.2.beta4 my missing html pages are not a problem anymore.

[strogdon]

I see that sphinx-1.2.2 is in the main tree and I'm having issues with it in prefix. I also noticed that sage-doc will fail and instead of aborting it installs anyway at the point of failure. This is probably not good.

[kiwifb]

Definitely not good. I probably need to add some "die"s in there. Can you post the failing bits?

[strogdon]

This is with sphinx-1.2.2 in prefix. I suppose is could be avoided by not using that sphinx version? I'll post the Gentoo result later today.

[reference] ... done (30785 js index entries)
[reference] Writing js search indexes... genindex py-modindex search
[reference] WARNING: favicon file 'favicon.ico' does not exist
[reference] WARNING: favicon file 'favicon.ico' does not exist
Traceback (most recent call last):
  File "doc/common/builder.py", line 83, in f
    execfile(sys.argv[0])
  File "/storage/strogdon/gentoo-redlizard/var/tmp/portage/sci-mathematics/sage-doc-9999-r1/work/sage-9999/src/doc/common/custom-sphinx-build.py", line 185, in <module>
    sphinx.cmdline.main(sys.argv)
  File "/storage/strogdon/gentoo-redlizard/usr/lib/python2.7/site-packages/sphinx/cmdline.py", line 264, in main
    print >>error
  File "/storage/strogdon/gentoo-redlizard/var/tmp/portage/sci-mathematics/sage-doc-9999-r1/work/sage-9999/src/doc/common/custom-sphinx-build.py", line 165, in write
    self._write(str)
  File "/storage/strogdon/gentoo-redlizard/var/tmp/portage/sci-mathematics/sage-doc-9999-r1/work/sage-9999/src/doc/common/custom-sphinx-build.py", line 139, in _write
    self._log_line(line)
  File "/storage/strogdon/gentoo-redlizard/var/tmp/portage/sci-mathematics/sage-doc-9999-r1/work/sage-9999/src/doc/common/custom-sphinx-build.py", line 108, in _log_line
    raise OSError(line)
OSError: [reference] WARNING: favicon file 'favicon.ico' does not exist

And then it installs

>>> Source compiled.
>>> Test phase [not enabled]: sci-mathematics/sage-doc-9999-r1

>>> Install sage-doc-9999-r1 into /storage/strogdon/gentoo-redlizard/var/tmp/portage/sci-mathematics/sage-doc-9999-r1/image/ category sci-mathematics
 ^[[32;01m*^[[0m python2_7: running distutils-r1_run_phase python_install
 ^[[32;01m*^[[0m python2_7: running distutils-r1_run_phase distutils-r1_python_install_all
>>> Completed installing sage-doc-9999-r1 into /storage/strogdon/gentoo-redlizard/var/tmp/portage/sci-mathematics/sage-doc-9999-r1/image/

[strogdon]

The failure in Gentoo is identical to the above. So perhaps sphinx-1.2.2 has problems. Maybe some subtle change that impacts how Sage is calling Sphinx.

[kiwifb]

Looks bad, looks for a file that is expected. I just added some die in the ebuild. Hopefully that will prevent emerge to go on after the failure.

[kiwifb]

find doc/ -name favicon*
doc/common/themes/sage/static/favicon.ico
doc/common/themes/sageref/static/favicon.ico

so we have some favicon.ico it just isn't found or it is looking for extra.

[kiwifb]

May be the following could be a stop gap

--- sage-6.2.beta4/src/doc/common/conf.py       2014-03-12 09:00:43.000000000 +1300
+++ sage-patching/src/doc/common/conf.py        2014-03-13 09:54:10.172776654 +1300
@@ -181,7 +181,7 @@
 # The name of an image file (within the static path) to use as favicon of the
 # docs.  This file should be a Windows icon file (.ico) being 16x16 or 32x32
 # pixels large.
-html_favicon = 'favicon.ico'
+#html_favicon = 'favicon.ico'

 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,

[strogdon]

This seems to fix things. Visually, I don't see a change over a doc-build with sphinx-1.1.3 where the icon is included without a problem. The only difference is that in the index html files this line

<link rel="shortcut icon" href="_static/favicon.ico"/>

does not appear. It's not clear why it is there.

[kiwifb]

Funky that would duplicate a line in sage/misc/doc.py. This is in a call that generates a web page as far as I can tell. So the patch may just be removing some dead code.

[strogdon]

Here are the changes between 1.2.1 and 1.2.2 in sphinx/builders/html.py that's causing the problem

@@ -597,15 +597,17 @@
         if self.config.html_logo:
             logobase = path.basename(self.config.html_logo)
             logotarget = path.join(self.outdir, '_static', logobase)
-            if not path.isfile(logobase):
-                self.warn('logo file %r does not exist' % logobase)
+            if not path.isfile(path.join(self.confdir, self.config.html_logo)):
+                self.warn('logo file %r does not exist' % self.config.html_logo)
             elif not path.isfile(logotarget):
                 copyfile(path.join(self.confdir, self.config.html_logo),
                          logotarget)
         if self.config.html_favicon:
             iconbase = path.basename(self.config.html_favicon)
             icontarget = path.join(self.outdir, '_static', iconbase)
-            if not path.isfile(icontarget):
+            if not path.isfile(path.join(self.confdir, self.config.html_favicon)):
+                self.warn('favicon file %r does not exist' % self.config.html_favicon)
+            elif not path.isfile(icontarget):
                 copyfile(path.join(self.confdir, self.config.html_favicon),
                          icontarget)
         self.info('done')

Apparently, sphinx is looking for favicon.ico in the source folder for every document that's to be built. If I copy favicon.ico into en/reference then the reference docs are built but fails with

[website  ] WARNING: favicon file 'favicon.ico' does not exist
[website  ] WARNING: favicon file 'favicon.ico' does not exist
Traceback (most recent call last):
  File "doc/common/builder.py", line 83, in f
    execfile(sys.argv[0])
  File "/storage/strogdon/gentoo-redlizard/var/tmp/portage/sci-mathematics/sage-doc-9999-r1/work/sage-9999/src/doc/common/custom-sphinx-build.py", line 185, in <module>
    sphinx.cmdline.main(sys.argv)
  File "/storage/strogdon/gentoo-redlizard/usr/lib/python2.7/site-packages/sphinx/cmdline.py", line 264, in main
    print >>error
  File "/storage/strogdon/gentoo-redlizard/var/tmp/portage/sci-mathematics/sage-doc-9999-r1/work/sage-9999/src/doc/common/custom-sphinx-build.py", line 165, in write
    self._write(str)
  File "/storage/strogdon/gentoo-redlizard/var/tmp/portage/sci-mathematics/sage-doc-9999-r1/work/sage-9999/src/doc/common/custom-sphinx-build.py", line 139, in _write
    self._log_line(line)
  File "/storage/strogdon/gentoo-redlizard/var/tmp/portage/sci-mathematics/sage-doc-9999-r1/work/sage-9999/src/doc/common/custom-sphinx-build.py", line 108, in _log_line
    raise OSError(line)
OSError: [website  ] WARNING: favicon file 'favicon.ico' does not exist

So en/website needs the file too. What a mess. Since the line for html_logo is commented out in conf.py then perhaps the best thing is comment out the line for html_favicon as well.

[kiwifb]

OK I will push the fix later today. But to me it looks very strange, you should have a warning but it should fall back to the old location if I am reading the code well. I think the failure is not intended but a side effect of something else that the sphinx devs may have overlooked.

[strogdon]

You're right. Sage must be raising the OSError. Try the following:

--- src/doc/common/custom-sphinx-build.py.orig  2014-03-13 23:05:26.180421561 -0500
+++ src/doc/common/custom-sphinx-build.py       2014-03-13 23:05:37.428445203 -0500
@@ -28,7 +28,8 @@
     re.compile('^looking for now-outdated files... none found'),
     re.compile('^building \[.*\]: targets for 0 source files that are out of date'),
     re.compile('^loading pickled environment... done'),
-    re.compile('^loading cross citations... done \([0-9]* citations\).')
+    re.compile('^loading cross citations... done \([0-9]* citations\).'),
+    re.compile('WARNING: favicon file \'favicon.ico\' does not exist')
     )

 # replacements: pairs of regular expressions and their replacements,

[kiwifb]

Oh that sounds devious but totally possible.

[kiwifb]

I suspect commit https://github.com/sagemath/sage/commit/eace780090b0bcc6445542589ab72c09ee6af4eb is at fault. I am looking at line 107 and 108 of custom-sphinx-build.py. So I'd say your last patch is actually the way to go.

[kiwifb]

I think we are done for now. We can now build the documentation for the "hot" ebuild.

[strogdon]

Not that this should be reopend but USE=html emerge -1 =sci-mathematics/sage-doc-9999-r1 fails here

Traceback (most recent call last):
  File "doc/common/builder.py", line 1477, in <module>
    getattr(get_builder(name), type)()
  File "doc/common/builder.py", line 276, in _wrapper
    getattr(get_builder(document), 'inventory')(*args, **kwds)
  File "doc/common/builder.py", line 487, in _wrapper
    x.get(99999)
  File "/storage/strogdon/gentoo-redlizard/usr/lib/python2.7/multiprocessing/pool.py", line 554, in get
    raise self._value
OSError: [libs     ] docstring of sage.libs.lrcalc.lrcalc:79: ERROR: Content block expected for the "seealso" directive; none found.

See the seealso syntax issue mentioned above but at a different location. This is somewhat awkward since to get sage-doc to build one had to patch sage in the sage ebuild to resolve the issue. I see no seealso patch in sage.

[kiwifb]

I haven't updated the ebuild for the new beta. I am waiting for us to cut a 6.2 ebuild before updating it. Upstream has not released updated doc tarballs that we use for stable releases yet. I think there are updates to sphinx and al. in the new beta which may explain some of the stuff you see.

[cschwan]

Oh, I didn'see this when I responded to your mail. If the 9999-r1-style docs are working then I'd vote for using them with 6.2 and higher!

[kiwifb]

I haven't encountered any problems. You are on sphinx 1.2.x?