./configure --disable-doc

[mkoeppe]

This switch will disable the docbuild and avoid installing its many dependencies (sphinx...).

In contrast to other packages, pplpy's docbuild is enabled unless explicitly disabled by setting SAGE_SPKG_INSTALL_DOCS=no. We move it to a separate package pplpy_doc and make it a dependency of sagemath_doc_html.

Split out from #31396.

We also add argon2_cffi and nbclient to the packages disabled upon --disable-notebook.

Depends on #29039

CC: @tobiasdiez @dimpase @jhpalmieri @sagetrac-tmonteil

Component: build: configure

Author: Matthias Koeppe, John Palmieri

Branch/Commit: 43221d0

Reviewer: John Palmieri, Matthias Koeppe

Issue created by migration from https://trac.sagemath.org/ticket/32759

[mkoeppe]

Branch: u/mkoeppe/__configure___disable_doc

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Commit: 18b4055

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

97b9fcf	configure.ac (--disable-doc): Disable also 'requests' and its dependencies, and direct sphinx dependencies
18b4055	build/pkgs/pplpy/dependencies: Depend on sphinx only if SAGE_SPKG_INSTALL_DOCS!=no

[mkoeppe]

Description changed:

--- 
+++ 
@@ -1,4 +1,7 @@
 This switch will disable the docbuild and installation of its many dependencies (sphinx...)
+
+In contrast to other packages, `ppl`'s docbuild is enabled unless explicitly disabled by setting `SAGE_SPKG_INSTALL_DOCS=no`.
+We conditionalize the dependency of `ppl` on this and also on `sphinx` being in the list of installed packages.

 Split out from #31396.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 18b4055 to dfd23fb

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

dfd23fb

build/pkgs/pplpy/dependencies: Fix up SAGE_SPKG_INSTALL_DOCS logic

[mkoeppe]

Author: Matthias Koeppe

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from dfd23fb to 7307c75

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

7307c75

configure.ac (--disabled-notebook): Also disable nbclient

[mkoeppe]

Description changed:

--- 
+++ 
@@ -5,3 +5,5 @@

 Split out from #31396.

+We also add another package to those that are disabled upon `--disable-notebook`.
+

[jhpalmieri]

comment:9

If --disable-doc is given, then should doc be removed from the all-start target, so that make doesn't try to build the docs?

[jhpalmieri]

comment:10

Also, after I used ./configure --disable-doc, packages like alabaster and sphinxcontrib-... were built anyway.

[mkoeppe]

comment:11

Replying to @jhpalmieri:

If --disable-doc is given, then should doc be removed from the all-start target, so that make doesn't try to build the docs?

Yes, that's a good idea.

[jhpalmieri]

comment:12

config.log says (for example)

configure:45112: result: sphinx-4.2.0:                                standard, but disabled using configure option

but maybe the problem is that I ran make instead of make build?

[mkoeppe]

comment:13

Yes, probably the all-start target pulled in the doc dependencies.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 7307c75 to 128de23

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

d13b8ae	configure.ac (--disable-doc): Disable also 'requests' and its dependencies, and direct sphinx dependencies
4121cb0	build/pkgs/pplpy/dependencies: Depend on sphinx only if SAGE_SPKG_INSTALL_DOCS!=no
8ca1189	build/pkgs/pplpy/dependencies: Fix up SAGE_SPKG_INSTALL_DOCS logic
ccf6fb5	configure.ac (--disabled-notebook): Also disable nbclient
8e26780	build/pkgs/sagemath_doc_{html,pdf}: New; delegate to them from targets doc-html etc.
eeb8777	m4/sage_spkg_collect.m4: Determine install tree for python3 via new trees.txt mechanism
c8e19a0	fixup
5a4424b	build/make/Makefile.in (all-sage): Do not include packages installed in SAGE_DOCS
67714b8	Merge #31356
128de23	configure.ac (--disable-doc): Disable the actual docbuild on 'make'

[mkoeppe]

Dependencies: #31356

[mkoeppe]

comment:16

Replying to @mkoeppe:

Replying to @jhpalmieri:

If --disable-doc is given, then should doc be removed from the all-start target, so that make doesn't try to build the docs?

Yes, that's a good idea.

This is implemented now, via #31356

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 128de23 to 8a7027a

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

eb65e03	build/pkgs/sagemath_doc_*/dependencies: Add sage_docbuild
7989d87	src/doc/Makefile: Handle errors from './sage --docbuild --all-documents'
d64fb76	build/make/Makefile.in: Undo reintroduction of SAGE_SKIP_PLOT_DIRECTIVE, SAGE_DOC_MATHJAX settings
6afc0d5	build/pkgs/sagemath_doc_html/spkg-install: Move handling of SAGE_DOC_JSMATH environment variable here from src/bin/sage-env
b2cf40c	build/make/Makefile.in: Undo reintroduction of SAGE_SKIP_PLOT_DIRECTIVE, SAGE_DOC_MATHJAX settings (fixup)
37a03bd	src/doc/en/developer/packaging.rst: Fix up section label
fc89a66	build/pkgs/sagemath_doc_pdf/spkg-install: Set SAGE_DOC_MATHJAX as a workaround
fd298c4	build/pkgs/sagemath_doc_pdf/dependencies: Depend on sagemath_doc_html
b0593ea	Merge #31356
8a7027a	configure.ac (--disable-doc): Disable the actual docbuild on 'make'

[mkoeppe]

comment:18

Rebased on top of updated #31356

[jhpalmieri]

comment:19

Should we expect doctests to pass with ./configure --disable-doc? I get

src/sage/misc/sagedoc.py  # 8 doctests failed
src/sage_docbuild/utils.py  # 5 doctests failed
src/sage_docbuild/__init__.py  # 38 doctests failed
src/sage_docbuild/sphinxbuild.py  # 13 doctests failed
src/sage/docs/conf.py  # 1 doctest failed

which is not surprising, but should they be tagged so that they are only tested when sagemath_doc_html is built?

[mkoeppe]

comment:20

git grep 'dochtml' shows that we already seem to have an optional tag for this kind of thing.

We should probably replace it by # optional - sagemath_doc_html and add the missing ones for the failing doctests.

[jhpalmieri]

comment:21

Right, and then in the top-level Makefile, this line needs to be changed:

TESTALL_FLAGS = --optional=sage,dochtml,optional,external,build

and maybe the last line in this block from src/sage/doctest/control.py:

            if have_git:
                self.files.append(opj(SAGE_SRC, 'sage_setup'))
                self.files.append(opj(SAGE_SRC, 'sage_docbuild'))

Perhaps define have_docs appropriately and have a separate if have_docs block. Either that or tag everything in sage_docbuild.

[jhpalmieri]

comment:22

Something like this?

diff --git a/src/sage/doctest/control.py b/src/sage/doctest/control.py
index 858d8cb567..8dbb693e22 100644
--- a/src/sage/doctest/control.py
+++ b/src/sage/doctest/control.py
@@ -727,13 +727,14 @@ class DocTestController(SageObject):
             Doctesting ...
         """
         opj = os.path.join
-        from sage.env import SAGE_SRC, SAGE_DOC_SRC, SAGE_ROOT, SAGE_ROOT_GIT
+        from sage.env import SAGE_SRC, SAGE_DOC_SRC, SAGE_DOC, SAGE_ROOT, SAGE_ROOT_GIT
         # SAGE_ROOT_GIT can be None on distributions which typically
         # only have the SAGE_LOCAL install tree but not SAGE_ROOT
         if SAGE_ROOT_GIT is not None:
             have_git = os.path.isdir(SAGE_ROOT_GIT)
         else:
             have_git = False
+        have_docs = os.path.isdir(opj(SAGE_DOC, 'html', 'en', 'tutorial'))

         def all_files():
             self.files.append(opj(SAGE_SRC, 'sage'))
@@ -742,7 +743,8 @@ class DocTestController(SageObject):
             # don't make sense to run outside a build environment
             if have_git:
                 self.files.append(opj(SAGE_SRC, 'sage_setup'))
-                self.files.append(opj(SAGE_SRC, 'sage_docbuild'))
+                if have_docs:
+                    self.files.append(opj(SAGE_SRC, 'sage_docbuild'))
             self.files.append(SAGE_DOC_SRC)

         if self.options.all or (self.options.new and not have_git):

[mkoeppe]

comment:23

I think it would make more sense to just test whether opj(SAGE_SRC, 'sage_docbuild') exists.

Same for sage_setup.

I think we can get rid of the have_git logic

[jhpalmieri]

comment:24

By the way, with make distclean && ./configure --disable-doc && make build, the various sphinx packages are still being built.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 8a7027a to b6167ca

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

5d962fa	configure.ac (--disable-notebook): Also disable argon2_cffi
da2f1cc	configure.ac (--disable-doc): New
4f0fee0	configure.ac (--disable-doc): Disable also 'requests' and its dependencies, and direct sphinx dependencies
92d06dc	build/pkgs/pplpy/dependencies: Depend on sphinx only if SAGE_SPKG_INSTALL_DOCS!=no
3b1f105	build/pkgs/pplpy/dependencies: Fix up SAGE_SPKG_INSTALL_DOCS logic
64273d1	configure.ac (--disabled-notebook): Also disable nbclient
b6167ca	configure.ac (--disable-doc): Disable the actual docbuild on 'make'

[mkoeppe]

comment:26

Replying to @mkoeppe:

Rebased on top of updated #31356

... I had lost most of the commits in this rebase.

Fixed.

[jhpalmieri]

comment:27

Replying to @mkoeppe:

I think it would make more sense to just test whether opj(SAGE_SRC, 'sage_docbuild') exists.

Same for sage_setup.

I think we can get rid of the have_git logic

Either that or test whether we can import sage_docbuild. I think the directory will exist (in a source distribution) regardless, but we don't want to test if we haven't at least built sage_docbuild. That seems cleaner than tagging everything in those files optional - sage_docbuild.

[mkoeppe]

comment:28

I think the doctesting logic is that the source tree (in SAGE_ROOT/src) is preferred over installed sources, which are only used for falling back (for example for use in downstream packaging). But I agree, when falling back to installed sources, then importing sage_docbuild and using its __path__ is the right solution

[jhpalmieri]

comment:29

My point is that the typical doctest in sage_docbuild/* starts with from sage_docbuild import ..., so we shouldn't bother to test (as part of make testall etc.) if we can't import sage_docbuild. So I suggest something like:

diff --git a/src/sage/doctest/control.py b/src/sage/doctest/control.py
index 858d8cb567..2bf0a4beb4 100644
--- a/src/sage/doctest/control.py
+++ b/src/sage/doctest/control.py
@@ -740,9 +740,13 @@ class DocTestController(SageObject):
             # Don't run these tests when not in the git repository; they are
             # of interest for building sage, but not for runtime behavior and
             # don't make sense to run outside a build environment
-            if have_git:
+            if os.path.isdir(opj(SAGE_SRC, 'sage_setup')):
                 self.files.append(opj(SAGE_SRC, 'sage_setup'))
+            try:
+                import sage_docbuild
                 self.files.append(opj(SAGE_SRC, 'sage_docbuild'))
+            except ImportError:
+                pass
             self.files.append(SAGE_DOC_SRC)

         if self.options.all or (self.options.new and not have_git):

[mkoeppe]

comment:30

Ah ok, I agree. Same for sage_setup then, I think

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

3556181

git grep -l '#.*optional.*dochtml' | xargs sed -i.bak 's/# optional - dochtml/# optional - sagemath_doc_html/'

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from b6167ca to 3556181

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

4c11aab

sed -i.bak 's/dochtml/sagemath_doc_html/' Makefile src/bin/sage-runtests src/doc/en/developer/doctesting.rst

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 3556181 to 4c11aab

[jhpalmieri]

comment:33

Is it possible that SAGE_DOC_SRC might be missing? We could do this:

diff --git a/src/sage/doctest/control.py b/src/sage/doctest/control.py
index 858d8cb567..86dbd24d1a 100644
--- a/src/sage/doctest/control.py
+++ b/src/sage/doctest/control.py
@@ -737,13 +737,22 @@ class DocTestController(SageObject):

         def all_files():
             self.files.append(opj(SAGE_SRC, 'sage'))
-            # Don't run these tests when not in the git repository; they are
-            # of interest for building sage, but not for runtime behavior and
-            # don't make sense to run outside a build environment
-            if have_git:
+            # Only test sage_setup and sage_docbuild if the relevant
+            # imports work. They may not work if not in a build
+            # environment or if the documentation build has been
+            # disabled.
+            try:
+                import sage_setup
                 self.files.append(opj(SAGE_SRC, 'sage_setup'))
+            except ImportError:
+                pass
+            try:
+                import sage_docbuild
                 self.files.append(opj(SAGE_SRC, 'sage_docbuild'))
-            self.files.append(SAGE_DOC_SRC)
+            except ImportError:
+                pass
+            if os.path.isdir(SAGE_DOC_SRC):
+                self.files.append(SAGE_DOC_SRC)

         if self.options.all or (self.options.new and not have_git):
             self.log("Doctesting entire Sage library.")

---

New commits:

4c11aab

sed -i.bak 's/dochtml/sagemath_doc_html/' Makefile src/bin/sage-runtests src/doc/en/developer/doctesting.rst

[jhpalmieri]

comment:34

Can we autodetect sagemath_doc_html as an optional tag to add for testing?

[mkoeppe]

comment:35

Yes, I think it should be autodetected already because sagemath_doc_html is just an installed package

[mkoeppe]

comment:36

Replying to @jhpalmieri:

Is it possible that SAGE_DOC_SRC might be missing?

Yes, I think so, for example in downstream packaging of sage

[jhpalmieri]

comment:37

Replying to @mkoeppe:

Yes, I think it should be autodetected already because sagemath_doc_html is just an installed package

Does that mean we shouldn't manually add it to TESTALL_FLAGS and let it happen automatically?

[mkoeppe]

comment:38

Yes, that's a good point

[mkoeppe]

comment:39

Not sure how to implement TESTALL_NODOC_FLAGS then

TESTALL_FLAGS = --optional=sage,sagemath_doc_html,optional,external,build
TESTALL_NODOC_FLAGS = --optional=sage,optional,external,build

[mkoeppe]

comment:40

(introduced in #31084)

[jhpalmieri]

comment:41

Deprecate the nodoc targets and tell people to use ./configure --disable-doc instead?

Or leave it with the two FLAGS being identical? Running make ptest-nodoc runs make build and so shouldn't build sagemath_doc_html, so it shouldn't run tests with that flag, while make ptest runs make and so will build sagemath_doc_html.

[mkoeppe]

comment:42

Thanks for the analysis, I like the 2nd option

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

afa2f80

Makefile: Get rid of TESTALL_NODOC_FLAGS