scikit-learn documentation fails to build

[justineuro]

Describe the bug Failed to build documentation for scikit-learn.

To Reproduce Steps to reproduce the behavior:

$ git clone https://github.com/scikit-learn/scikit-learn.git
$ cd scikit-learn
$ cd docs
$ make html SPHINXOPTS="-D language=en"

Expected behavior Documentation in _build/html/stable.

Environment info

• OS: Ubuntu 20.02 (focal)
• Python version: 3.8.5
• Sphinx version: 1.8.5
• Docutils version: 0.16 release
• Jinja2 version: 2.10.1

Additional context sphinx-err-jzzfqf5p.log

Thanks for any help you can extend.

[tk0miya]

It seems the error was raised inside sphinx_gallery extension. So please check it.

  File "/usr/lib/python3/dist-packages/sphinx_gallery/gen_gallery.py", line 220, in generate_gallery_rst
    this_fhindex, this_computation_times = generate_dir_rst(src_dir, target_dir, gallery_conf,
  File "/usr/lib/python3/dist-packages/sphinx_gallery/gen_rst.py", line 458, in generate_dir_rst
    intro, time_elapsed = generate_file_rst(
  File "/usr/lib/python3/dist-packages/sphinx_gallery/gen_rst.py", line 611, in generate_file_rst
    binder_conf = check_binder_conf(gallery_conf.get('binder'))
  File "/usr/lib/python3/dist-packages/sphinx_gallery/binder.py", line 208, in check_binder_conf
    raise ValueError('binder_conf is missing values for: {}'.format(
ValueError: binder_conf is missing values for: ['url']

[justineuro]

Hi!

Updating sphinx-gallery resulted in the build process going further but not to completion. The build process now stops after about 36% of the sources have been read with error:

raise ParseError("%s is not a item name" % text)`.

More details in the attached log file: sphinx-err-t00tr9vy.log

[tk0miya]

Please read the log file if possible. The error was raised inside numpydoc module. It's an extension you installed, right?

  File "/usr/lib/python3/dist-packages/numpydoc/docscrape.py", line 272, in _parse_see_also
    push_item(current_func, rest)
  File "/usr/lib/python3/dist-packages/numpydoc/docscrape.py", line 243, in push_item
    name, role = parse_item_name(name)
  File "/usr/lib/python3/dist-packages/numpydoc/docscrape.py", line 238, in parse_item_name
    raise ParseError("%s is not a item name" % text)

[justineuro]

After updating numpydoc and sphinx, all the sources were read but I now get this error (unfortunately no *.log file is generated):

Traceback (most recent call last):
  File "$HOME/.local/lib/python3.8/site-packages/sphinx/cmd/build.py", line 280, in build_main
    app.build(args.force_all, filenames)
  File "$HOME/.local/lib/python3.8/site-packages/sphinx/application.py", line 352, in build
    self.builder.build_update()
  File "$HOME/.local/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 296, in build_update
    self.build(to_build,
  File "$HOME/.local/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 310, in build
    updated_docnames = set(self.read())
  File "$HOME/.local/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 415, in read
    self._read_parallel(docnames, nproc=self.app.parallel)
  File "$HOME/.local/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 466, in _read_parallel
    tasks.join()
  File "$HOME/.local/lib/python3.8/site-packages/sphinx/util/parallel.py", line 107, in join
    self._join_one()
  File "$HOME/.local/lib/python3.8/site-packages/sphinx/util/parallel.py", line 114, in _join_one
    raise SphinxParallelError(*result)
sphinx.errors.SphinxParallelError: sphinx.errors.ExtensionError: Handler <function mangle_signature at 0x7f6579baadc0> for event 'autodoc-process-signature' threw an exception (exception: :class:`~sklearn.kernel_ridge.KernelRidge` : Kernel ridge regression is not a item name in "Linear least squares with l2 regularization.\n\nMinimizes the objective function::\n\n||y - Xw||^2_2 + alpha * ||w||^2_2\n\nThis model solves a regression model where the loss function is\nthe linear least squares function and regularization is given by\nthe l2-norm. Also known as Ridge Regression or Tikhonov regularization.\nThis estimator has built-in support for multi-variate regression\n(i.e., when y is a 2d-array of shape (n_samples, n_targets)).\n\nRead more in the :ref:`User Guide <ridge_regression>`.\n\nParameters\n----------\nalpha : {float, ndarray of shape (n_targets,)}, default=1.0\n    Regularization strength; must be a positive float. Regularization\n    improves the conditioning of the problem and reduces the variance of\n    the estimates. Larger values specify stronger regularization.\n    Alpha corresponds to ``1 / (2C)`` in other linear models such as\n    :class:`~sklearn.linear_model.LogisticRegression` or\n    :class:`~sklearn.svm.LinearSVC`. If an array is passed, penalties are\n    assumed to be specific to the targets. Hence they must correspond in\n    number.\n\nfit_intercept : bool, default=True\n    Whether to fit the intercept for this model. If set\n    to false, no intercept will be used in calculations\n    (i.e. ``X`` and ``y`` are expected to be centered).\n\nnormalize : bool, default=False\n    This parameter is ignored when ``fit_intercept`` is set to False.\n    If True, the regressors X will be normalized before regression by\n    subtracting the mean and dividing by the l2-norm.\n    If you wish to standardize, please use\n    :class:`~sklearn.preprocessing.StandardScaler` before calling ``fit``\n    on an estimator with ``normalize=False``.\n\ncopy_X : bool, default=True\n    If True, X will be copied; else, it may be overwritten.\n\nmax_iter : int, default=None\n    Maximum number of iterations for conjugate gradient solver.\n    For 'sparse_cg' and 'lsqr' solvers, the default value is determined\n    by scipy.sparse.linalg. For 'sag' solver, the default value is 1000.\n\ntol : float, default=1e-3\n    Precision of the solution.\n\nsolver : {'auto', 'svd', 'cholesky', 'lsqr', 'sparse_cg', 'sag', 'saga'},         default='auto'\n    Solver to use in the computational routines:\n\n    - 'auto' chooses the solver automatically based on the type of data.\n\n    - 'svd' uses a Singular Value Decomposition of X to compute the Ridge\n      coefficients. More stable for singular matrices than 'cholesky'.\n\n    - 'cholesky' uses the standard scipy.linalg.solve function to\n      obtain a closed-form solution.\n\n    - 'sparse_cg' uses the conjugate gradient solver as found in\n      scipy.sparse.linalg.cg. As an iterative algorithm, this solver is\n      more appropriate than 'cholesky' for large-scale data\n      (possibility to set `tol` and `max_iter`).\n\n    - 'lsqr' uses the dedicated regularized least-squares routine\n      scipy.sparse.linalg.lsqr. It is the fastest and uses an iterative\n      procedure.\n\n    - 'sag' uses a Stochastic Average Gradient descent, and 'saga' uses\n      its improved, unbiased version named SAGA. Both methods also use an\n      iterative procedure, and are often faster than other solvers when\n      both n_samples and n_features are large. Note that 'sag' and\n      'saga' fast convergence is only guaranteed on features with\n      approximately the same scale. You can preprocess the data with a\n      scaler from sklearn.preprocessing.\n\n    All last five solvers support both dense and sparse data. However, only\n    'sag' and 'sparse_cg' supports sparse input when `fit_intercept` is\n    True.\n\n    .. versionadded:: 0.17\n       Stochastic Average Gradient descent solver.\n    .. versionadded:: 0.19\n       SAGA solver.\n\nrandom_state : int, RandomState instance, default=None\n    Used when ``solver`` == 'sag' or 'saga' to shuffle the data.\n    See :term:`Glossary <random_state>` for details.\n\n    .. versionadded:: 0.17\n       `random_state` to support Stochastic Average Gradient.\n\nAttributes\n----------\ncoef_ : ndarray of shape (n_features,) or (n_targets, n_features)\n    Weight vector(s).\n\nintercept_ : float or ndarray of shape (n_targets,)\n    Independent term in decision function. Set to 0.0 if\n    ``fit_intercept = False``.\n\nn_iter_ : None or ndarray of shape (n_targets,)\n    Actual number of iterations for each target. Available only for\n    sag and lsqr solvers. Other solvers will return None.\n\n    .. versionadded:: 0.17\n\nSee Also\n--------\nRidgeClassifier : Ridge classifier.\nRidgeCV : Ridge regression with built-in cross validation.\n:class:`~sklearn.kernel_ridge.KernelRidge` : Kernel ridge regression\n    combines ridge regression with the kernel trick.\n\nExamples\n--------\n>>> from sklearn.linear_model import Ridge\n>>> import numpy as np\n>>> n_samples, n_features = 10, 5\n>>> rng = np.random.RandomState(0)\n>>> y = rng.randn(n_samples)\n>>> X = rng.randn(n_samples, n_features)\n>>> clf = Ridge(alpha=1.0)\n>>> clf.fit(X, y)\nRidge()")

Sphinx parallel build error:
sphinx.errors.ExtensionError: Handler <function mangle_signature at 0x7f6579baadc0> for event 'autodoc-process-signature' threw an exception (exception: :class:`~sklearn.kernel_ridge.KernelRidge` : Kernel ridge regression is not a item name in "Linear least squares with l2 regularization.\n\nMinimizes the objective function::\n\n||y - Xw||^2_2 + alpha * ||w||^2_2\n\nThis model solves a regression model where the loss function is\nthe linear least squares function and regularization is given by\nthe l2-norm. Also known as Ridge Regression or Tikhonov regularization.\nThis estimator has built-in support for multi-variate regression\n(i.e., when y is a 2d-array of shape (n_samples, n_targets)).\n\nRead more in the :ref:`User Guide <ridge_regression>`.\n\nParameters\n----------\nalpha : {float, ndarray of shape (n_targets,)}, default=1.0\n    Regularization strength; must be a positive float. Regularization\n    improves the conditioning of the problem and reduces the variance of\n    the estimates. Larger values specify stronger regularization.\n    Alpha corresponds to ``1 / (2C)`` in other linear models such as\n    :class:`~sklearn.linear_model.LogisticRegression` or\n    :class:`~sklearn.svm.LinearSVC`. If an array is passed, penalties are\n    assumed to be specific to the targets. Hence they must correspond in\n    number.\n\nfit_intercept : bool, default=True\n    Whether to fit the intercept for this model. If set\n    to false, no intercept will be used in calculations\n    (i.e. ``X`` and ``y`` are expected to be centered).\n\nnormalize : bool, default=False\n    This parameter is ignored when ``fit_intercept`` is set to False.\n    If True, the regressors X will be normalized before regression by\n    subtracting the mean and dividing by the l2-norm.\n    If you wish to standardize, please use\n    :class:`~sklearn.preprocessing.StandardScaler` before calling ``fit``\n    on an estimator with ``normalize=False``.\n\ncopy_X : bool, default=True\n    If True, X will be copied; else, it may be overwritten.\n\nmax_iter : int, default=None\n    Maximum number of iterations for conjugate gradient solver.\n    For 'sparse_cg' and 'lsqr' solvers, the default value is determined\n    by scipy.sparse.linalg. For 'sag' solver, the default value is 1000.\n\ntol : float, default=1e-3\n    Precision of the solution.\n\nsolver : {'auto', 'svd', 'cholesky', 'lsqr', 'sparse_cg', 'sag', 'saga'},         default='auto'\n    Solver to use in the computational routines:\n\n    - 'auto' chooses the solver automatically based on the type of data.\n\n    - 'svd' uses a Singular Value Decomposition of X to compute the Ridge\n      coefficients. More stable for singular matrices than 'cholesky'.\n\n    - 'cholesky' uses the standard scipy.linalg.solve function to\n      obtain a closed-form solution.\n\n    - 'sparse_cg' uses the conjugate gradient solver as found in\n      scipy.sparse.linalg.cg. As an iterative algorithm, this solver is\n      more appropriate than 'cholesky' for large-scale data\n      (possibility to set `tol` and `max_iter`).\n\n    - 'lsqr' uses the dedicated regularized least-squares routine\n      scipy.sparse.linalg.lsqr. It is the fastest and uses an iterative\n      procedure.\n\n    - 'sag' uses a Stochastic Average Gradient descent, and 'saga' uses\n      its improved, unbiased version named SAGA. Both methods also use an\n      iterative procedure, and are often faster than other solvers when\n      both n_samples and n_features are large. Note that 'sag' and\n      'saga' fast convergence is only guaranteed on features with\n      approximately the same scale. You can preprocess the data with a\n      scaler from sklearn.preprocessing.\n\n    All last five solvers support both dense and sparse data. However, only\n    'sag' and 'sparse_cg' supports sparse input when `fit_intercept` is\n    True.\n\n    .. versionadded:: 0.17\n       Stochastic Average Gradient descent solver.\n    .. versionadded:: 0.19\n       SAGA solver.\n\nrandom_state : int, RandomState instance, default=None\n    Used when ``solver`` == 'sag' or 'saga' to shuffle the data.\n    See :term:`Glossary <random_state>` for details.\n\n    .. versionadded:: 0.17\n       `random_state` to support Stochastic Average Gradient.\n\nAttributes\n----------\ncoef_ : ndarray of shape (n_features,) or (n_targets, n_features)\n    Weight vector(s).\n\nintercept_ : float or ndarray of shape (n_targets,)\n    Independent term in decision function. Set to 0.0 if\n    ``fit_intercept = False``.\n\nn_iter_ : None or ndarray of shape (n_targets,)\n    Actual number of iterations for each target. Available only for\n    sag and lsqr solvers. Other solvers will return None.\n\n    .. versionadded:: 0.17\n\nSee Also\n--------\nRidgeClassifier : Ridge classifier.\nRidgeCV : Ridge regression with built-in cross validation.\n:class:`~sklearn.kernel_ridge.KernelRidge` : Kernel ridge regression\n    combines ridge regression with the kernel trick.\n\nExamples\n--------\n>>> from sklearn.linear_model import Ridge\n>>> import numpy as np\n>>> n_samples, n_features = 10, 5\n>>> rng = np.random.RandomState(0)\n>>> y = rng.randn(n_samples)\n>>> X = rng.randn(n_samples, n_features)\n>>> clf = Ridge(alpha=1.0)\n>>> clf.fit(X, y)\nRidge()")
make: *** [Makefile:48: html] Error 2

[justineuro]

Reverting back to sphinx-1.8.5 (the updated version is sphinx-3.4.3), the building process apparently gets frozen; no *.log file is generated and the last few lines of the final message on the terminal are:

building [mo]: targets for 0 po files that are out of date
building [html]: targets for 946 source files that are out of date
updating environment: 946 added, 0 changed, 0 removed
reading sources... [100%] supervised_learning .. whats_new/v1.0                                                                                                            
waiting for workers...

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/sphinx/cmd/build.py", line 304, in build_main
    app.build(args.force_all, filenames)
  File "/usr/lib/python3/dist-packages/sphinx/application.py", line 341, in build
    self.builder.build_update()
  File "/usr/lib/python3/dist-packages/sphinx/builders/__init__.py", line 345, in build_update
    self.build(to_build,
  File "/usr/lib/python3/dist-packages/sphinx/builders/__init__.py", line 360, in build
    updated_docnames = set(self.read())
  File "/usr/lib/python3/dist-packages/sphinx/builders/__init__.py", line 466, in read
    self._read_parallel(docnames, nproc=self.app.parallel)
  File "/usr/lib/python3/dist-packages/sphinx/builders/__init__.py", line 521, in _read_parallel
    tasks.join()
  File "/usr/lib/python3/dist-packages/sphinx/util/parallel.py", line 114, in join
    self._join_one()
  File "/usr/lib/python3/dist-packages/sphinx/util/parallel.py", line 122, in _join_one
    raise SphinxParallelError(*result)
sphinx.errors.SphinxParallelError: AttributeError: 'bool' object has no attribute 'split'

Sphinx parallel build error:
AttributeError: 'bool' object has no attribute 'split'

[tk0miya]

To investigate what happened, please run Sphinx with non-parallel mode.

sphinx.errors.ExtensionError: Handler <function mangle_signature at 0x7f6579baadc0> for event 'autodoc-process-signature' threw an exception(...)

AFAIK, the mangle_signature handler might be a part of numpydoc.

[justineuro]

Still no success. Running:

sphinx-build -b html -T -d _build/doctrees  -j1  . _build/html/stable

no longer indicates a parallel build error but the following error still occurs:

AttributeError: 'bool' object has no attribute 'split'

A *.log file is attached: sphinx-err-qghnx5pn.log

[tk0miya]

@justineuro Could you try it again with the latest Sphinx? It's an error caused because you're using new configurations in your project. You can't use old Sphinx to your project.

[justineuro]

The build process was successful this time (using sphinx v.3.4.3)!. There was, however, numerous WARNING: messages (4119 of them). Many, many thanks!

[tk0miya]

Hmm.... that's strange to me. I still don't understand why SphinxParallelError was raised https://github.com/sphinx-doc/sphinx/issues/8730#issuecomment-766517157. Anyway, good to hear. Enjoy.

[justineuro]

After updating the packages of my Ubuntu install, numpydoc, sphinx, and git pulling (0.24.XX branch; 0.24.1-7-gfd890fb4a), I no longer get the error when building in parallel mode. Btw, the WARNING: messages were of the form

WARNING: Failed to get a constructor signature for sklearn.base.BaseEstimator: patch_signature() got an unexpected keyword argument 'type_aliases'

I am not sure if this is useful. Again, many thanks!

[justineuro]

I was also able to build the main branch (0.4-25519-g88be2abb7) but there were three errors printed in the end. Please see the attached file for info. scikit-learn-doc-git-main-0.4-25519-g88be2abb7.txt

[tk0miya]

Thank you for sharing. All I can say is that scikit-learn customizes Sphinx via patches and extensions. So it's difficult to support it. I hope their team may support to resolve the error in next time.

[justineuro]

Much obliged!
Wishing you all the very best!