Absolute line number in errors

[amueller]

At scikit-learn we've been struggling a lot with finding and fixing errors in the sphinx built. We're using automethod, autoclass and automodule with numpydoc. A problem we keep running into is that autoclass and automodule don't provide absolute line number or even files for where the error is. Instead, they provide the full docstring of that class or method, together with a line number in that docstring. That means that each error generates an error message about one computer screen long, such as

["KernelPCA(n_components=None, kernel='linear', gamma=None, degree=3, coef0=1, kernel_params=None, alpha=1.0, fit_inverse_transform=False, eigen_solver='auto', tol=0, max_iter=None, remove_zero_eig=False, random_state=None, copy_X=True, n_jobs=1)", ':module: sklearn.decomposition', '', '', '', 'Kernel Principal component analysis (KPCA)', '', 'Non-linear dimensionality reduction through the use of kernels (see', ':ref:metrics).', '', 'Read more in the :ref:User Guide <kernel_PCA>.', '', ':Parameters:', '', ' n_components : int, default=None', '', ' Number of components. If None, all non-zero components are kept.', ' ', '', ' kernel : "linear" | "poly" | "rbf" | "sigmoid" | "cosine" | "precomputed"', '', ' Kernel. Default="linear".', ' ', '', ' degree : int, default=3', '', ' Degree for poly kernels. Ignored by other kernels.', ' ', '', ' gamma : float, optional', '', ' Kernel coefficient for rbf and poly kernels. Default: 1/n_features.', ' Ignored by other kernels.', ' ', '', ' coef0 : float, optional', '', ' Independent term in poly and sigmoid kernels.', ' Ignored by other kernels.', ' ', '', ' kernel_params : mapping of string to any, optional', '', ' Parameters (keyword arguments) and values for kernel passed as', ' callable object. Ignored by other kernels.', ' ', '', ' alpha : int, default=1.0', '', ' Hyperparameter of the ridge regression that learns the', ' inverse transform (when fit_inverse_transform=True).', ' ', '', ' fit_inverse_transform : bool, default=False', '', ' Learn the inverse transform for non-precomputed kernels.', ' (i.e. learn to find the pre-image of a point)', ' ', '', " eigen_solver : string ['auto'|'dense'|'arpack']", '', ' Select eigensolver to use. If n_components is much less than', ' the number of training samples, arpack may be more efficient', ' than the dense eigensolver.', ' ', '', ' tol : float, default=0', '', ' Convergence tolerance for arpack. If zero, optimal value will be', ' chosen by arpack.', ' ', '', ' max_iter : int, default=None', '', ' Maximum number of iterations for arpack. If None, optimal value will', ' be chosen by arpack.', ' ', '', ' remove_zero_eig : boolean, default=False', '', ' If True, then all components with zero eigenvalues are removed, so', ' that the number of components in the output may be < n_components', ' (and sometimes even zero due to numerical instability).', ' When n_components is None, this parameter is ignored and components', ' with zero eigenvalues are removed regardless.', ' ', '', ' random_state : int seed, RandomState instance, or None, default=None', '', ' A pseudo random number generator used for the initialization of the', " residuals when eigen_solver == 'arpack'.", ' ', '', ' n_jobs : int, default=1', '', ' The number of parallel jobs to run.', ' If -1, then the number of jobs is set to the number of CPU cores.', ' ', '', ' copy_X : boolean, default=True', '', ' If True, input X is copied and stored by the model in the X_fit_', ' attribute. If no further changes will be done to X, setting', ' copy_X=False saves memory by storing a reference.', '', ':Attributes:', '', ' lambdas_ : array, (n_components,)', '', ' Eigenvalues of the centered kernel matrix in decreasing order.', ' If n_components and remove_zero_eig are not set,', ' then all values are stored.', ' ', '', ' alphas_ : array, (n_samples, n_components)', '', ' Eigenvectors of the centered kernel matrix. If n_components and', ' remove_zero_eig are not set, then all components are stored.', ' ', '', ' dualcoef : array, (n_samples, n_features)', '', ' Inverse transform matrix. If fit_inverse_transform=False,', ' dualcoef is not present.', ' ', '', ' X_transformedfit : array, (n_samples, n_components)', '', ' Projection of the fitted data on the kernel principal components.', ' ', '', ' Xfit : (n_samples, n_features)', '', ' The data used to fit the model. If copy_X=False, then X_fit_ is', ' a reference.', '', '.. rubric:: References', '', 'Kernel PCA was introduced in:', ' Bernhard Schoelkopf, Alexander J. Smola,', ' and Klaus-Robert Mueller. 1999. Kernel principal', ' component analysis. In Advances in kernel methods,', ' MIT Press, Cambridge, MA, USA 327-352.', '', '.. only:: latex', '', ' ', '', '.. rubric:: Methods', '', '.. autosummary::', '', ' fit', ' fit_transform', ' get_params', ' inverse_transform', ' set_params', ' transform', '.. automethod:: init', '', '.. py:method:: KernelPCA.fit(X, y=None)', ' :module: sklearn.decomposition', '', ' ', ' ', ' Fit the model from data in X.', ' ', ' ', ' :Parameters:', ' ', ' X: array-like, shape (n_samples, n_features) : ', ' ', ' Training vector, where n_samples in the number of samples', ' and n_features is the number of features.', ' ', ' :Returns:', ' ', ' self : object', ' ', ' Returns the instance itself.', '', '.. py:method:: KernelPCA.fit_transform(X, y=None, params)', ' :module: sklearn.decomposition', '', ' ', ' ', ' Fit the model from data in X and transform X.', ' ', ' ', ' :Parameters:', ' ', ' X: array-like, shape (n_samples, n_features) : ', ' ', ' Training vector, where n_samples in the number of samples', ' and n_features is the number of features.', ' ', ' :Returns:', ' ', ' X_new: array-like, shape (n_samples, n_components) : ', ' ', ' ', '', '.. py:method:: KernelPCA.get_params(deep=True)', ' :module: sklearn.decomposition', '', ' ', ' ', ' Get parameters for this estimator.', ' ', ' ', ' :Parameters:', ' ', ' deep: boolean, optional : ', ' ', ' If True, will return the parameters for this estimator and', ' contained subobjects that are estimators.', ' ', ' :Returns:', ' ', ' params : mapping of string to any', ' ', ' Parameter names mapped to their values.', '', '.. py:method:: KernelPCA.inverse_transform(X)', ' :module: sklearn.decomposition', '', ' ', ' ', ' Transform X back to original space.', ' ', ' ', ' :Parameters:', ' ', ' X: array-like, shape (n_samples, n_components) : ', ' ', ' ', ' ', ' :Returns:', ' ', ' X_new: array-like, shape (n_samples, n_features) : ', ' ', ' ', ' ', ' .. rubric:: References', ' ', ' "Learning to Find Pre-Images", G BakIr et al, 2004.', ' ', ' .. only:: latex', ' ', ' ', '', '.. py:method:: KernelPCA.set_params(params)', ' :module: sklearn.decomposition', '', ' ', ' ', ' Set the parameters of this estimator.', ' ', ' The method works on simple estimators as well as on nested objects', ' (such as pipelines). The latter have parameters of the form', " <component>__<parameter> so that it's possible to update each", ' component of a nested object.', ' ', ' :Returns:', ' ', ' self : ', ' ', ' ', '', '.. py:method:: KernelPCA.transform(X)', ' :module: sklearn.decomposition', '', ' ', ' ', ' Transform X.', ' ', ' ', ' :Parameters:', ' ', ' X: array-like, shape (n_samples, n_features) : ', ' ', ' ', ' ', ' :Returns:', ' ', ' X_new: array-like, shape (n_samples, n_components) : ']:104: (ERROR/3) Unknown target name: "dual_coef".

which makes it really hard for us to find and fix the issue. It would be great to get an error that points to the original file where that source came from, and to the line.

Any help would be much appreciated. I'm more than happy to chip in if anyone can tell me how to achieve that.

[tk0miya]

Indeed, this is terrible...

Could you give me small example project to reproduce it? The docs of scikit-learn is too complicated to reproduce the problem to me...

[amueller]

I'll try to come up with something. It's basically anytime you use autodoc. You could use my wordcloud project which is much smaller but I'll come up with a minimal example. I think the issues is how the autodoc document / parsing structure works. I've been using sphinx for a while but I'm new to the code. It looks like the node that has the error has no reference to the python file it came from.

Sent from phone. Please excuse spelling and brevity.

On Sep 7, 2016 12:42 AM, "Takeshi KOMIYA" notifications@github.com wrote:

Indeed, this is terrible...

Could you give me small example project to reproduce it? The docs of scikit-learn is too complicated to reproduce the problem to me...

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/sphinx-doc/sphinx/issues/2940#issuecomment-245173548, or mute the thread https://github.com/notifications/unsubscribe-auth/AAbcFo7PQmj8o-TwW0zda9CY_k4l9ubDks5qnkDEgaJpZM4J2VqG .

[amueller]

I tried to create a minimal example here, and what I'm getting is

/home/andy/checkout/sphinx_autodoc_debug/main.py:docstring of main.my_function:17: ERROR: Unknown target name: "reference".

Which is much nicer. I'll go and try to reproduce what happens with sklearn. What would be even nicer than what is in this example is to have the line in the python file.

[amueller]

I could reproduce with a minimal example. Seems numpydoc is the problem. Still any help would be much appreciated.

[amueller]

I think the problem is here: https://github.com/numpy/numpydoc/blob/master/numpydoc/numpydoc.py#L202

But I have no idea how to fix it :(

[amueller]

Turns out the problem was in numpydoc.