Make the sphinx extension compatible with manpage writer

[kloczek]

On building my packages I'm using sphinx-build command with -n switch which shows warmings about missing references. These are not critical issues. Here is the output with warnings:

+ /usr/bin/sphinx-build -n -T -b man docs/source build/sphinx/man
Running Sphinx v4.5.0
Initializing Spelling Checker 7.3.2
making output directory... done
loading intersphinx inventory from https://twistedmatrix.com/documents/current/api/objects.inv...
loading intersphinx inventory from https://bw2.github.io/ConfigArgParse/objects.inv...
loading intersphinx inventory from https://docs.python.org/3/objects.inv...
building [mo]: targets for 0 po files that are out of date
building [man]: all manpages
updating environment: [new config] 22 added, 0 changed, 0 removed
reading sources... [100%] transition
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
writing... pydoctor.1 { quickstart codedoc docformat/index docformat/epytext docformat/epytext/epytext_demo docformat/restructuredtext docformat/restructuredtext/restructuredtext_demo docformat/google-numpy docformat/google/google_demo docformat/numpy/numpy_demo sphinx-integration customize help faq transition contrib readme api/index } /home/tkloczko/rpmbuild/BUILD/pydoctor-22.5.0/docs/source/docformat/google-numpy.rst:22: WARNING: py:mod reference target not found: pydoctor.napoleon
/home/tkloczko/rpmbuild/BUILD/pydoctor-22.5.0/docs/source/sphinx-integration.rst:75: WARNING: py:mod reference target not found: pydoctor.sphinx_ext.build_apidocs
/home/tkloczko/rpmbuild/BUILD/pydoctor-22.5.0/docs/source/customize.rst:103: WARNING: py:mod reference target not found: pydoctor.qnmatch
/home/tkloczko/rpmbuild/BUILD/pydoctor-22.5.0/docs/source/customize.rst:111: WARNING: py:class reference target not found: pydoctor.zopeinterface.ZopeInterfaceSystem
/home/tkloczko/rpmbuild/BUILD/pydoctor-22.5.0/docs/source/customize.rst:125: WARNING: py:class reference target not found: pydoctor.templatewriter.TemplateWriter
done
build succeeded, 5 warnings.

[tristanlatr]

Hello @kloczek,

You're missing our inthersphinx URL, please add --inthersphinx=https://pydoctor.readthedocs.io/en/latest/api/objects.inv

[tristanlatr]

Hi @kloczek, sorry I did not understand the issue at first glance. It seems that the pydoctor intersphinx is not added your sphinx app.

We're using tox to test pydoctor, maybe you would have more luck by using it as well? Because right now, it passes inside tox env, with flag -W -n.

[kloczek]

Using tox IMO is wrong because it assumes that documentation will be generated against installed module and it requires install module. Only handful of python modules still is doing that and NONE of the non-python projects.

Nevertheless probably you are using tox because with current source tree it is not possible to use directly sphinx-build. To fix that I'm using patch

--- a/docs/source/conf.py~      2022-03-31 14:37:49.000000000 +0000
+++ b/docs/source/conf.py       2022-05-01 18:55:06.479309978 +0000
@@ -10,9 +10,9 @@
 # add these directories to sys.path here. If the directory is relative to the
 # documentation root, use os.path.abspath to make it absolute, like shown here.
 #
-# import os
-# import sys
-# sys.path.insert(0, os.path.abspath('.'))
+import os
+import sys
+sys.path.insert(0, os.path.abspath('../..'))

 import os
 import subprocess

That patch fixes exactly what is mentioned in the comment above modified lines and it allows skip assembly tox venv. In other words: with that patch it is quarantine that documentation will be generated out of source tree (like in any other project outside python domain) and it allows generate documentation faster (KISS principle).

[tristanlatr]

Hi @kloczek,

I think what your talking about it unrelated to the issue your facing. Pydoctor works by static analysis, sys.path tweaks should not be necessary.

I suspect, because of the -b man argument, the local inventory are not generated. The core of the issue is here: https://github.com/twisted/pydoctor/blob/018971f1deda1af77bf6b07bc445c41c5c95cef2/pydoctor/sphinx_ext/build_apidocs.py#L77

[kloczek]

I think what your talking about it unrelated to the issue your facing. Pydoctor works by static analysis, sys.path tweaks should not be necessary.

That patch it is not pydoctor tweak. It is tweak of the sphinx copy.py file to allow build pydoctor documentation without have pydoctor installed 😋 Just please try that command which is on top of this ticket 😄

[tristanlatr]

That patch it is not pydoctor tweak. It is tweak of the sphinx copy.py file to allow build pydoctor documentation without have pydoctor installed 😋

OK I get it now, but I'm usure if it's the best way to fix the issue. Would it be possible to use PYTHONPATH env variable to fix importing pydoctor without touching the conf.py?

Also, it seems that there is another potential issue: right now, the local objects.inv (which is used to resolve the links to pydoctor.*) is generated only if the builder is html, so it's normal to me that some links can't resolve when building with -b man

See https://github.com/twisted/pydoctor/blob/018971f1deda1af77bf6b07bc445c41c5c95cef2/pydoctor/sphinx_ext/build_apidocs.py#L77

Just please try that command which is on top of this ticket 😄

I'll try it out when I have a moment, yes.

[kloczek]

FYI that teak is even mentioned in template/example sphinx copy.py file https://www.sphinx-doc.org/en/master/usage/configuration.html#example-of-configuration-file

[tristanlatr]

@kloczek, I've tried the following command locally

sphinx-build -n -T -b man docs/source build/man

And of course, if pydoctor is not accessible in PYTHONPATH, it fails with an ImportError. But I don't understand how this has anything to do with the sphinx warning "reference target not found".

When pydoctor (and all docs requirements) are accessible in the path (either installed or the sys.path is teaked), the command fails with the 2 warnings listed up-there.

Again, the reason for those warnings is that the sphinx inventory of pydoctor is not loaded in the sphinx app, because the builder name is 'man' and not 'html'. Currently, the sphinx integration only works for HTML builds, simply because pydoctor only outputs HTML for now.

Hope this helps,

[kloczek]

And of course, if pydoctor is not accessible in PYTHONPATH, it fails with an ImportError. But I don't understand how this has anything to do with the sphinx warning "reference target not found".

And that what fixes suggested patch .. Almost all OSS projects if have possibility generated documentation that documentation is generated out of the content of copy of the cource tree .. not installed stuff. Few exception still remains ONLY in python modules domain.

[tristanlatr]

@kloczek, all I’m saying it that tweaking the sys.path will not make these warnings disappear.

Now, if you like to submit a pull request in which you tweak the sys.path in conf.py such that the docs can be generated without pydoctor being installed, please go ahead and I’ll accept it.

[kloczek]

@kloczek, all I’m saying it that tweaking the sys.path will not make these warnings disappear.

And that what I said as well 😄 That patch fixes building documentation without installing it. In other words this ticket is about two independent issues.

[tristanlatr]

In other words this ticket is about two independent issues.

let’s focus on one independent issue per ticket, this will avoid future confusions like this.

Let’s have this ticket be about Sphinx warnings only. For the sys.path tweak, it’s not a real issue to me, more a matter of environment.

About these warnings: they are present only because your building with the man page builder and not the html builder. Because pydoctor cannot output man page, the extensions does not generate the appropriate objects.inv file.

Right now, pydoctor is no where close to support different kind of outputs, the model is still quite couple with html rendering logic. And I doubt that generating html when the user expects man pages is not helping.

But maybe the extension could generate the objects.inv file no matter the builder in use? This way the links could resolve, but the files wouldn’t actually exist.

What do you think @kloczek ?

[kloczek]

Issu eis that seruptools<>sphins integration is already sheduled as deprecated. With that I've started switching in my build procedures to use straight sphinx-build command so in such case that small patch is necessary 🤔 IMO documentation shoul be always build out of source tree (like it is in any project outside pytthon modules domain).

Warnings it is separated subject.

[tristanlatr]

Whether the documentation have specific requirements is not a problem in my opinion.

Now, I'll close this issue since there is no way to get the man page build work for now and it looks like these warnings are not a real problem.

Please next time do not talk about two independent issue in the same ticket, this will avoid confusion.

[kloczek]

FYI: those refs as links are rendered only in html outpout.

[kloczek]

Gentle ping .. any update? 🤔 Just tested 24.3.3 and I still see nitpick watings

+ /usr/bin/sphinx-build -n -T -b man docs/source build/sphinx/man
Running Sphinx v7.3.7
Initializing Spelling Checker 8.0.0
loading intersphinx inventory from https://tristanlatr.github.io/apidocs/twisted/objects.inv...
loading intersphinx inventory from https://bw2.github.io/ConfigArgParse/objects.inv...
loading intersphinx inventory from https://docs.python.org/3/objects.inv...
building [mo]: targets for 0 po files that are out of date
writing output...
building [man]: all manpages
updating environment: [new config] 22 added, 0 changed, 0 removed
reading sources... [100%] transition
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
writing... python-pydoctor.3 { quickstart codedoc docformat/index docformat/epytext docformat/epytext_demo/index docformat/restructuredtext docformat/restructuredtext_demo/index docformat/google-numpy docformat/google_demo/index docformat/numpy_demo/index sphinx-integration customize help faq transition contrib readme api/index } /home/tkloczko/rpmbuild/BUILD/pydoctor-24.3.3/docs/source/docformat/google-numpy.rst:22: WARNING: py:mod reference target not found: pydoctor.napoleon
/home/tkloczko/rpmbuild/BUILD/pydoctor-24.3.3/docs/source/sphinx-integration.rst:75: WARNING: py:mod reference target not found: pydoctor.sphinx_ext.build_apidocs
/home/tkloczko/rpmbuild/BUILD/pydoctor-24.3.3/docs/source/customize.rst:103: WARNING: py:mod reference target not found: pydoctor.qnmatch
/home/tkloczko/rpmbuild/BUILD/pydoctor-24.3.3/docs/source/customize.rst:111: WARNING: py:class reference target not found: pydoctor.model.System
/home/tkloczko/rpmbuild/BUILD/pydoctor-24.3.3/docs/source/customize.rst:127: WARNING: py:class reference target not found: pydoctor.extensions.ExtRegistrar
/home/tkloczko/rpmbuild/BUILD/pydoctor-24.3.3/docs/source/customize.rst:133: WARNING: py:class reference target not found: pydoctor.model.Documentable
/home/tkloczko/rpmbuild/BUILD/pydoctor-24.3.3/docs/source/customize.rst:136: WARNING: py:mod reference target not found: pydoctor.extensions.zopeinterface
/home/tkloczko/rpmbuild/BUILD/pydoctor-24.3.3/docs/source/customize.rst:136: WARNING: py:mod reference target not found: pydoctor.extensions.deprecate
/home/tkloczko/rpmbuild/BUILD/pydoctor-24.3.3/docs/source/customize.rst:212: WARNING: py:class reference target not found: pydoctor.model.System
/home/tkloczko/rpmbuild/BUILD/pydoctor-24.3.3/docs/source/customize.rst:212: WARNING: py:meth reference target not found: pydoctor.model.System.__init__
/home/tkloczko/rpmbuild/BUILD/pydoctor-24.3.3/docs/source/customize.rst:212: WARNING: py:class reference target not found: pydoctor.sphinx.SphinxInventory
/home/tkloczko/rpmbuild/BUILD/pydoctor-24.3.3/docs/source/customize.rst:218: WARNING: py:mod reference target not found: pydoctor.extensions
/home/tkloczko/rpmbuild/BUILD/pydoctor-24.3.3/docs/source/customize.rst:224: WARNING: py:class reference target not found: pydoctor.templatewriter.TemplateWriter
/home/tkloczko/rpmbuild/BUILD/pydoctor-24.3.3/docs/source/customize.rst:224: WARNING: py:class reference target not found: pydoctor.templatewriter.IWriter
/home/tkloczko/rpmbuild/BUILD/pydoctor-24.3.3/docs/source/customize.rst:229: WARNING: py:class reference target not found: pydoctor.templatewriter.IWriter
done
build succeeded, 15 warnings.

[tristanlatr]

Hello @kloczek,

Looks like you don't have the pydoctor inventory in your sphinx app, this is because our sphinx extension is only compatible with the html writer. Not the man page writer. See my last comment

The issue should be "Make the sphinx extension compatible with manpage writer", until this, these warnings are normal and should be ignored.

[kloczek]

Man page is perfectly fine and nothing is missing for that. Nitpick warnings are affecting html, qch and pdf output. In rendered document exact keywords are only highlighted without links.

Please use -n option to see the same warnings.

[tristanlatr]

Looks like you’re using -b man which uses the man page writer. Our extension (responsible to generate the intershpinx links before Sphinx is run) only generate it if the writer name is html (see the code that I linked in may 2022). I believe this is why the manpage writer fails to link these names.

Now you say that the issue is present for the html build.. Can you show me the logs of the html build? Not the man page build please.

[kloczek]

Again .. please try to use -n with html and you will see THE SAME warnings.

[tristanlatr]

Hello @kloczek this is what I did and here is the result (absolute path replaced by /.../ ):

sphinx-build -n -T -b html docs/source build/docs
Running Sphinx v7.3.7
Initializing Spelling Checker 8.0.0
loading pickled environment... failed: '_options'
done
Building 'main' pydoctor API docs as:
--make-html
--quiet
--html-output=/.../pydoctor/build/docs/api/
--project-name=pydoctor
--project-version=24.3.3.dev0
--docformat=epytext
--privacy=HIDDEN:pydoctor.test
--project-url=../index.html
/.../pydoctor/pydoctor
--html-viewsource-base=https://github.com/twisted/pydoctor/tree/24.3.x
--project-base-dir=/Volumes/SDCARD/Mac/Tristan/Documents/Git-repos/pydoctor
--config=/.../pydoctor/setup.cfg
Building 'custom_template_demo' pydoctor API docs as:
--make-html
--quiet
--html-output=/.../pydoctor/build/docs/custom_template_demo/
--project-version=24.3.3.dev0
--template-dir=/.../pydoctor/docs/sample_template
/.../pydoctor/pydoctor
--html-viewsource-base=https://github.com/twisted/pydoctor/tree/24.3.x
--project-base-dir=/Volumes/SDCARD/Mac/Tristan/Documents/Git-repos/pydoctor
--config=/.../pydoctor/setup.cfg
--config=/.../pydoctor/docs/source/custom_template_demo/pyproject.toml
-qqq
Building 'epydoc_demo' pydoctor API docs as:
--make-html
--quiet
--html-output=/.../pydoctor/build/docs/docformat/epytext_demo
--project-name=pydoctor-epytext-demo
--project-version=1.3.0
--docformat=epytext
--sidebar-toc-depth=3
--project-url=../epytext.html
--theme=readthedocs
/.../pydoctor/docs/epytext_demo
--html-viewsource-base=https://github.com/twisted/pydoctor/tree/24.3.x
--project-base-dir=/Volumes/SDCARD/Mac/Tristan/Documents/Git-repos/pydoctor
--config=/.../pydoctor/setup.cfg
Building 'restructuredtext_demo' pydoctor API docs as:
--make-html
--quiet
--html-output=/.../pydoctor/build/docs/docformat/restructuredtext_demo
--project-name=pydoctor-restructuredtext-demo
--project-version=1.0.0
--docformat=restructuredtext
--sidebar-toc-depth=3
--project-url=../restructuredtext.html
--process-types
/.../pydoctor/docs/restructuredtext_demo
--html-viewsource-base=https://github.com/twisted/pydoctor/tree/24.3.x
--project-base-dir=/Volumes/SDCARD/Mac/Tristan/Documents/Git-repos/pydoctor
--config=/.../pydoctor/setup.cfg
Building 'numpy_demo' pydoctor API docs as:
--make-html
--quiet
--html-output=/.../pydoctor/build/docs/docformat/numpy_demo
--project-name=pydoctor-numpy-style-demo
--project-version=1.0.0
--project-url=../google-numpy.html
--theme=readthedocs
/.../pydoctor/docs/numpy_demo
/.../pydoctor/pydoctor/napoleon
--html-viewsource-base=https://github.com/twisted/pydoctor/tree/24.3.x
--project-base-dir=/Volumes/SDCARD/Mac/Tristan/Documents/Git-repos/pydoctor
--config=/.../pydoctor/setup.cfg
Building 'google_demo' pydoctor API docs as:
--make-html
--quiet
--html-output=/.../pydoctor/build/docs/docformat/google_demo
--project-name=pydoctor-google-style-demo
--project-version=1.0.0
--docformat=google
--project-url=../google-numpy.html
--theme=readthedocs
/.../pydoctor/docs/google_demo
--html-viewsource-base=https://github.com/twisted/pydoctor/tree/24.3.x
--project-base-dir=/Volumes/SDCARD/Mac/Tristan/Documents/Git-repos/pydoctor
--config=/.../pydoctor/setup.cfg
loading intersphinx inventory from https://tristanlatr.github.io/apidocs/twisted/objects.inv...
loading intersphinx inventory from https://bw2.github.io/ConfigArgParse/objects.inv...
loading intersphinx inventory from https://docs.python.org/3/objects.inv...
loading intersphinx inventory from /.../pydoctor/build/docs/api.pydoctor_temp/objects.inv...
loading intersphinx inventory from /.../pydoctor/build/docs/docformat/epytext_demo.pydoctor_temp/objects.inv...
loading intersphinx inventory from /.../pydoctor/build/docs/docformat/restructuredtext_demo.pydoctor_temp/objects.inv...
loading intersphinx inventory from /.../pydoctor/build/docs/docformat/numpy_demo.pydoctor_temp/objects.inv...
loading intersphinx inventory from /.../pydoctor/build/docs/docformat/google_demo.pydoctor_temp/objects.inv...
building [mo]: targets for 0 po files that are out of date
writing output... 
building [html]: targets for 22 source files that are out of date
updating environment: [new config] 22 added, 0 changed, 0 removed
reading sources... [100%] transition
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
copying assets... copying static files... done
copying extra files... done
done
writing output... [100%] transition
generating indices... genindex done
writing additional pages... search done
dumping search index in English (code: en)... done
dumping object inventory... done
build succeeded.

Note the sphinx inventories loading from files, this is because I used the html writer that triggers our sphinx extension and generate the shpinx inventories before writing output.

[tristanlatr]

That patch it is not pydoctor tweak. It is tweak of the sphinx copy.py file to allow build pydoctor documentation without have pydoctor installed 😋

I get it now: what your trying to do is simply not possible since the pydoctor documentation (generated by Sphinx html build) depends on pydoctor itself.

edit: But with the patch of sys.path it might do the trick with the html build only, not the man pages build.

[kloczek]

edit: But with the patch of sys.path it might do the trick with the html build only, not the man pages build.

FYI I'm using such patch

--- a/docs/source/conf.py
+++ b/docs/source/conf.py
@@ -10,9 +10,9 @@
 # add these directories to sys.path here. If the directory is relative to the
 # documentation root, use os.path.abspath to make it absolute, like shown here.
 #
-# import os
-# import sys
-# sys.path.insert(0, os.path.abspath('.'))
+import os
+import sys
+sys.path.insert(0, os.path.abspath("../.."))

 import os
 import subprocess

Do you want above as PR? 🤔

[tristanlatr]

Maybe the sys.path tweak can be done if the pydoctor import fails with ImportError, so we’re not patching when pydoctor is already installed. But otherwise looks like a useful patch.