Better error message when Sphinx extension fails at end of build due to renamed output path

[jgarbers]

I've just started using _pydoctor.sphinx_ext.buildapidocs in my documentation project, replacing Sphinx's autodoc. After adding the extension and _pydoctorargs to my conf.py, sphinx-build fails with this error:

Handler <function on_build_finished at 0x1090307c0> for event 'build-finished' threw an exception (exception: [Errno 2] No such file or directory: '/Users/jgarbers3/lab/ev/docs/build/html/api' -> '/Users/jgarbers3/lab/ev/docs/build/html/api.sphinx_files')

My sphinx-build output path is set to "docs/build", and the extension configuration specifies "--html-output={outdir}/api/".

Reviewing the source for the extension, I think I have an idea of why this is failing. At the end of _on_builderinited, the extension renames the output path (here, "docs/build/html/api") to include a _.pydoctortemp suffix. Then, in _on_buildfinished, the code tries to rename the original output path ("docs/build/html/api") to include a _.sphinxfiles suffix. However, that path does not exist, since it was renamed to "docs/build/html/api.pydoctor_temp" at the end of the build init event handler, and the exception shown above triggers.

I've patched the extension to work around the error, and as a newcomer to pydoctor it seems likely that I've missed something. But I don't see any other references to what a _.sphinxfiles directory might be for, so I'm wondering if that rename step might have been left in inadvertently.

I'm using Python 3.11.3, Sphinx 7.0.1 and the pydoctor extension 23.4.1.

Thanks in advance for any guidance anyone can offer!

[tristanlatr]

Hello @jgarbers,

Thanks for the report, I must admit I'm not really familiar with this part of the code. I suspect something is wrong with error handling inside the sphinx extension. This is not the first time I notice the sphinx extensions failing the whole build with No such file or directory error when there are violations or warnings.

@adiroiban would you be kind enough to look into fixing this issue?

@jgarbers Can you provide a minimal reproducer? What does your patch looks like?

Thanks a lot,

[jgarbers]

Thanks @tristanlatr ! I'll try to do a minimal reproducer later today.

My workaround patch just checks for the existence of the output path before trying to rename it, but as far as I can tell the output path will never exist, so this is just a band-aid in lieu of an actual understanding of the problem.

I do have a number of warnings being generated during the pydoctor build process -- lots of docstring formatting errors to fix! -- so that aligns with your observations of failing withNo such file or directory errors when warnings are present. I'm working on eliminating those warnings, and will remove my patch once they're gone to see if there is indeed a relationship there.

I sure appreciate your prompt response - thanks!

[adiroiban]

I see that the code is poorly documented... my bad.

I see that this was done as part of https://github.com/twisted/pydoctor/pull/330

---

If I remember correctly this was an egg and chicken problem.

You want to build the narrative Sphinx docs to reference pydoctor ... but pydoctor is not built yet... and from pydoctor you want to reference normal Sphinx pages... which are not built yet.

So this is why the extension will try to build pydoctor first in "inventory-only" mode.

That is the pydoctor sphinx-inventory is generated, so that sphinx build can refernce API pages... and after that, pydoctor is called again, this time to generate pydoctor HTML files and at this point it has the main sphinx inventory for the narrative pages.

---

Can you do a clean build of pydoctor or sphinx without this extension ?

AFAIK, the extension works... this is what it's used for https://pydoctor.readthedocs.io/en/latest/

[jgarbers]

Thanks for jumping in, @adiroiban ! I'm a bit confused about what you're asking, though.

Can you do a clean build of pydoctor or sphinx without this extension ?

I'm trying to use the extension in my own project, not to build pydoctor or sphinx itself. If I disable the extension, sphinx-build runs successfully on my project with no errors or warnings -- but of course I have no API documentation!

Not sure if this answer helps to address your question.

Can you clarify what the directory with the ".sphinx_files" suffix is for?

Thanks again!

[jgarbers]

Another data point: after eliminating all warnings and errors from the pydoctor build (by cleaning up all of my bad docstrings), the problem I originally reported no longer occurs. The "api.sphinx_files" directory now appears in the build. It contains the HTML rendering of the "index.rst" file I included per the "hack" suggested in the extension documentation to have a link from my Sphinx TOC into the pydoctor-generated HTML.

I have no idea how or why the errors and warnings could have provoked the original problem, but all seems to be well now. Please don't hesitate to close this issue unless you think it's worth any further investigation.

Again thanks all for the code and help!

[adiroiban]

I guess that pydoctor build was failing ... so no files were generated...so the sphinx extension didn't had a valid set of pydoctor files to work with.

My request was to run pydoctor as a standalone tool to generate your API docs. For example to use pydoctor the way it is documented here https://pydoctor.readthedocs.io/en/latest/quickstart.html#example

---

I am closing this

[tristanlatr]

A better error message for this situation would be good.

[tristanlatr]

You want to build the narrative Sphinx docs to reference pydoctor ... but pydoctor is not built yet... and from pydoctor you want to reference normal Sphinx pages... which are not built yet.

I think this statement is simply wrong: until #609 is not fixed, pydoctor simply cannot reference sphinx narrative documentation. So the chicken and egg problem must around something else. (EDIT: It's probably around the creation of the sphinx toctree, the rst file api/index.rts must be rendered as part of the sphinx build but the pydoctor build must be done before the spinx build so sphinx pages can access pydoctor intersphinx, so that's why we're doing this weird renames around generated files)

I guess that pydoctor build was failing ... so no files were generated...so the sphinx extension didn't had a valid set of pydoctor files to work with.

This should be considered as a bug. Pydoctor almost never crashes, it just report a non-zero exit code too often. So the real problem is that the sphinx extension only works for clean pydoctor builds, which is not normal...

@adiroiban

[tristanlatr]

@jgarbers, Were you using Shpinx's option -W/warningiserror ?

[tristanlatr]

@jgarbers, if you have time to provide reproducer for this issue, it would really appreciated. Thanks.

[tristanlatr]

So the real problem is that the sphinx extension only works for clean pydoctor builds, which is not normal...

After some investigation, it seems that it's more complicated than this. I can have a failing pydoctor run (with invalid links and bad epytext) and still build the sphinx documentation properly.

So I really need your reproducer @jgarbers...

[tristanlatr]

@adiroiban your help would be very appreciated trouble shooting this bug.

[adiroiban]

I don't know how to help here.

I think that is best to reproduce this over GitHub action. In this way we can investigate the error.

Or put the code into a public repo and try to reproduce.

the pydoctor sphinx integration was done on the "quick and dirty" side and only for systems like Read The Docs, where you can't call pydoctor as a separate job.

[tristanlatr]

@jgarbers any chances you can share a reproducer?

[jgarbers]

Hello, @tristanlatr , and sorry for the delay in responding. Not sure how far down I can boil the problem, but I'll see what I can do Monday morning. Been a while since I was dealing with this so it may take a bit to get the context back. Fingers crossed!