DOC: Build documentation manually instead of using sphinx

[datapythonista]

I wasted several days of my life in the last months trying to understand sphinx code, and seeing how make it perform simple things like changing the depth of the sidebar table of contents, add breadcrumbs, and similar things.

Not sure what went wrong with sphinx development, but its code lacks any reasonable architecture, and the complexity for what we make it do is huge compared to what it should be.

So, given that the only two options I see are give up and have a suboptimal website for pandas, or stop using sphinx, I decided to start working on the latter.

There will surely be some challenges, but the core of what we need is simply:

• Extract information from .rst files (docutil and numpydoc will do all or most of the job in this)
• Render the extracted content to html templates (jinja2 will do that)
• Manage the cross-references among documents (that will need to be implemented, but doesn't seem so much work or so challenging)

I'll work in this as a separate reusable project, that can be imported as a dependency.

CC: @jorisvandenbossche @TomAugspurger

[jorisvandenbossche]

I would be very hesitant to start on such an endeavour.

But I have to admit, I don't really know how many of the features we use are actually already supported by docutils, and how much are add-ons by sphinx.
Eg what about autodoc/autosummary, the ipython directive, math support, special directives (warning, note, versionadded, deprecated, ..), ... ?

Manage the cross-references among documents (that will need to be implemented, but doesn't seem so much work or so challenging)

There is not only the cross-referencing among documents, but also referencing functions, referencing docs of other projects, ..

If it is mainly the final "putting the things together" in a decent structure: I don't know to what extent it is possible to still use sphinx as a kind of preprocessor on the files (to deal with the sphinx-specific constructs), with the idea that it outputs something more general that can be injected in custom templates (just general thought, didn't think this through ..)

[TomAugspurger]

Agreed with Joris. Maintaining something like that doesn't sound fun.

Sphinx's internals are extremely complicated, but we've found their maintainers to be responsive when we report issues.

On Wed, Nov 21, 2018 at 4:29 AM Joris Van den Bossche < notifications@github.com> wrote:

I would be very hesitant to start on such an endeavour.

But I have to admit, I don't really know how many of the features we use are actually already supported by docutils, and how much are add-ons by sphinx. Eg what about autodoc/autosummary, the ipython directive, math support, special directives (warning, note, versionadded, deprecated, ..), ... ?

Manage the cross-references among documents (that will need to be implemented, but doesn't seem so much work or so challenging)

There is not only the cross-referencing among documents, but also referencing functions, referencing docs of other projects, ..

If it is mainly the final "putting the things together" in a decent structure: I don't know to what extent it is possible to still use sphinx as a kind of preprocessor on the files (to deal with the sphinx-specific constructs), with the idea that it outputs something more general that can be injected in custom templates (just general thought, didn't think this through ..)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pandas-dev/pandas/issues/23763#issuecomment-440614457, or mute the thread https://github.com/notifications/unsubscribe-auth/ABQHInpkln0XbqZDry3A9E1uYCcgE15mks5uxSsggaJpZM4Yn6Dr .

[datapythonista]

I also agree with @jorisvandenbossche, and that's why I spent SO many hours trying to make sphinx do what we need. But after around one week wasted only in trying to understand how the sidebar toc tree is built, and still not much idea what is going on... I'm quite convinced that even if there will be challenges in building a sphinx replacement, I'll be saving a lot of time. There are so many things I want to customize, and even something very small is incredible complex with sphinx.

In any case, I'll build this as an independent project, I don't expect anyone else from the pandas team to be involved, and won't propose migrating until our website can be fully generated with it. So, no big deal on trying. :)

In any case, I appreciate the feedback, and will update you if I make progress with this.

[jorisvandenbossche]

Can you give a bit more details on the other things you want to customize that might be difficult with sphinx? (the navigation structure is indeed an important and complicated one)

[TomAugspurger]

FYI, if we're still looking to do a top- and side-navbar layout, https://github.com/dask/dask-sphinx-theme may be helpful. That's the theme behind http://docs.dask.org/en/latest/

On Wed, Nov 21, 2018 at 6:18 AM Joris Van den Bossche < notifications@github.com> wrote:

Can you give a bit more details on the other things you want to customize that might be difficult with sphinx? (the navigation structure is indeed an important and complicated one)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pandas-dev/pandas/issues/23763#issuecomment-440643130, or mute the thread https://github.com/notifications/unsubscribe-auth/ABQHIp05WZYDuRno_WagvCEnMzNkHEiGks5uxUSOgaJpZM4Yn6Dr .

[datapythonista]

@jorisvandenbossche I think there will be plenty of small things we'd like to change in the docs. For example I'd like to add breadcrumbs, or avoid that numpydoc parameters are rendered inside a <blockquote> tag, breaking the formatting.

Will take a look at the dask sidebar, but I think for pandas will render too much information, as it happens now.

[datapythonista]

Closing this, at least for now.