Read the Docs Addons enabled by default and injected Sphinx context removal

[humitos]

Hi all 👋🏼. I want to communicate we are deprecating auto-injected Sphinx context and enabling Read the Docs Addons by default on October 7th, 2024. Read the public announcement at our blog post Read the Docs Addons enabled by default.

I'm opening this issue here as a way to connect with other Sphinx theme authors and coordinate the required changes on those themes before reaching the deprecation date and make the transition as smooth as possible:

• @jbms & @2bndy5 (jbms/sphinx-immaterial)
• @agoose77 & @choldgraf (executablebooks/sphinx-book-theme)
• @pradyunsg (pradyunsg/furo)
• @drammock (pydata/pydata-sphinx-theme)
• @mgeier (mgeier/insipid-sphinx-theme)

Feel free to ping other theme authors you may know here so they are aware of the upcoming changes as well.

Summary of the changes

Read the Docs will stop installing readthedocs-sphinx-ext Python package by default and stop appending extra configurations to the Sphinx's conf.py when building projects.

We've created projects on Read the Docs to show the Sphinx configuration for the old and new behavior. This allow us to understand what are the differences (search for html_context on each page, as an example):

• Page with current and deprecated behavior: https://test-builds.readthedocs.io/en/latest/sphinx-context.html
• Page with new default behavior: https://test-builds-disable-sphinx-manipulation.readthedocs.io/en/latest/sphinx-context.html

Details of the changes

When building on Read the Docs, there are some changes performed automatically via a Sphinx configuration file by conf.py.tmpl:

• There are some workarounds to support old versions of Sphinx
• Injects plenty of variables in the html_context
• Adds _static directory to html_static_path if it exists
• Sets alabaster theme if there is no theme defined
• Set html_baseurl as canonical_url if Sphinx >=1.8
• Install readthedocs_ext.readthedocs as a Sphinx extension
• Install readthedocs_ext.external_version_warning if the Read the Docs version is external
• Define some LaTeX configurations based on project's language

Template variables available via html_context

If you are depending on any of this variables injected into the HTML context, you will need to perform some changes on your theme. Note there are some environment variables that Read the Docs passes to the build process that you may be able to use for this purpose. As an example, note that html_context["commit"] won't injected anymore, but you can use the environment variable READTHEDOCS_GIT_COMMIT_HASH instead to get this value.

Read the Docs embeded flyout replaced by new addons

We are moving away from the HTML blob injection we are currently doing to generate the flyout into a #readthedocs-embed-flyout selector. We already implemented Read the Docs Addons as a replacement to this approach which is modular and customizable.

The removal of the HTML blob injection means that if the theme were using an explicit selector to decide where to inject the flyout menu, it has to be migrated to listen to the readthedocs-addons-data-ready JavaScript event to get all the Read the Docs data in a JSON-like[^1] object and generate the flyout as you prefer. Note that you can style and render the flyout as you prefer --even splitting it into different selectors, placement or integrate it more with your theme.

Example integrations using the JavaScript custom event:

• Read the Docs Sphinx theme kept the exact look&feel than before.
• CPython documentation doesn't render the flyout as-is but split it into version and language selectors at the top-left of the page.

You are free to integrate this data as you prefer, the only requirement here is to mention the documentation is hosted by Read the Docs somewhere in the page 😄

EthicalAds

If your theme is using a specific placement for the EthicalAd as described in our documentation via #ethical-ad-placement, it should keep working properly.

If that's not the case, Read the Docs will try to place the ad in the best placement possible based on some heuristic.

---

Hopefully, I was able to communicate this clearly enough 😅 . I'm happy to keep this issue open or open one issue per theme, depending on your needs, and I'm happy to collaborate with you all with this migration to the new Read the Docs Addons. Let me know if you have any doubt or if you want me to help you with anything related to this work 👍🏼

[^1]: Unfortunately, not yet documented 😓, but you can find an example of it at https://docs.readthedocs.io/_/addons/?client-version=0.17.0&api-version=1&project-slug=docs&version-slug=stable

[2bndy5]

I'm glad to see this getting finalized. I've been periodically checking on this progress, but I was hesitant to implement anything without a definitive date. 👍🏼

Is there a plan to document the "JSON-like" data? (possibly with a schema?)

[humitos]

Is there a plan to document the "JSON-like" data? (possibly with a schema?)

I have not started documenting this yet. However, we've implemented a way for theme author to pin to a particular version of the JSON-like data, so it won't change over time for that theme even if we keep updating this object for any reason. To pin the version we require the theme to define it via:

<meta name="readthedocs-addons-api-version" content="1">

I don't have too much experience with schemas, but I know that @stsewd have done something similar for our readthedocs.yaml file. So, I think we would follow something similar at some point --but we will definitely start with the documentation first.

[humitos]

@2bndy5 oh, by the way, there is a Material for MkDocs example I built that uses the new addons integration to build the version selector at the top (as Mike plugin does) that you can find at https://test-builds.readthedocs.io/en/mkdocs-material/ to check out how it looks in case you want to follow something similar.

[2bndy5]

I noticed that. I'm more inclined to replace/augment the repo info both in header bar and fly-out hamburger menu (on mobile). My last attempt didn't go so well (using old approach).

[daltzctr]

What exactly is the purpose behind readthedocs_ext.readthedocs and should users start adding this dependency to their requirements and conf.py?

What exactly does the flyout injection? The extension or readthedocs addons?

Does this back-break older builds of documentation?

[humitos]

What exactly is the purpose behind readthedocs_ext.readthedocs and should users start adding this dependency to their requirements and conf.py?

The purpose of that extension was "to give Read the Docs a way to execute code at build time inside the Sphinx process to integrate better with the platform". This is not required anymore and users should not install this extension.

What exactly does the flyout injection? The extension or readthedocs addons?

Since we moved away from installing that extension and we don't have control on the Sphinx build process anymore, all the integration was migrated to JavaScript and happens on the client side now. The JavaScript file that contains all the addons is injected via a Cloudflare Worker when serving the documentation.

BTW, the repository for the addons is at https://github.com/readthedocs/addons, in case you want to chime in.

One of the goals for this was to make the platform being documentation agnostic. Now, Read the Docs supports any documentation tool and all of them have the same integrations in place 🎉

Does this back-break older builds of documentation?

It shouldn't, but if it does, please let us know and we will try to fix it.

You can test this integration by enabling the addons in your project and then hitting an old documentation version.

[humitos]

Hello everybody 👋🏼 -- I wanted to give an update regarding the work we've done in our Sphinx theme. We've integrated the version and language selectors into the navigation bar itself: below the documentation's title and above the search input.

You can see this live example at https://test-builds.readthedocs.io/en/latest/

We've included this feature in sphinx-rtd-theme=3.0.0rc2. Please, give it a try and let us know any feedback you may have. We plan to do the final 3.0.0 release on October 7th together with addons and Sphinx context changes.

If you are a theme developer and want to follow this approach, you can take a look at this PR https://github.com/readthedocs/sphinx_rtd_theme/pull/1601 to use it as inspiration. Note that it uses Read the Docs Addons' CustomEvent, so your project has to have addons enabled to work.

[2bndy5]

Will that become part of stable channel? I assume this question was discussed in a separate thread. The nice thing about having the flyout menu incorporated into the nav menu was that the floating menu didn't obstruct the page's main body, but in 3.0.0-rc2 there's now a floating menu:

[humitos]

Let me clarify. There are different concepts here;

1. Read the Docs Addons flyout: it's shown floating at the bottom right. It's injected in all the pages served by Read the Docs and it's independent from the documentation tool used. It can be disabled from the project's settings page. It's not a Sphinx-specific feature.
2. Sphinx Read the Docs theme's re-implementation of the flyout: it's shown integrated in the navbar at the bottom left. It requires defining flyout_display="attached" in the Sphinx's conf.py. It's independent from the "Read the Docs Addons flyout" mentioned before as they could live together. It's disabled by default.
3. Sphinx Read the Docs theme's version/language selectors: they are integrated in the navbar and shown at the top left. It's independent from Read the Docs Addons flyout and also from the Sphinx theme's re-implementation. They are enabled by default and can be disabled by declaring version_selector=False and language_selector=False

With all of that, you can pick the combination that meets your needs best. If the Read the Docs Addons flyout covers some important content from your documentation project, you can disable it.

@2bndy5 Let me know if this clarifies what is what and how to interact with each of them.

[2bndy5]

It requires defining flyout_display="attached" in the Sphinx's conf.py

Is this supposed to go in the html_theme_options? I haven't been keeping up with python stuff lately (currently enthralled in rust). Is there a docs link (about accepted values and the default if not set)?

I wasn't aware of that complexity this feature introduces.

[humitos]

Is this supposed to go in the html_theme_options? Is there a docs link (about accepted values and the default if not set)?

Yes, it goes under html_theme_options. You can find the docs at https://sphinx-rtd-theme.readthedocs.io/en/latest/configuring.html

[2bndy5]

Thanks, that clarifies it for me indeed.

[mgeier]

Thanks for the heads-up!

1. Read the Docs Addons flyout: it's shown floating at the bottom right. It's injected in all the pages served by Read the Docs and it's independent from the documentation tool used. It can be disabled from the project's settings page. It's not a Sphinx-specific feature.

I'm confused ...

So there is no way for a Sphinx theme to move the flyout from the bottom right to somewhere else?

The only thing we can do is to create an additional place in our theme to show that information and to tell the users of our theme to please disable the bottom-right one in their project settings?

This seems very cumbersome ...

[r0qs]

Hi, I’ve read through this issue and the blog post about enabling RTD addons by default, but I’m still a bit unclear on one point. Will it still be possible to disable the addons via the RTD dashboard?

From my understanding, making the addons enabled by default, as mentioned in the blog post and this issue, does not necessarily imply that we won’t have the option to disable them. However, I came across this recent change (https://github.com/readthedocs/readthedocs.org/pull/11628), which completely removes the ability to disable addons. I’m curious about the reasoning behind this change, as the announcement only stated that addons would be enabled by default, without indicating that opting out would not be possible. Did I perhaps miss something?

Read the Docs embeded flyout replaced by new addons

We are moving away from the HTML blob injection we are currently doing to generate the flyout into a #readthedocs-embed-flyout selector. We already implemented Read the Docs Addons as a replacement to this approach which is modular and customizable.

The removal of the HTML blob injection means that if the theme were using an explicit selector to decide where to inject the flyout menu, it has to be migrated to listen to the readthedocs-addons-data-ready JavaScript event to get all the Read the Docs data in a JSON-like1 object and generate the flyout as you prefer. Note that you can style and render the flyout as you prefer --even splitting it into different selectors, placement or integrate it more with your theme.

Example integrations using the JavaScript custom event:

* [Read the Docs Sphinx theme](https://github.com/readthedocs/sphinx_rtd_theme/pull/1526) kept the exact look&feel than before.

* [CPython documentation](https://github.com/python/cpython/pull/116966) doesn't render the flyout as-is but split it into version and language selectors at the top-left of the page.

The reason I’m concerned is that our project has some custom DOM manipulations to adjust the placement of the flyout menu using CSS selectors, which I recently - and unfortunately - discovered in our code: https://github.com/ethereum/solidity/blob/develop/docs/_static/js/initialize.js#L25.

For newer versions of our documentation, I’ve implemented a workaround based on https://github.com/readthedocs/sphinx_rtd_theme/pull/1526, but it doesn’t solve the issue for our older documentation versions, which depended on the .rst-versions element that was removed along with the embedded flyout. Thus, enabling addons breaks the layout of all released documentation.

So, my question is: what would be the recommended approach to address this issue for old documentation versions that relied on such HTML blob injection? For now, the only solution I see is to avoid enabling the addons, but it seems that this will also not be possible. Do you have any alternative suggestions?

[2bndy5]

It would be nice if certain add-ons could be disabled per RTD build. Currently, it only seems to have a global setting.