rendering wrong in docutils 0.17 (unordered list, maybe more?)

[rkdarst]

I am giving this issue a broad title to collect other issues.

Problem

With docutils 0.17, it seems that bullet points don't render properly (the marker is missing), spacing too little (see images below)

An immediate fix is already in sphinx_rtd_theme==0.5.2 (just released), see (#1111, #1112 for issues, #1114 for fix), but that seems to only pin docutils==0.16 so other things that broke should still be tracked. I tested using the sphinx_rtd_theme docs themselves. #1112 shows this with a problem in the sidebar.

With 0.16:

With 0.17:

I originally thought this was another Sphinx problem, but it seems to not be present in the default Sphinx theme, but (only/at least) sphinx_rtd_theme.

Reproducible Project

sphinx_rtd_theme with docutils==0.16. Installed from the master branch pip install docutils=0.17 immediately after.

git clone https://github.com/readthedocs/sphinx_rtd_theme
cd sphinx_rtd_theme
git checkout ca2719b
pip install -e . -r docs/requirements.txt docutils==0.17
(cd docs ; make clean html)
# open build/html/changelog.html, does not work
pip install docutils==0.16
# refresh page, works

Error Logs/Results

none I see as relevant

Expected Results

...

Environment Info

• Python Version: Python 3.7.3
• Sphinx Version: Sphinx 3.5.3
• RTD Theme Version: ca2719b

Package                       Version   Location                          
----------------------------- --------- ----------------------------------
alabaster                     0.7.12    
Babel                         2.9.0     
certifi                       2020.12.5 
chardet                       4.0.0     
docutils                      0.16      
idna                          2.10      
imagesize                     1.2.0     
Jinja2                        2.11.3    
MarkupSafe                    1.1.1     
packaging                     20.9      
pip                           18.1      
pkg-resources                 0.0.0     
Pygments                      2.8.1     
pyparsing                     2.4.7     
pytz                          2021.1    
requests                      2.25.1    
setuptools                    40.8.0    
six                           1.15.0    
snowballstemmer               2.1.0     
Sphinx                        3.5.3     
sphinx-rtd-theme              0.5.2     /home/$USER/git/sphinx_rtd_theme
sphinxcontrib-applehelp       1.0.2     
sphinxcontrib-devhelp         1.0.2     
sphinxcontrib-htmlhelp        1.0.3     
sphinxcontrib-httpdomain      1.7.0     
sphinxcontrib-jsmath          1.0.1     
sphinxcontrib-qthelp          1.0.3     
sphinxcontrib-serializinghtml 1.1.4     
urllib3                       1.26.4    

See also

• Docutils 0.17 changelog: https://docutils.sourceforge.io/RELEASE-NOTES.html#id46 (lists changes in outputs)
• https://github.com/sphinx-doc/sphinx/issues/9061 (sidebar again, looks like sphinx_rtd_theme in that example). I first reported this issue there but it could'nt be reproduced, seems to be sphinx_rtd_theme but not the default theme
• Plus some more PRs around sphinx-doc/sphinx

[Blendify]

In the 0.17 changelog, I do not see any mention of changes to unordered list. Perhaps this is a bug in sphinx?

[bugwelle]

This can also be seen in this project's docs as well:

• https://sphinx-rtd-theme.readthedocs.io/en/0.5.1/changelog.html
• https://sphinx-rtd-theme.readthedocs.io/en/latest/changelog.html

0.5.1	latest

By hard-coding docutils to v0.16, everything is back to normal:

pip3 install sphinx sphinx-rtd-theme docutils==0.16 

Edit:// My bad, the first post also mentions the changelog as an example 🤦

[rkdarst]

Yes, that's an example of the problem.

When I was first trying to debug it I downgraded Sphinx, but it didn't help. I also tried the default Sphinx theme and unordered lists were OK. Of course, it could still be a bug whose root is somewhere else, but is only visible here...

Unfortunately I haven't had time to go deeper, but now everyone has an example to look at.

[stsewd]

there was an error in the requirements https://github.com/readthedocs/sphinx_rtd_theme/pull/1118, docs from the theme should be good now https://sphinx-rtd-theme.readthedocs.io/en/latest/changelog.html

[Blendify]

I still want to address docutils 0.17 and sphinx4 support for the upcoming 1.0 release.

[1313e]

Names of a table of contents are now also rendered incorrectly in the contents menu on the left it seems. This can be seen in my last render of my CMasher docs: https://cmasher.readthedocs.io/index.html The table of contents names are now in white and left-aligned instead of blue and center-aligned.

[stsewd]

@1313e you need to set the theme version explicitly to 0.5.2 https://github.com/1313e/CMasher/blob/master/docs/requirements_docs.txt, the default version may not be the latest see https://docs.readthedocs.io/en/stable/guides/reproducible-builds.html#don-t-rely-on-implicit-dependencies

[1313e]

Alright, will fix that. However, docutils v0.17 will still cause the issue I mentioned.

[Blendify]

The issue with docutils 0.17 is the change from <div class="section"> to <section>. Our css selector for lists is .section ol we will need to update our css to fix this.

I will have a fix out soon.

[Blendify]

Looked into this and we need to fix our css for the new semantic tags <main>, <section>, <header>, <footer>, <aside>, <figure>, and <figcaption>.

[armando-fandango]

Looked into this and we need to fix our css for the new semantic tags <main>, <section>, <header>, <footer>, <aside>, <figure>, and <figcaption>.

Is this fixed and new version of RTD theme available please ?

[klieret]

Any news on this? This is quite a deal breaker for the theme unfortunately :pensive:

[Blendify]

I opened a PR: #1185 to fix this issue

[paulgessinger]

This seems to be solved for everyone by updating sphinx_rtd_theme. I still have this issue with sphinx 5.0.2, docutils 0.17.1 and sphinx_rtd_theme 1.0.0. Downgrading docutils to 0.16.0 fixes the bullet points.

[ThiefMaster]

Same for me - latest of everything does NOT fix this...

[khider]

Same problem and downgrading docutils gives me an error message:

looking for now-outdated files... none found pickling environment... done checking consistency... done preparing documents... done writing output... [ 50%] index
Exception occurred: File "/Users/deborahkhider/opt/anaconda3/envs/paleodev/lib/python3.9/site-packages/docutils/nodes.py", line 439, in copy return self.class(reprunicode(self), rawsource=self.rawsource) AttributeError: 'Text' object has no attribute 'rawsource' The full traceback has been saved in /var/folders/xj/p7h9764x7cx0by8547l04rrr0000gn/T/sphinx-err-tjn_5u89.log, if you want to report the issue to the developers. Please also report this if it was a user error, so that a better error message can be provided next time. A bug report can be filed in the tracker at https://github.com/sphinx-doc/sphinx/issues. Thanks!

[bskinn]

@paulgessinger, @ThiefMaster, @khider -- Are you building on RtD?

If so, beware that if your project was created before a certain date (20 Oct 2020 for personal, 21 Jan 2021 for business), RtD applies some default upper-bound constraints on Sphinx and sphinx-rtd-theme that you have to overcome with a lower-bound pin in your requirements file. More details at this SO answer.

[jdegenstein]

FYI we attempted to use the lower-bound pin described in the most recent comment / SO post but it was not enough to fix the bullet rendering problem. This is what worked for us in our project (build123d):

sphinx_rtd_theme>=0.5.1
docutils<0.17

[bskinn]

Do you have an explicit Sphinx constraint/pin in build123d, @jdegenstein?

I'm only able to avoid the docutils<0.17 constraint because I've constrained/pinned both Sphinx and sphinx_rtd_theme.

[jdegenstein]

We do have sphinx==5.3.0 pinned as well here. For some reason readthedocs builds are ignoring the requirements for sphinx_rtd_theme.

Successfully installed ... docutils-0.19 sphinx-6.1.1 sphinx-rtd-theme-0.5.1 ...

(note that sphinx=6.1.1 is later replace with 5.3.0 in the next pip command) https://readthedocs.org/projects/build123d/builds/19095422/

Is there a problem with the pip package? Just a blind idea.

[jdegenstein]

To be clear, everything is now working fine for me. I am trying to inform the devs of this project that there might (emphasis on might) be a problem with the pip package.

Why does this command (from RTD build linked above) below install docutils-0.19 sphinx-6.1.1 sphinx-rtd-theme-0.5.1 ? It should not do that as far as I can tell because sphinx-rtd-theme (and its pip package) SHOULD have a requirement of docutils<0.17. Again, emphasis on should. Even the latest versions of sphinx-rtd-theme do NOT support docutils=0.19!

/home/docs/checkouts/readthedocs.org/user_builds/build123d/envs/latest/bin/python -m pip install --upgrade --no-cache-dir pillow mock==1.0.1 alabaster>=0.7,<0.8,!=0.7.5 commonmark==0.9.1 recommonmark==0.5.0 sphinx sphinx-rtd-theme readthedocs-sphinx-ext<2.3

[benjaoming]

@jdegenstein thanks for going through this. sphinx-rtd-theme 0.5.1 didn't put upper bounds on neither Sphinx nor docutils: https://github.com/readthedocs/sphinx_rtd_theme/blob/0.5.1/setup.py#L118 - we should definitely look at why the build process seems to be producing an incompatible set of packages and why it pins sphinx-rtd-theme but not Sphinx (that seems odd).

There's a lot of legacy in these commands, an intention to avoid breaking older setups.

[jdegenstein]

OK, thank you very much for the response @benjaoming. I can see how the missing docutils pin would cause this. For anyone reading, it would be advisable to pin sphinx_rtd_theme>=0.5.2 (instead of 0.5.1 as is widely advised on SO) because 0.5.2 was the first to pin the docutils version to <0.17.

Regarding readthedocs, it runs two commands to install most of the packages, firstly: /home/docs/checkouts/readthedocs.org/user_builds/build123d/envs/latest/bin/python -m pip install --upgrade --no-cache-dir pillow mock==1.0.1 alabaster>=0.7,<0.8,!=0.7.5 commonmark==0.9.1 recommonmark==0.5.0 sphinx sphinx-rtd-theme readthedocs-sphinx-ext<2.3 Which results in installing some packages that are silently incompatible with each other (docutils-0.19 sphinx-6.1.1 sphinx-rtd-theme-0.5.1)

and then the next step run on readthedocs is this: /home/docs/checkouts/readthedocs.org/user_builds/build123d/envs/latest/bin/python -m pip install --exists-action=w --no-cache-dir -r docs/requirements.txt Which correctly pulls in and replaces packages as specified in the requirements.txt file.

So the mystery for me is solved. The answer is DO NOT use sphinx-rtd-theme<=0.5.1 and DO pin your version of sphinx (5.3.0 in my case).

[benjaoming]

I'll break this down, not least to understand for future reference.

• Nothing is wrong - but a few things can be improved or tweaked (see long version).
• If a project configures the Python version in .readthedocs.yaml, it should also specify the requirements, especially including Sphinx. It's actually required*! @jdegenstein you're already doing this :100:
• Projects that don't use .readthedocs.yaml or don't specify the Python version will run the default image (Python 3.7), and will only have Sphinx 5.3 with an old version of the theme.

*) It is required to specify requirements when setting the Python version, as the current Sphinx builder will otherwise install the broken set of docutils-0.19 sphinx-6.1.1 sphinx-rtd-theme-0.5.1 that @jdegenstein identified.

The longer version:

On documentation images that do not specify Python nor Sphinx requirements, it's becoming increasingly impossible to choose a one-size-fits-all. The current default image is not going to be updated much, neither is the Sphinx builder.

Updating and maintaining a .readthedocs.yaml and requirements.txt is the responsibility of the documentation owner, but for instance sphinx-rtd-theme still has wide support so getting compatible packages should be quite easy.

Because the default Sphinx builder is running on Python 3.7, it luckily doesn't get Sphinx 6, as it has python_requires = >=3.8. Running the command with newer Python versions is yielding broken setups. Because of this illegal state, we could consider introducing new configuration constraints on .readthedocs.yaml (for instance that either requirements or Sphinx version are mandatory configuration options once the Python version is also specified).

[benjaoming]

Addition to how the Read the Docs default Sphinx environment installs its default dependencies: Once we release a final version 1.2.0 of sphinx-rtd-theme, it will change the default builder's package resolution from docutils-0.19 sphinx-6.1.1 sphinx-rtd-theme-0.5.1 to docutils-0.19 sphinx-6.1.1 sphinx-rtd-theme-1.2.0 on Python 3.8+ and docutils-0.19 sphinx-5.3.0 sphinx-rtd-theme-1.2.0 on Python <3.8. Currently, we haven't tested if docutils 0.18 or 0.19 will be picked in this scenario, but hopefully it's not a sensitive issue. We'll get docutils 0.19 support investigated very soon, too.

The pip resolver currently backtracks from 1.1.1 down to 0.5.1 (a version from the era where upper bounds were considered unnecessary).

[GregDMeyer]

I have deeply read the above and am still a little confused which packages exactly I need to pin to get this to work. The docs build fine locally, but still don't have bullets on readthedocs.

My docs (most recent iteration of) requirements.txt, although I've found that many different combinations seem to successfully build locally and break on readthedocs:

Sphinx==5.3.0
ipykernel==6.21.0
nbsphinx==0.8.12
sphinx_rtd_theme==1.2.0rc2
myst-parser==0.18.1
docutils==0.18.1

My .readthedocs.yaml:

version: 2

build:
  os: ubuntu-22.04
  tools:
    python: "3.11"

python:
  install:
  - requirements: docs/requirements.txt

edit: I should note that I've found that when I pin docutils==0.16, I get this error when building locally, I guess because I am using myst:

ERROR: Error in "include" directive:
unknown option: "parser".

.. include:: pubs.md
   :parser: myst_parser.sphinx_

[benjaoming]

@GregDMeyer since you are using Read the Docs, can you create a build with the configuration and point to the build log and a page where the issue is demonstrated? Thanks :pray:

[GregDMeyer]

@benjaoming Thanks for following up! I actually got it working since then by doing pip freeze > requirements.txt---pinning literally every package to the same version that I have locally. This doesn't feel like a particularly clean solution though ;-)

I just tried to revert to a commit where it was broken in order to give you an example in case you want to dig further, and... the docs look fine, from the exact same commit that was broken before. Is there some sort of caching or something that could cause this to happen? In any case, my bullet points look fine now, so I guess no biggie, I'm fine to not spend any more time worrying about it!