docs: "yield from" breaks Pygments syntax coloring in doc examples

[918f67d7-4fec-4a8d-93e3-6530aeb1e57e]

BPO	14503
Nosy	@birkenfeld, @amauryfa, @bitdancer, @berkerpeksag
Files	screenshot.png

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = None closed_at = created_at = labels = ['type-bug', 'docs'] title = 'docs: "yield from" breaks Pygments syntax coloring in doc examples' updated_at = user = 'https://bugs.python.org/RamchandraApte' ``` bugs.python.org fields: ```python activity = actor = 'georg.brandl' assignee = 'docs@python' closed = True closed_date = closer = 'georg.brandl' components = ['Documentation'] creation = creator = 'Ramchandra Apte' dependencies = [] files = ['25135'] hgrepos = [] issue_num = 14503 keywords = [] message_count = 10.0 messages = ['157551', '157552', '157579', '157584', '157587', '157589', '157593', '157595', '157596', '171647'] nosy_count = 6.0 nosy_names = ['georg.brandl', 'amaury.forgeotdarc', 'r.david.murray', 'docs@python', 'Ramchandra Apte', 'berker.peksag'] pr_nums = [] priority = 'normal' resolution = 'fixed' stage = 'needs patch' status = 'closed' superseder = None type = 'behavior' url = 'https://bugs.python.org/issue14503' versions = ['Python 3.3', 'Python 3.4'] ```

[918f67d7-4fec-4a8d-93e3-6530aeb1e57e]

In http://docs.python.org/dev/whatsnew/3.3.html#pep-380-syntax-for-delegating-to-a-subgenerator , two code samples

[918f67d7-4fec-4a8d-93e3-6530aeb1e57e]

Whoops - In http://docs.python.org/dev/whatsnew/3.3.html#pep-380-syntax-for-delegating-to-a-subgenerator , two code samples are not highlighted.

[bitdancer]

I'm not seeing any unhighlighted examples in that section.

[berkerpeksag]

I can reproduce. See screenshot.png.

[bitdancer]

Berker: you can reproduce the bug, or the fact that they are highlighted? The png looks like they are highlighted, so I assume the latter.

[amauryfa]

they are highlighted No. Keywords normally appear in bold font, and with another color. This is not the case in these blocks.

Probably because the language detection does not work with the new "yield from" construct. Sphinx should always use the latest Python...

[bitdancer]

Ah, you mean they are not *syntax* highlighted. Now I understand. Sorry for missing that.

My understanding is that Sphinx does not use Python directly to parse the code and highlight it, it uses pygments, which uses regexes. So this basically amounts to a feature request for pygments.

[amauryfa]

Not exactly. Sphinx first tries to see if the block is a Python script, and to do so it uses the Python compiler: https://bitbucket.org/birkenfeld/sphinx/src/164f59b2d946/sphinx/highlighting.py#cl-117 In case of SyntaxError, it treats the whole block as plain text and does not try to use pygmentq at all.

[bitdancer]

Huh. Well, in another issue Georg said it was now possible to upgrade the doc build toolchain to Python3.

[birkenfeld]

I've updated the version of Pygments used by 3.3 and 3.4 branches, which fixes this and the "raise from" issue.