The title from `toc_tokens` ignores the `smarty` extension

[oprypin]

Current actual result:

>>> import markdown
>>> md = markdown.Markdown(extensions=['toc', 'smarty'])
>>> md.convert('# *Foo* --- `bar`')
'<h1 id="foo-bar"><em>Foo</em> &mdash; <code>bar</code></h1>'
>>> md.toc_tokens
[{'level': 1, 'id': 'foo-bar', 'name': 'Foo --- bar', 'children': []}]

Expected result:

[{'level': 1, 'id': 'foo-bar', 'name': 'Foo — bar', 'children': []}]

Background where I discovered this: https://github.com/mkdocs/mkdocs/issues/3357#issuecomment-1925446505

This happens because smarty runs at a lower priority than toc (unlike most other treeprocessors) and it doesn't have a chance to kick in.

• https://github.com/Python-Markdown/markdown/blob/a63e6f3416f2e4fc7c2d5edb2768ceed9218df20/markdown/extensions/toc.py#L404
• https://github.com/Python-Markdown/markdown/blob/a63e6f3416f2e4fc7c2d5edb2768ceed9218df20/markdown/extensions/smarty.py#L272

This other extension is presumably affected in the same way, probably among others: https://github.com/facelessuser/pymdown-extensions/blob/8f5283f71e8833f1ba0fa0bbe2680e1ad2c6cb19/pymdownx/smartsymbols.py#L167

Something should be done so that all the rest of the treeprocessors are also applied before saving the title. Maybe postprocessors too, rather than applying the unescape functionality out-of-band.

[oprypin]

By the way, please don't solve this by lowering the priority of the toc extension, so much stuff I made already depends on its precise priority 🥲

[waylan]

I didn't have much involvement in the smarty extension, which was added much later than TOC. I wonder why it was given a lower priority. @mitya57 do you recall? I seem to recall you being involved in the development of the smarty extension.

I ask why because, barring any good reason not to, I have no object in raising the priority to be before TOC.

[mitya57]

When I added smarty treeprocessor in 47ec0cb1ea31125da1481fb82319f0a1cdfd20e1, I probably just pushed it to the end without thinking too much.

The tests pass when changing its priority, so I think it's fine to do that. Submitted PR as #1440.