`tests/test_syntax.py::test_extensions[compare54]` fails on PyPy3.10 7.3.15

[mgorny]

Description

The test suite fails when run on PyPy3.10 7.3.15:

$ pypy3 run_tests.py 
========================================================= test session starts =========================================================
platform linux -- Python 3.10.13[pypy-7.3.15-final], pytest-7.4.4, pluggy-1.4.0
rootdir: /tmp/pymdown-extensions
plugins: xprocess-0.23.0, xdist-3.5.0, subtests-0.11.0, asyncio-0.23.5, anyio-4.3.0
asyncio: mode=strict
collected 65 items                                                                                                                    

tests/test_syntax.py ......................................................F..........                                          [100%]

============================================================== FAILURES ===============================================================
_____________________________________________________ test_extensions[compare54] ______________________________________________________

compare = (OrderedDict([('extensions', OrderedDict([('markdown.extensions.attr_list', None), ('pymdownx.highlight', OrderedDict(...83e020bce0>)])])]))])), ('css', [])]), '/tmp/pymdown-extensions/tests/extensions/superfences/superfences (normal).txt')

    def test_extensions(compare):
        """Test extensions."""

>       compare_results(*compare)

tests/test_syntax.py:150: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_syntax.py:51: in compare_results
    check_markdown(testfile, extension, extension_config, wrapper, update)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

testfile = '/tmp/pymdown-extensions/tests/extensions/superfences/superfences (normal).txt'
extension = ['markdown.extensions.attr_list', 'pymdownx.highlight', 'pymdownx.superfences']
extension_config = {'pymdownx.highlight': OrderedDict([('extend_pygments_lang', [OrderedDict([('name', 'php-inline'), ('lang', 'php'), ('..., 'sequence'), ('class', 'uml-sequence-diagram'), ('format', <function fence_code_format at 0x00007f83e020bce0>)])])])}
wrapper = '%s', update = False

    def check_markdown(testfile, extension, extension_config, wrapper, update=False):
        """Check the markdown."""

        expected_html = os.path.splitext(testfile)[0] + '.html'
        with codecs.open(testfile, 'r', encoding='utf-8') as f:
            source = f.read()

        results = wrapper % markdown.Markdown(
            extensions=extension, extension_configs=extension_config
        ).convert(source)

        try:
            with codecs.open(expected_html, 'r', encoding='utf-8') as f:
                expected = f.read().replace("\r\n", "\n")
        except Exception:
            expected = ''

        diff = list(
            difflib.unified_diff(
                expected.splitlines(True),
                results.splitlines(True),
                expected_html,
                os.path.join(os.path.dirname(testfile), 'results.html'),
                n=3
            )
        )
        if diff:
            if update:
                print('Updated: %s' % expected_html)
                with codecs.open(expected_html, 'w', encoding='utf-8') as f:
                    f.write(results)
            else:
>               raise Exception(
                    'Output from "%s" failed to match expected '
                    'output.\n\n%s' % (testfile, ''.join(diff))
E                   Exception: Output from "/tmp/pymdown-extensions/tests/extensions/superfences/superfences (normal).txt" failed to match expected output.
E                   
E                   --- /tmp/pymdown-extensions/tests/extensions/superfences/superfences (normal).html
E                   +++ /tmp/pymdown-extensions/tests/extensions/superfences/results.html
E                   @@ -77,7 +77,7 @@
E                    ------------------------------------------------------------
E                    A   F#m Bm  E   C#m D   E7  A
E                    A#  Gm  Cm  F   Dm  D#  F7  A#
E                   -B♭  Gm  Cm  F   Dm  E♭m F7  B♭
E                   +B♭    Gm  Cm  F   Dm  E♭m   F7  B♭
E                    </code></pre></div>
E                    </li>
E                    <li>

tests/test_syntax.py:86: Exception
======================================================= short test summary info =======================================================
FAILED tests/test_syntax.py::test_extensions[compare54] - Exception: Output from "/tmp/pymdown-extensions/tests/extensions/superfences/superfences (normal).txt" failed to match expected ou...
==================================================== 1 failed, 64 passed in 18.32s ====================================================
========================================================= test session starts =========================================================
platform linux -- Python 3.10.13[pypy-7.3.15-final], pytest-7.4.4, pluggy-1.4.0
rootdir: /tmp/pymdown-extensions
plugins: xprocess-0.23.0, xdist-3.5.0, subtests-0.11.0, asyncio-0.23.5, anyio-4.3.0
asyncio: mode=strict
collected 13 items                                                                                                                    

tests/test_targeted.py .............                                                                                            [100%]

========================================================= 13 passed in 0.12s ==========================================================

Minimal Reproduction

pypy3 run_tests.py

Version(s) & System Info

• Operating System: Gentoo Linux amd64
• Python Version: PyPy3.10 7.3.15
• Package Version: 10.7 (8f5283f71e8833f1ba0fa0bbe2680e1ad2c6cb19)

[facelessuser]

Looks like PyPy is doing something wrong. All of our tests are passing for CPython which we have passing tests for, which we formally support.

[mgorny]

I'm happy to help but the code is too complex for me to even start figuring out where something could be going wrong here.

[mgorny]

Hmm, there is something wrong with that code on CPython too, just isn't covered by tests. If I use a wide character, it is incorrectly aligned on CPython too, e.g.:

E                   -梓  Gm  Cm  F   Dm  E♭m F7  B♭
E                   +梓   Gm  Cm  F   Dm  E♭m F7  B♭

[mgorny]

I'm happy to help but the code is too complex for me to even start figuring out where something could be going wrong here.

It seems that str.expandtabs() works incorrectly on PyPy. I'll file a bug about it.

Still, using str.expandtabs() doesn't produce correct results for wide unicode characters, I guess you need wcwidth for that.

[facelessuser]

Seems to work in my tests:

Can you provide a failing case?

[mgorny]

Sure. All I did was a quick patch:

diff --git a/tests/extensions/superfences/superfences (normal).html b/tests/extensions/superfences/superfences (normal).html
index cc9444ce..e025d6cb 100644
--- a/tests/extensions/superfences/superfences (normal).html    
+++ b/tests/extensions/superfences/superfences (normal).html    
@@ -77,7 +77,7 @@ T   Tp  Sp  D   Dp  S   D7  T
 ------------------------------------------------------------
 A   F#m Bm  E   C#m D   E7  A
 A#  Gm  Cm  F   Dm  D#  F7  A#
-B♭  Gm  Cm  F   Dm  E♭m F7  B♭
+梓  Gm  Cm  F   Dm  E♭m F7  B♭
 </code></pre></div>
 </li>
 <li>
diff --git a/tests/extensions/superfences/superfences (normal).txt b/tests/extensions/superfences/superfences (normal).txt
index 565f960a..7d428b44 100644
--- a/tests/extensions/superfences/superfences (normal).txt     
+++ b/tests/extensions/superfences/superfences (normal).txt     
@@ -87,7 +87,7 @@ as a fenced code block.
        ------------------------------------------------------------
        A       F#m     Bm      E       C#m     D       E7      A
        A#      Gm      Cm      F       Dm      D#      F7      A#
-       B♭      Gm      Cm      F       Dm      E♭m     F7      B♭
+       梓      Gm      Cm      F       Dm      E♭m     F7      B♭

• Here is a highlighted code block with line numbers:

[mgorny]

Hmm, sorry, I think GitHub broke tabs.

Lemme attach it instead: unicode.patch.txt

[facelessuser]

Yep, seems to be working here in CPython. Maybe it is an environment issue you are having?

If you are running "normal" it will break. Tabs are not preserved as Python Markdown converts all tabs to spaces. But if you enable the preserve_tabs option, it will preserve tabs: https://facelessuser.github.io/pymdown-extensions/extensions/superfences/#preserve-tabs.

[mgorny]

That's what I'm saying. In "normal", it's replacing tabs with the wrong number of spaces.

[facelessuser]

That's not an error that's how Python Markdown works. It's even documented in the link I provided. I had to create a special mode as a workaround to navigate around out, and I've suggested you enable it. Python Markdown replaced every tab with for spaces, regardless of what you think it should do, hence the special mode.

[mgorny]

Sure. I'm just pointing out that it is using the wrong number of spaces to do that. Do you want me to file a separate bug about that?

[facelessuser]

@mgorny, that's because Python Markdown has already messed with the tabs when you are in normal mode, which is why we have a special tab mode via the option I described. All you are pointing out is why I created the mode.

preserve_tabs changes the way SuperFences works and forces it to inject itself much earlier in the Python Markdown process, before Python Markdown mangles tabs. In normal mode there are no tabs for SuperFences to translate as they've already been translated. Let's compare Python Markdown's fenced code extension with SuperFences. In normal mode, we match them, because they've already messed up the tabs.

import markdown

MD = """

============================================================ T Tp Sp D Dp S D7 T

A F#m Bm E C#m D E7 A A# Gm Cm F Dm D# F7 A# 梓 Gm Cm F Dm E♭m F7 B♭

"""

html = markdown.markdown(
    MD,
    extensions=['fenced_code'],
)

print(html)

html = markdown.markdown(
    MD,
    extensions=['pymdownx.superfences'],
)

print(html)

html = markdown.markdown(
    MD,
    extensions=['pymdownx.superfences'],
    extension_configs={
        'pymdownx.superfences': {
            'preserve_tabs': True
        }
    }
)

print(html)

Output:

<pre><code>============================================================
T   Tp  Sp  D   Dp  S   D7  T
------------------------------------------------------------
A   F#m Bm  E   C#m D   E7  A
A#  Gm  Cm  F   Dm  D#  F7  A#
梓   Gm  Cm  F   Dm  E♭m F7  B♭
</code></pre>
<div class="highlight"><pre><span></span><code>============================================================
T   Tp  Sp  D   Dp  S   D7  T
------------------------------------------------------------
A   F#m Bm  E   C#m D   E7  A
A#  Gm  Cm  F   Dm  D#  F7  A#
梓   Gm  Cm  F   Dm  E♭m F7  B♭
</code></pre></div>
<div class="highlight"><pre><span></span><code>============================================================
T   Tp  Sp  D   Dp  S   D7  T
------------------------------------------------------------
A   F#m Bm  E   C#m D   E7  A
A#  Gm  Cm  F   Dm  D#  F7  A#
梓   Gm  Cm  F   Dm  E♭m F7  B♭
</code></pre></div>

So, this is a bug for Python Markdown. If they were to fix this, we'd no longer need a special mode.

[facelessuser]

I will preface this with, what one man considers a bug, others may consider expected behavior. I cannot promise Python Markdown would enhance their tab logic, so I provide parity with them, and if you care, you can enable preserve_tabs. 99% of the time, I don't care because I don't use tabs, but when I do, preserve_tabs is the way to go.

[facelessuser]

For the stated reasons, there is nothing to fix. Closing issue.