New version removes blank in doctest, so all doctests fail.

[henrifroese]

Description

After updating from 19.10b0 to 20.8, black removes trailing whitespaces in our doctests (that Pandas returns), so all the doctests fail. In the example below, the trailing blank is after "Texthero" in the doctest's output.

To Reproduce

Create a file

def f(s):
    """
    Remove content within parentheses '()' and the parentheses by themself.

    Examples
    --------
    >>> import pandas as pd
    >>> s = pd.Series("Texthero (is not a superhero!)")
    >>> f(s)
    0    Texthero 
    dtype: object
    """
    return s.str.replace(r"\([^()]*\)", "")

if __name__ == "__main__":
    import doctest
    doctest.testmod()

2. run doctests through e.g. python3 main.py -v -> they pass
3. format file with old black version 19 -> they still pass
4. format file with new black version 20 -> they fail

Expected behavior

Black should not remove the trailing blank in the doctests.

Environment.

• Version: master
• OS and Python version: Happens on Windows/Mac/Linux locally and in our Travis-CI builds.

Does this bug also happen on master?

Yes

[ambv]

Thanks for your report! I agree Black should do better here but bear in mind that your doctests are very brittle as is. If you tell your text editor (or .editorconfig) to automatically remove trailing spaces on save, the test will fail, too. I'd suggest using >>> repr(f(s)) instead of just >>> f(s) in your doctests. This way you're no longer relying on trailing invisible characters.

[henrifroese]

That makes sense. However, we love using doctests for both documentation and very simple testing. Using repr(f(s)) makes the docstring less intuitive/readable for users, and we'd really like to be able to show users how to use the function while still profiting from the testing.

[dlax]

Another option is the NORMALIZE_WHITESPACE doctest flag. With that, you can remove trailing whitespaces while keeping a doctest pass:

    >>> f(s)  # doctest: +NORMALIZE_WHITESPACE
    0    Texthero
    dtype: object

or

if __name__ == "__main__":
    import doctest
    doctest.testmod(optionflags=doctest.NORMALIZE_WHITESPACE)

[pawamoy]

This prevents using two trailing spaces (line break) in Markdown docstrings :confused:

[pawamoy]

Ah, found a more recent issue for this: https://github.com/psf/black/issues/3306

[peterjc]

Curiously black 24.2.0 does not treat trailing whitespace in all doctests equally, consider this test case:

test_case.py.txt

black 24.2.0 does not alter the module level doctest, just the function doctest.

ruff lint 0.3.1 alters both, reported as https://github.com/astral-sh/ruff/issues/10275

[MichaReiser]

@peterjc Black doesn't format module docstrings in their stable style (it's a recent preview style addition), whereas Ruff shipped module docstring formatting in the latest stable release.

# Input

"""test
"""

# Black stable
"""test
"""

# Black preview
"""test"""

# Ruff
"""test"""