search

samuelcolvin / python-devtools

Dev tools for python
https://python-devtools.helpmanual.io/
MIT License
985 stars 47 forks source link

0.11.0: Issue with several tests due to string matching #132

Open dvzrv opened 1 year ago

dvzrv commented 1 year ago

Hi! I'm packaging this project for Arch Linux and ran into a few problems when trying to run tests for 0.11.0.

============================= test session starts ==============================
platform linux -- Python 3.11.3, pytest-7.4.0, pluggy-1.0.0 -- /usr/bin/python
cachedir: .pytest_cache
rootdir: /build/python-devtools/src/devtools-0.11.0
configfile: pyproject.toml
testpaths: tests
plugins: mock-3.10.0, pretty-1.2.0
collecting ... collected 117 items

tests/test_ansi.py::test_colours PASSED                                  [  0%]
tests/test_ansi.py::test_no_reset PASSED                                 [  1%]
tests/test_ansi.py::test_combine_styles PASSED                           [  2%]
tests/test_ansi.py::test_no_styles PASSED                                [  3%]
tests/test_ansi.py::test_style_str PASSED                                [  4%]
tests/test_ansi.py::test_non_str_input PASSED                            [  5%]
tests/test_ansi.py::test_invalid_style_str PASSED                        [  5%]
tests/test_ansi.py::test_print_not_tty PASSED                            [  6%]
tests/test_ansi.py::test_print_is_tty PASSED                             [  7%]
tests/test_ansi.py::test_print_tty_error PASSED                          [  8%]
tests/test_ansi.py::test_get_styles PASSED                               [  9%]
tests/test_ansi.py::test_repr_str PASSED                                 [ 10%]
tests/test_custom_pretty.py::test_simple PASSED                          [ 11%]
tests/test_custom_pretty.py::test_skip PASSED                            [ 11%]
tests/test_custom_pretty.py::test_yield_other PASSED                     [ 12%]
tests/test_custom_pretty.py::test_pretty_not_func PASSED                 [ 13%]
tests/test_custom_pretty.py::test_pretty_class PASSED                    [ 14%]
tests/test_custom_pretty.py::test_pydantic_pretty SKIPPED (numpy not
installed)                                                               [ 15%]
tests/test_expr_render.py::test_simple PASSED                            [ 16%]
tests/test_expr_render.py::test_subscription PASSED                      [ 17%]
tests/test_expr_render.py::test_exotic_types PASSED                      [ 17%]
tests/test_expr_render.py::test_newline PASSED                           [ 18%]
tests/test_expr_render.py::test_trailing_bracket PASSED                  [ 19%]
tests/test_expr_render.py::test_multiline PASSED                         [ 20%]
tests/test_expr_render.py::test_multiline_trailing_bracket PASSED        [ 21%]
tests/test_expr_render.py::test_kwargs PASSED                            [ 22%]
tests/test_expr_render.py::test_kwargs_multiline PASSED                  [ 23%]
tests/test_expr_render.py::test_multiple_trailing_lines PASSED           [ 23%]
tests/test_expr_render.py::test_very_nested_last_statement PASSED        [ 24%]
tests/test_expr_render.py::test_syntax_warning PASSED                    [ 25%]
tests/test_expr_render.py::test_no_syntax_warning PASSED                 [ 26%]
tests/test_expr_render.py::test_await PASSED                             [ 27%]
tests/test_expr_render.py::test_other_debug_arg PASSED                   [ 28%]
tests/test_expr_render.py::test_other_debug_arg_not_literal PASSED       [ 29%]
tests/test_expr_render.py::test_executing_failure PASSED                 [ 29%]
tests/test_expr_render.py::test_format_inside_error PASSED               [ 30%]
tests/test_insert_assert.py::test_insert_assert FAILED                   [ 31%] [ 31%]
tests/test_insert_assert.py:22 test_insert_assert - assert "def test…
tests/test_insert_assert.py::test_insert_assert_no_pretty FAILED         [ 32%] [ 32%]
tests/test_insert_assert.py:40 test_insert_assert_no_pretty - assert…
tests/test_insert_assert.py::test_insert_assert_print PASSED             [ 33%]
tests/test_insert_assert.py::test_insert_assert_fail PASSED              [ 34%]
tests/test_insert_assert.py::test_deep FAILED                            [ 35%] [ 35%]
tests/test_insert_assert.py:80 test_deep - assert "def test_deep(ins…
tests/test_insert_assert.py::test_enum FAILED                            [ 35%] [ 35%]
tests/test_insert_assert.py:103 test_enum - AssertionError: assert '…
tests/test_insert_assert.py::test_insert_assert_black FAILED             [ 36%] [ 36%]
tests/test_insert_assert.py:126 test_insert_assert_black - assert "'…
tests/test_insert_assert.py::test_insert_assert_repeat PASSED            [ 37%]
tests/test_main.py::test_print PASSED                                    [ 38%]
tests/test_main.py::test_print_kwargs PASSED                             [ 39%]
tests/test_main.py::test_print_generator PASSED                          [ 40%]
tests/test_main.py::test_format PASSED                                   [ 41%]
tests/test_main.py::test_print_subprocess PASSED                         [ 41%]
tests/test_main.py::test_odd_path PASSED                                 [ 42%]
tests/test_main.py::test_small_call_frame PASSED                         [ 43%]
tests/test_main.py::test_small_call_frame_warning PASSED                 [ 44%]
tests/test_main.py::test_kwargs PASSED                                   [ 45%]
tests/test_main.py::test_kwargs_orderless PASSED                         [ 46%]
tests/test_main.py::test_simple_vars PASSED                              [ 47%]
tests/test_main.py::test_attributes PASSED                               [ 47%]
tests/test_main.py::test_eval PASSED                                     [ 48%]
tests/test_main.py::test_warnings_disabled PASSED                        [ 49%]
tests/test_main.py::test_eval_kwargs PASSED                              [ 50%]
tests/test_main.py::test_exec PASSED                                     [ 51%]
tests/test_main.py::test_colours PASSED                                  [ 52%]
tests/test_main.py::test_colours_warnings PASSED                         [ 52%]
tests/test_main.py::test_inspect_error PASSED                            [ 53%]
tests/test_main.py::test_breakpoint PASSED                               [ 54%]
tests/test_main.py::test_starred_kwargs PASSED                           [ 55%]
tests/test_main.py::test_pretty_error PASSED                             [ 56%]
tests/test_main.py::test_multiple_debugs PASSED                          [ 57%]
tests/test_main.py::test_return_args PASSED                              [ 58%]
tests/test_prettier.py::test_dict PASSED                                 [ 58%]
tests/test_prettier.py::test_print PASSED                                [ 59%]
tests/test_prettier.py::test_colours PASSED                              [ 60%]
tests/test_prettier.py::test_list PASSED                                 [ 61%]
tests/test_prettier.py::test_set PASSED                                  [ 62%]
tests/test_prettier.py::test_tuple PASSED                                [ 63%]
tests/test_prettier.py::test_generator PASSED                            [ 64%]
tests/test_prettier.py::test_named_tuple PASSED                          [ 64%]
tests/test_prettier.py::test_generator_no_yield PASSED                   [ 65%]
tests/test_prettier.py::test_str PASSED                                  [ 66%]
tests/test_prettier.py::test_str_repr PASSED                             [ 67%]
tests/test_prettier.py::test_bytes PASSED                                [ 68%]
tests/test_prettier.py::test_short_bytes PASSED                          [ 69%]
tests/test_prettier.py::test_bytearray PASSED                            [ 70%]
tests/test_prettier.py::test_bytearray_short PASSED                      [ 70%]
tests/test_prettier.py::test_map PASSED                                  [ 71%]
tests/test_prettier.py::test_filter PASSED                               [ 72%]
tests/test_prettier.py::test_counter PASSED                              [ 73%]
tests/test_prettier.py::test_dataclass PASSED                            [ 74%]
tests/test_prettier.py::test_nested_dataclasses PASSED                   [ 75%]
tests/test_prettier.py::test_indent_numpy PASSED                         [ 76%]
tests/test_prettier.py::test_indent_numpy_short PASSED                   [ 76%]
tests/test_prettier.py::test_ordered_dict PASSED                         [ 77%]
tests/test_prettier.py::test_frozenset PASSED                            [ 78%]
tests/test_prettier.py::test_deep_objects PASSED                         [ 79%]
tests/test_prettier.py::test_call_args PASSED                            [ 80%]
tests/test_prettier.py::test_multidict PASSED                            [ 81%]
tests/test_prettier.py::test_cimultidict PASSED                          [ 82%]
tests/test_prettier.py::test_os_environ PASSED                           [ 82%]
tests/test_prettier.py::test_dir PASSED                                  [ 83%]
tests/test_prettier.py::test_instance_dict PASSED                        [ 84%]
tests/test_prettier.py::test_class_dict PASSED                           [ 85%]
tests/test_prettier.py::test_dictlike PASSED                             [ 86%]
tests/test_prettier.py::test_asyncpg_record PASSED                       [ 87%]
tests/test_prettier.py::test_dict_type PASSED                            [ 88%]
tests/test_prettier.py::test_sqlalchemy_object PASSED                    [ 88%]
tests/test_prettier.py::test_ast_expr PASSED                             [ 89%]
tests/test_prettier.py::test_ast_module PASSED                           [ 90%]
tests/test_timer.py::test_simple PASSED                                  [ 91%]
tests/test_timer.py::test_multiple PASSED                                [ 92%]
tests/test_timer.py::test_unfinished PASSED                              [ 93%]
tests/test_timer.py::test_multiple_not_verbose PASSED                    [ 94%]
tests/test_timer.py::test_unfinished_summary PASSED                      [ 94%]
tests/test_timer.py::test_summary_not_started PASSED                     [ 95%]
tests/test_utils.py::test_env_true PASSED                                [ 96%]
tests/test_utils.py::test_env_bool PASSED                                [ 97%]
tests/test_utils.py::test_use_highlight_manually_set PASSED              [ 98%]
tests/test_utils.py::test_use_highlight_auto_not_win PASSED              [ 99%]
tests/test_utils.py::test_use_highlight_auto_win SKIPPED (not windows
os)                                                                      [100%]

=================================== FAILURES ===================================
______________________________ test_insert_assert ______________________________

pytester_pretty = <pytest_pretty.PytesterWrapper object at 0x7f73cc6ab410>

    def test_insert_assert(pytester_pretty):
        os.environ.pop('CI', None)
        pytester_pretty.makeconftest(config)
        test_file = pytester_pretty.makepyfile(default_test)
        result = pytester_pretty.runpytest()
        result.assert_outcomes(passed=2)
        # print(result.outlines)
>       assert test_file.read_text() == (
            'def test_ok():\n'
            '    assert 1 + 2 == 3\n'
            '\n'
            'def test_string_assert(insert_assert):\n'
            "    thing = 'foobar'\n"
            '    # insert_assert(thing)\n'
            '    assert thing == "foobar"'
        )
E       assert "def test_ok():\n    assert 1 + 2 == 3\n\ndef test_string_assert(insert_assert):\n    thing = 'foobar'\n    # insert_assert(thing)\n    assert thing == 'foobar'" == 'def test_ok():\n    assert 1 + 2 == 3\n\ndef test_string_assert(insert_assert):\n    thing = \'foobar\'\n    # insert_assert(thing)\n    a
ssert thing == "foobar"'
E           def test_ok():
E               assert 1 + 2 == 3
E
E           def test_string_assert(insert_assert):
E               thing = 'foobar'
E               # insert_assert(thing)
E         -     assert thing == "foobar"
E         ?                     ^      ^
E         +     assert thing == 'foobar'
E         ?                     ^      ^

/build/python-devtools/src/devtools-0.11.0/tests/test_insert_assert.py:30: AssertionError
----------------------------- Captured stdout call -----------------------------
============================= test session starts ==============================
platform linux -- Python 3.11.3, pytest-7.4.0, pluggy-1.0.0
rootdir: /tmp/pytest-of-builduser/pytest-0/test_insert_assert0
plugins: mock-3.10.0, pretty-1.2.0
collected 2 items

test_insert_assert.py ..i                                                [100%]

=============================== warnings summary ===============================
../../../../usr/lib/python3.11/site-packages/_pytest/config/__init__.py:755
  /usr/lib/python3.11/site-packages/_pytest/config/__init__.py:755: PytestAssertRewriteWarning: Module already imported so cannot be rewritten: devtools.pytest_plugin
    self.import_plugin(import_spec)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
Replaced 1 insert_assert() call in 1 file
Results (0.01s):
         2 passed
         1 warning
         1 insert assert
_________________________ test_insert_assert_no_pretty _________________________

pytester = <Pytester PosixPath('/tmp/pytest-of-builduser/pytest-0/test_insert_assert_no_pretty0')>

    def test_insert_assert_no_pretty(pytester):
        os.environ.pop('CI', None)
        pytester.makeconftest(config)
        test_file = pytester.makepyfile(default_test)
        result = pytester.runpytest('-p', 'no:pretty')
        result.assert_outcomes(passed=2)
>       assert test_file.read_text() == (
            'def test_ok():\n'
            '    assert 1 + 2 == 3\n'
            '\n'
            'def test_string_assert(insert_assert):\n'
            "    thing = 'foobar'\n"
            '    # insert_assert(thing)\n'
            '    assert thing == "foobar"'
        )
E       assert "def test_ok():\n    assert 1 + 2 == 3\n\ndef test_string_assert(insert_assert):\n    thing = 'foobar'\n    # insert_assert(thing)\n    assert thing == 'foobar'" == 'def test_ok():\n    assert 1 + 2 == 3\n\ndef test_string_assert(insert_assert):\n    thing = \'foobar\'\n    # insert_assert(thing)\n    a
ssert thing == "foobar"'
E           def test_ok():
E               assert 1 + 2 == 3
E
E           def test_string_assert(insert_assert):
E               thing = 'foobar'
E               # insert_assert(thing)
E         -     assert thing == "foobar"
E         ?                     ^      ^
E         +     assert thing == 'foobar'
E         ?                     ^      ^

/build/python-devtools/src/devtools-0.11.0/tests/test_insert_assert.py:47: AssertionError
----------------------------- Captured stdout call -----------------------------
============================= test session starts ==============================
platform linux -- Python 3.11.3, pytest-7.4.0, pluggy-1.0.0
rootdir: /tmp/pytest-of-builduser/pytest-0/test_insert_assert_no_pretty0
plugins: mock-3.10.0
collected 2 items

test_insert_assert_no_pretty.py ..i                                      [100%]

=============================== warnings summary ===============================
../../../../usr/lib/python3.11/site-packages/_pytest/config/__init__.py:755
  /usr/lib/python3.11/site-packages/_pytest/config/__init__.py:755: PytestAssertRewriteWarning: Module already imported so cannot be rewritten: devtools.pytest_plugin
    self.import_plugin(import_spec)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
Replaced 1 insert_assert() call in 1 file
================ 2 passed, 1 warning, 1 insert assert in 0.00s =================
__________________________________ test_deep ___________________________________

pytester_pretty = <pytest_pretty.PytesterWrapper object at 0x7f73cc6a9510>

    def test_deep(pytester_pretty):
        os.environ.pop('CI', None)
        pytester_pretty.makeconftest(config)
        # language=Python
        test_file = pytester_pretty.makepyfile(
            """
        def test_deep(insert_assert):
            insert_assert([{'a': i, 'b': 2 * 2} for i in range(3)])
        """
        )
        result = pytester_pretty.runpytest()
        result.assert_outcomes(passed=1)
>       assert test_file.read_text() == (
            'def test_deep(insert_assert):\n'
            "    # insert_assert([{'a': i, 'b': 2 * 2} for i in range(3)])\n"
            '    assert [{"a": i, "b": 2 * 2} for i in range(3)] == [\n'
            '        {"a": 0, "b": 4},\n'
            '        {"a": 1, "b": 4},\n'
            '        {"a": 2, "b": 4},\n'
            '    ]'
        )
E       assert "def test_deep(insert_assert):\n    # insert_assert([{'a': i, 'b': 2 * 2} for i in range(3)])\n    assert [{'a': i, 'b': 2 * 2} for i in range(3)] == [{'a': 0, 'b': 4}, {'a': 1, 'b': 4}, {'a': 2, 'b': 4}]" == 'def test_deep(insert_assert):\n    # insert_assert([{\'a\': i, \'b\': 2 * 2} for i in range(3)
])\n    assert [{"a": i, "b": 2 * 2} for i in range(3)] == [\n        {"a": 0, "b": 4},\n        {"a": 1, "b": 4},\n        {"a": 2, "b": 4},\n    ]'
E           def test_deep(insert_assert):
E               # insert_assert([{'a': i, 'b': 2 * 2} for i in range(3)])
E         +     assert [{'a': i, 'b': 2 * 2} for i in range(3)] == [{'a': 0, 'b': 4}, {'a': 1, 'b': 4}, {'a': 2, 'b': 4}]
E         -     assert [{"a": i, "b": 2 * 2} for i in range(3)] == [
E         -         {"a": 0, "b": 4},
E         -         {"a": 1, "b": 4},
E         -         {"a": 2, "b": 4},
E         -     ]

/build/python-devtools/src/devtools-0.11.0/tests/test_insert_assert.py:93: AssertionError
----------------------------- Captured stdout call -----------------------------
============================= test session starts ==============================
platform linux -- Python 3.11.3, pytest-7.4.0, pluggy-1.0.0
rootdir: /tmp/pytest-of-builduser/pytest-0/test_deep0
plugins: mock-3.10.0, pretty-1.2.0
collected 1 item

test_deep.py .i                                                          [100%]

=============================== warnings summary ===============================
../../../../usr/lib/python3.11/site-packages/_pytest/config/__init__.py:755
  /usr/lib/python3.11/site-packages/_pytest/config/__init__.py:755: PytestAssertRewriteWarning: Module already imported so cannot be rewritten: devtools.pytest_plugin
    self.import_plugin(import_spec)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
Replaced 1 insert_assert() call in 1 file
Results (0.01s):
         1 passed
         1 warning
         1 insert assert
__________________________________ test_enum ___________________________________

pytester_pretty = <pytest_pretty.PytesterWrapper object at 0x7f73cc488290>
capsys = <_pytest.capture.CaptureFixture object at 0x7f73cc4885d0>

    def test_enum(pytester_pretty, capsys):
        os.environ.pop('CI', None)
        pytester_pretty.makeconftest(config)
        # language=Python
        pytester_pretty.makepyfile(
            """
    from enum import Enum

    class Foo(Enum):
        A = 1
        B = 2

    def test_deep(insert_assert):
        x = Foo.A
        insert_assert(x)
        """
        )
        result = pytester_pretty.runpytest('--insert-assert-print')
        result.assert_outcomes(passed=1)
        captured = capsys.readouterr()
>       assert '    assert x == Foo.A\n' in captured.out
E       AssertionError: assert '    assert x == Foo.A\n' in '============================= test session starts ==============================\nplatform linux -- Python 3.11.3, py...---------------\n\nPrinted 1 insert_assert() call in 1 file\nResults (0.01s):\n         1 passed\n         1 warning\n'
E        +  where '============================= test session starts ==============================\nplatform linux -- Python 3.11.3, py...---------------\n\nPrinted 1 insert_assert() call in 1 file\nResults (0.01s):\n         1 passed\n         1 warning\n' = CaptureResult(out='============================= test sess
ion starts ==============================\nplatform linux --...------\n\nPrinted 1 insert_assert() call in 1 file\nResults (0.01s):\n         1 passed\n         1 warning\n', err='').out

/build/python-devtools/src/devtools-0.11.0/tests/test_insert_assert.py:124: AssertionError
___________________________ test_insert_assert_black ___________________________

tmp_path = PosixPath('/tmp/pytest-of-builduser/pytest-0/test_insert_assert_black0')

    def test_insert_assert_black(tmp_path):
        old_wd = os.getcwd()
        try:
            os.chdir(tmp_path)
            (tmp_path / 'pyproject.toml').write_text(
                """\
    [tool.black]
    target-version = ["py39"]
    skip-string-normalization = true"""
            )
            load_black.cache_clear()
        finally:
            os.chdir(old_wd)

        f = load_black()
        # no string normalization
>       assert f("'foobar'") == "'foobar'\n"
E       assert "'foobar'" == "'foobar'\n"
E         - 'foobar'
E         ?         -
E         + 'foobar'

tests/test_insert_assert.py:143: AssertionError
                              Summary of Failures
┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃               ┃               ┃  Function     ┃              ┃               ┃
┃  File         ┃  Function     ┃  Line         ┃  Error Line  ┃  Error        ┃
┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│  tests/test…  │  test_inser…  │  23           │  30          │  AssertionE…  │
│  tests/test…  │  test_inser…  │  41           │  47          │  AssertionE…  │
│  tests/test…  │  test_deep    │  81           │  93          │  AssertionE…  │
│  tests/test…  │  test_enum    │  104          │  124         │  AssertionE…  │
│  tests/test…  │  test_inser…  │  127          │  143         │  AssertionE…  │
└───────────────┴───────────────┴───────────────┴──────────────┴───────────────┘
Results (0.33s):
         5 failed
       110 passed
         2 skipped

I noticed that I had to package pytest-pretty as well to run the tests. I would like to mention, that while I guess the output is nice for debugging to some degree, it is problematic in environments with limited line length and it hides the full name of the tests, which makes it hard to see what's going on.

I need the full test name for being able to disable tests via pytest --deselect.

dvzrv commented 1 year ago

This is still an issue with 0.12.2:

python-devtools-0.12.2-1-x86_64-check.log