BadTestExecutionCommandsException due to --import-mode=append

[StevenAngel-ai]

I have an issue trying to run mutmut, and when running with debug = True, I get the following output:

 Running stats     python -m pytest  -vv -x -q --import-mode=append --rootdir=.
ImportError while loading conftest '/Users/sangel/Code/autobot/components/conftest.py'.
conftest.py:54: in <module>
    from test_infrastructure.mocks import firestore_mock
test_infrastructure/mocks/firestore_mock.py:57: in <module>
    from common import firestore_tools
common/firestore_tools.py:53: in <module>
    from common import config
common/config.py:48: in <module>
    from common.getenv import getenv
common/getenv.py:50: in <module>
    from common.cache import cache
common/cache.py:65: in <module>
    from common.tools import getenv_no_cache
common/tools.py:3718: in <module>
    def x_generate_random_id__mutmut_2(size: int = 4, chars: str = string.ascii_uppercase - string.digits):
E   TypeError: unsupported operand type(s) for -: 'str' and 'str'
    exit code 4

Traceback (most recent call last):
  File "/Users/sangel/Code/autobot/.venv/bin/mutmut", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/Users/sangel/Code/autobot/.venv/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sangel/Code/autobot/.venv/lib/python3.12/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/Users/sangel/Code/autobot/.venv/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sangel/Code/autobot/.venv/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sangel/Code/autobot/.venv/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sangel/Code/autobot/.venv/lib/python3.12/site-packages/mutmut/__main__.py", line 1238, in run
    collect_or_load_stats(runner)
  File "/Users/sangel/Code/autobot/.venv/lib/python3.12/site-packages/mutmut/__main__.py", line 1064, in collect_or_load_stats
    run_stats_collection(runner)
  File "/Users/sangel/Code/autobot/.venv/lib/python3.12/site-packages/mutmut/__main__.py", line 1042, in run_stats_collection
    collect_stats_exit_code = runner.run_stats(tests=tests)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sangel/Code/autobot/.venv/lib/python3.12/site-packages/mutmut/__main__.py", line 724, in run_stats
    return int(self.execute_pytest(['-x', '-q', '--import-mode=append'] + list(tests), plugins=[stats_collector]))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sangel/Code/autobot/.venv/lib/python3.12/site-packages/mutmut/__main__.py", line 705, in execute_pytest
    raise BadTestExecutionCommandsException(params)
mutmut.__main__.BadTestExecutionCommandsException: ['-ra', '--color=yes', '--disable-socket', '--allow-unix-socket', '-vv', '-x', '-q', '--import-mode=append', '--rootdir=.']
My current theory is that --import-mode=append is just not right for the repo. Given that I have --import-mode=importlib in my pytest.ini for normal test running. And when I run pytest with the option --import-mode=append, it also errors out.

import file mismatch:
imported module 'service_test' has this __file__ attribute:
  /Users/sangel/Code/autobot/components/admin_svc/service_test.py
which is not the same as the test file we want to collect:
  /Users/sangel/Code/autobot/components/batch_query_svc/service_test.py
HINT: remove __pycache__ / .pyc files and/or use a unique basename for your test file modules

Because this repo has multiple services, which each have a service_test.py. And per the docs for pytest, with prepend and append, all your test files must have unique names. https://docs.pytest.org/en/7.1.x/explanation/pythonpath.html

Can we just either let mutmut pick up the import mode from the pytest.ini or default it to importlib instead of append?

[boxed]

I don't think we can. For mutmut to work the mutants directory must come first in sys.path, otherwise the non-mutated code can be imported first, breaking mutation testing.

That those files have the same same shouldn't be a problem, but maybe you need to run mutmut separately for each? How do you run tests normally?

[StevenAngel-ai]

Normally tests are run one of two ways. Either through the VS Code test discovery or through a script we have that runs coverage as well and works in our CI Pipeline.

The following is how we setup pytest to run

[pytest]
# Reference: https://docs.pytest.org/en/latest/explanation/goodpractices.html#choosing-an-import-mode
addopts =
    -ra
    -q
    --color=yes
    # --import-mode=importlib is recommended for new pytest projects
    --import-mode=importlib
    # pytest-socket: unit-tests by default are not allowed to make remote calls
    --disable-socket
    # pytest-socket: ... but since we use async
    --allow-unix-socket
python_files =
    *_test.py
python_functions =
    test_*
testpaths =
    components
filterwarnings =
    ignore:pkg_resources is deprecated as an API:DeprecationWarning
    ignore::DeprecationWarning:pkg_resources
    ignore::DeprecationWarning:vertexai.preview
norecursedirs=
    tmp/*

Running mutmut just in our common folder, which does not have any duplicate names, does seem to avoid the issue, presuming the following is unconnected.

Traceback (most recent call last):
  File "/Users/sangel/Code/autobot/.venv/bin/mutmut", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/Users/sangel/Code/autobot/.venv/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sangel/Code/autobot/.venv/lib/python3.12/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/Users/sangel/Code/autobot/.venv/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sangel/Code/autobot/.venv/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sangel/Code/autobot/.venv/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sangel/Code/autobot/.venv/lib/python3.12/site-packages/mutmut/__main__.py", line 1216, in run
    create_mutants()
  File "/Users/sangel/Code/autobot/.venv/lib/python3.12/site-packages/mutmut/__main__.py", line 219, in create_mutants
    create_mutants_for_file(path, output_path)
  File "/Users/sangel/Code/autobot/.venv/lib/python3.12/site-packages/mutmut/__main__.py", line 255, in create_mutants_for_file
    mutant_names, hash_by_function_name = write_all_mutants_to_file(out=out, source=source, filename=filename)
                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sangel/Code/autobot/.venv/lib/python3.12/site-packages/mutmut/__main__.py", line 299, in write_all_mutants_to_file
    for type_, x, name_and_hash, mutant_name in yield_mutants_for_module(ast, no_mutate_lines):
                                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sangel/Code/autobot/.venv/lib/python3.12/site-packages/mutmut/__main__.py", line 575, in yield_mutants_for_module
    yield from yield_mutants_for_class(child_node, no_mutate_lines=no_mutate_lines)
  File "/Users/sangel/Code/autobot/.venv/lib/python3.12/site-packages/mutmut/__main__.py", line 531, in yield_mutants_for_class
    yield from yield_mutants_for_class_body(child_node, no_mutate_lines=no_mutate_lines)
  File "/Users/sangel/Code/autobot/.venv/lib/python3.12/site-packages/mutmut/__main__.py", line 542, in yield_mutants_for_class_body
    yield from yield_mutants_for_function(child_node, class_name=class_name, no_mutate_lines=no_mutate_lines)
  File "/Users/sangel/Code/autobot/.venv/lib/python3.12/site-packages/mutmut/__main__.py", line 497, in yield_mutants_for_function
    with rename_function_node(node, suffix='orig', class_name=class_name):
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sangel/.pyenv/versions/3.12.6/lib/python3.12/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/Users/sangel/Code/autobot/.venv/lib/python3.12/site-packages/mutmut/__main__.py", line 346, in rename_function_node
    mangled_name = mangle_function_name(name=orig_name, class_name=class_name)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sangel/Code/autobot/.venv/lib/python3.12/site-packages/mutmut/__main__.py", line 785, in mangle_function_name
    assert CLASS_NAME_SEPARATOR not in name