Strange behaviour with dataclass mutation

[EdgyEdgemond]

I have a dataclass that is mutated (removing the dataclass decorator) which fails tests if applied, but does not get picked up as a handled mutation. I have cleared the cache, and checked tests with the mutation applied (confirming they do in fact fail)

I've replicated a minimal example that I think points to where the problem originates, but not why it originates.

Given the following code and tests, it correctly detects the mutation and it is killed.

code.py

from dataclasses import dataclass

@dataclass
class MyClass:
    a: str
    b: int

tests/test.py

from .. import code

def test_my_class():
    c = code.MyClass(a="str", b=1)

2. Checking mutants
⠇ 4/4  🎉 4  ⏰ 0  🤔 0  🙁 0  🔇 0

But if the dataclass is used in the declaring module, then the mutation is detected as still alive.

code.py

from dataclasses import dataclass

@dataclass
class MyClass:
    a: str
    b: int

c = MyClass(a="str", b=1)

tests/test.py

from .. import code

def test_my_class():
    c = code.MyClass(a="str", b=1)

def test_c():
    assert code.c == code.MyClass(a="str", b=1)

2. Checking mutants
⠹ 4/4  🎉 3  ⏰ 0  🤔 0  🙁 1  🔇 0

$ mutmut show code.py
...
Survived 🙁 (1)

---- code.py (1) ----

# mutant 1
--- code.py
+++ code.py
@@ -1,7 +1,5 @@
 from dataclasses import dataclass

-
-@dataclass
 class MyClass:
     a: str
     b: int

[EdgyEdgemond]

My assumption is it doesn't detect test failure, due to the fact that the tests do not run and in fact fail to be collected (due to the invalid use of a now non dataclass object with no init)

____________________________________ ERROR collecting tests/test.py _____________________________________
tests/test.py:1: in <module>
    from .. import code
code.py:9: in <module>
    c = MyClass("str", 1)
E   TypeError: MyClass() takes no arguments

[EdgyEdgemond]

From local testing I believe this would fix the issue (status code 2 is returned by pytest when test collection fails, despite the documation claiming it occurs when a user cancels the test run), returncode == 0 should also fix it, but can't confirm status 3, 4, 5 aren't use cases for a mutation survival.

def tests_pass(config: Config, callback) -> bool:
    """
    :return: :obj:`True` if the tests pass, otherwise :obj:`False`
    """
    if config.using_testmon:
        copy('.testmondata-initial', '.testmondata')

    use_special_case = True

    # Special case for hammett! We can do in-process test running which is much faster
    if use_special_case and config.test_command.startswith(hammett_prefix):
        return hammett_tests_pass(config, callback)

    returncode = popen_streaming_output(config.test_command, callback, timeout=config.baseline_time_elapsed * 10)
    -return returncode != 1
    +return returncode not in (1, 2)

https://docs.pytest.org/en/latest/reference/exit-codes.html

[boxed]

Ooh. Good catch! I had a recent similarly weird surviving mutant that clearly fails tests that I didn't have time to investigate. I bet it's this thing!

[boxed]

I just released mutmut 3, which is a big rewrite. I believe this issue no longer applies anymore. Feel free to reopen it if it still exists.