Muter hangs when mutation change causes test hang

aim2120 commented 3 months ago

Hi all, sorry for all the recent posts, but I wanted to report another issue I'm observing when running muter.

Sometimes, muter will cause a certain async code path to not be fulfilled and cause a test hang. An example of this is when using withCheckedContinuation and the side effect of calling the continuation resume is removed.

I would love to be able to pass -test-timeouts-enabled and -maximum-test-execution-time-allowance to the xcodebuild command via the muter config, but muter appears to manipulate the test-without-building arguments such that I cannot pass additional arguments like this.

Would you be open to a PR to allow for additional arbitrary arguments to be used for test-without-building? Another option would be to implement timeouts directly in the muter config.

Nikoloutsos commented 3 months ago

Hello @aim2120 and thank you for reporting this 🙌

Have you tried executionTimeAllowance?

In addition, as a developer, I would like an opt-in way to force muter to exclude specific code from mutation. I imagine it like the following:

await withCheckedContinuation { continuation in
        fetchMessages { messages in
            // muter: skip
            continuation.resume(returning: messages)
        }
    }

aim2120 commented 3 months ago

Hey @Nikoloutsos thanks for the response! Unfortunately executionTimeAllowance doesn't scale reasonably well for me. I'm working in a codebase with many other developers and trying to enable mutation testing for all of them, without needing to change all of their test code.

Honestly, the best solution I see is to allow muter to enable timeouts itself via config. That way muter can guarantee that it will complete, regardless of the behavior of the test executable.

I was trying some stuff out locally and it seems -test-timeouts-enabled and -maximum-test-execution-time-allowance aren't a great solution, since they automatically retry multiple times (I haven't found a way to disable this retry behavior using xcodebuild).

rakaramos commented 2 months ago

hi @aim2120, could you provide a brief code snippet where I can reproduce the issue and check if the flags are enough? In the mean time, you can disable the operators from being injected into this part of the code

aim2120 commented 2 months ago

Sure, here's an SPM library that hangs when you run muter. This is because the mutation applied removes the line continuation.resume.

MuterHangingExample.zip

So there's actually a few negatives about the flags -test-timeouts-enabled and -maximum-test-execution-time-allowance that I didn't realize when I wrote the initial post:

They only work if the test executable is xcodebuild, so we can't use with swift
They will automatically retry the test, so that the test has to timeout multiple times before xcodebuild fails. This causes muter to take much much longer, since a timeout of 3 minutes for example becomes (3 * # of retries) minutes.

How would you feel about introducing timeouts to muter itself? This could be configured via the muter config, with a default value of nil if none is found, for backwards compatibility. This would mean that we can control the maximum test time, regardless of whether we're using xcodebuild or swift (or anything else), and doesn't require any fidgeting around with the limitations of the command executable.

aim2120 commented 2 months ago

Oh, and just to reiterate what I said in my previous comment: I'm hoping to unlock using muter across many projects contributed to by multiple developers, so it's not feasible for me to figure out exactly which lines of code are potential hang points for mutations.

muter-mutation-testing / muter

Muter hangs when mutation change causes test hang #279