muter-mutation-testing / muter

🔎 Automated mutation testing for Swift 🕳️
MIT License
491 stars 39 forks source link

Muter discovered 0 mutants in 0 files or wrongfully fails tests when running on a new project with default config #260

Closed adgifs closed 2 months ago

adgifs commented 6 months ago

I have created a new app project with SwiftUI and tried to run muter on it without success.

Automated mutation testing for Swift

You are running version 16

Want help? Have suggestions? Want to get involved?
 ↳ https://github.com/muter-mutation-testing/muter/issues
+------------------------------------------------+

🔎 Checking for new versions...
✅ You are already using the latest of Muter
Loaded config at path: /Users/André/Documents/MutationTesting/MutationTestIntroduction/muter.conf.yml
Copying your project to a temporary directory for testing...
Finished copying your project to a temporary directory for mutation testing.
You can find your copied project here:

/Users/André/Documents/MutationTesting/MutationTestIntroduction_mutated

This directory will also serve as a backup for any XCTest logs that are generated by running your test suite.
Running tests with coverage enabled to determine which files to mutate
+-----------------+
Discovering Swift files which Muter will analyze...
In total, Muter discovered 0 Swift files

⚠️ ⚠️ ⚠️ ⚠️ ⚠️  Muter has encountered an error  ⚠️ ⚠️ ⚠️ ⚠️ ⚠️
Muter wasn't able to discover any code it could mutation test.

This is likely caused by misconfiguring Muter, usually by excluding a directory that contains your code.

⚠️ ⚠️ ⚠️ ⚠️ ⚠️  See the Muter error log above this line  ⚠️ ⚠️ ⚠️ ⚠️ ⚠️

If you think this is a bug, or want help figuring out what could be happening, please open an issue at
https://github.com/muter-mutation-testing/muter/issues

If relevant, I'm running it with the clone of the repository but I have tried homebrew and with no success either. Besides that, I'm running it with the following Xcode version:

Xcode 15.0.1
Build version 15A507

Configuration:

arguments:
- -project
- MutationTestIntroduction.xcodeproj
- -scheme
- MutationTestIntroduction
- -destination
- platform=iOS Simulator,name=iPhone SE (3rd generation)
- test
executable: /usr/bin/xcodebuild
exclude: []
excludeCalls: []

My configuration file is the generated one and it isn't excluding anything. Running one of the apps in the repository does work unlike in mine. The same happened in other projects of mine that I have tried to run it. Curiously, if I use muter --files-to-mutate MutationTestIntroduction/Extensions/IntExtension.swift, it does pick up the file and runs successfully. What's the issue exactly here? Thank you in advance and happy holidays!

Edit:

Curiously, if I test the IntExtension.swift with muter --files-to-mutate MutationTestIntroduction/Extensions/IntExtension.swift it fails, even though the unit test is solid. If I add that file to the ExampleApp, which I'm able to work with the command muter alone, it tests and passes my test. In fact, every mutation test passes because of runtime error.

// IntExtension.swift

extension Int {
    var isGreaterOrEqualThanTwo: Bool {
        self >= 2
    }
}

// IntExtensionTests.swift

final class IntExtensionTests: XCTestCase {
    func testIsGreaterOrEqualThanTwo() {
        XCTAssertTrue(3.isGreaterOrEqualThanTwo)
        XCTAssertTrue(2.isGreaterOrEqualThanTwo)
        XCTAssertFalse(1.isGreaterOrEqualThanTwo)
    }
}

I have tried to create yet another base project, now with a slightly different setup and, even though it is able to detect IntExtension.swift, the mutation survives, even though it should pass and it does pass when using ExampleApp. I'm very confused... 🫠

Nikoloutsos commented 5 months ago

Hello @adgifs 👋 That's a nice observation :) Up until my latest commit d44e78d0faf1dd2a2ebde92b7a8159873ef39c97 muter was ignoring any files that contained the word test in the path. So since you named your project MutationTestIntroduction I think it ignores everything. After that commit it is fixed and should be working as expected. (Please read the README.md for more information on how muter exclude files)

At this moment the latest release version does not include that. So your only option is to manually clone the project at Master branch and install it.

@rakaramos Do you think we can release a new version just for that?

rakaramos commented 4 months ago

@Nikoloutsos I think I'll have more time in a few weeks to prepare for a new release 👍

rakaramos commented 4 months ago

@adgifs does muter build from the master branch fixes the issue for you?