muter-mutation-testing / muter

🔎 Automated mutation testing for Swift 🕳️
MIT License
498 stars 39 forks source link

In total, Muter discovered 0 mutants in 0 files #207

Closed paulz closed 1 year ago

paulz commented 3 years ago

 _____       _
|     | _ _ | |_  ___  ___
| | | || | ||  _|| -_||  _|
|_|_|_||___||_|  |___||_|

Automated mutation testing for Swift

You are running version v15
...
2021-07-13 22:15:31.462 xcodebuild[85462:15079600] [MT] IDETestOperationsObserverDebug: 2.171 elapsed -- Testing started completed.
2021-07-13 22:15:31.462 xcodebuild[85462:15079600] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start
2021-07-13 22:15:31.462 xcodebuild[85462:15079600] [MT] IDETestOperationsObserverDebug: 2.171 sec, +2.171 sec -- end
...
+-----------------+
Discovering Swift files which Muter will analyze...
In total, Muter discovered 112 Swift files

...
Long.swift
List.swift
OfMyProjectFiles.swift
...
+-----------------+
Analyzing source files to find mutants which can be inserted into your project...
In total, Muter discovered 0 mutants in 0 files

⚠️ ⚠️ ⚠️ ⚠️ ⚠️  Muter has encountered an error  ⚠️ ⚠️ ⚠️ ⚠️ ⚠️
Muter wasn't able to discover any code it could mutation test.

This is likely caused by misconfiguring Muter, usually by excluding a directory that contains your code.

⚠️ ⚠️ ⚠️ ⚠️ ⚠️  See the Muter error log above this line  ⚠️ ⚠️ ⚠️ ⚠️ ⚠️

If you feel like this is a bug, or want help figuring out what could be happening, please open an issue at
https://github.com/muter-mutation-testing/muter/issues

macOS project with xcodebuild

xcodebuild -version
Xcode 12.5.1
Build version 12E507
paulz commented 3 years ago

also tried installing muter from source, same result also tried running muter on ExampleMacOSApp:


 _____       _
|     | _ _ | |_  ___  ___
| | | || | ||  _|| -_||  _|
|_|_|_||___||_|  |___||_|

Automated mutation testing for Swift

You are running version v15

Want help? Have suggestions? Want to get involved?
 ↳ https://github.com/muter-mutation-testing/muter/issues
+------------------------------------------------+

Copying your project to a temporary directory for testing...
Finished copying your project to a temporary directory for mutation testing

You can find your copied project here:

/var/folders/7y/zdmjf8891x775kpc00pv66kc0000gn/T/TemporaryItems/NSIRD_muter_RSyMJo/ExampleMacOSApp

This directory will also serve as a backup for any XCTest logs that are generated by running your test suite
Running tests with coverage enabled to determine which files to mutate
2021-07-13 22:31:05.575 xcodebuild[88062:15095362] [MT] DVTAssertions: Warning in /Library/Caches/com.apple.xbs/Sources/IDEFrameworks/IDEFrameworks-18212/IDEFoundation/Testing/IDETestRunSpecificationBuilder.m:678
Details:  Failed to compute path to baseline file during test run spec construction: <XCTHTestRunSpecification: 0x7ffad60989a0>
Object:   <IDETestRunSpecificationBuilder>
Method:   +testRunSpecificationsForTestingSpecifiers:scheme:buildables:withBuildParameters:additionalEnvironmentVariables:additionalCommandLineArguments:testRerunPolicy:includeClangProfileParameters:shouldDebugAppExtensions:error:
Thread:   <NSThread: 0x7ffaf5c18d80>{number = 1, name = main}
Please file a bug at https://feedbackassistant.apple.com with this warning message and any useful information you can provide.
2021-07-13 22:31:07.930 xcodebuild[88062:15095362] [MT] DVTLocalComputerLogAspect: generateRuntimeProfileDirectoryPathForApplicationWithBundleIdentifier:runnableLocation:error:: Failed to create directory within application data container because either the app record ({ bundleID: chefski.ExampleMacOSApp, URL: file:///Users/paulz/Library/Developer/Xcode/DerivedData/ExampleMacOSApp-cfmrmpzerfdivobalwgqjrurwpbm/Build/Products/Debug/ExampleMacOSApp.app }) or the data container URL ((null)) was nil; falling back to superclass implementation.
2021-07-13 22:31:08.986 xcodebuild[88062:15095362] [MT] IDETestOperationsObserverDebug: 1.073 elapsed -- Testing started completed.
2021-07-13 22:31:08.986 xcodebuild[88062:15095362] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start
2021-07-13 22:31:08.986 xcodebuild[88062:15095362] [MT] IDETestOperationsObserverDebug: 1.073 sec, +1.073 sec -- end
2021-07-13 22:31:09.334 xccov[88342:15097013] Requested but did not find extension point with identifier Xcode.IDEFoundation.IDEResultKitSerializationConverter
+-----------------+
Discovering Swift files which Muter will analyze...
In total, Muter discovered 1 Swift files

AppDelegate.swift
+-----------------+
Analyzing source files to find mutants which can be inserted into your project...
In total, Muter discovered 0 mutants in 0 files

⚠️ ⚠️ ⚠️ ⚠️ ⚠️  Muter has encountered an error  ⚠️ ⚠️ ⚠️ ⚠️ ⚠️
Muter wasn't able to discover any code it could mutation test.

This is likely caused by misconfiguring Muter, usually by excluding a directory that contains your code.

⚠️ ⚠️ ⚠️ ⚠️ ⚠️  See the Muter error log above this line  ⚠️ ⚠️ ⚠️ ⚠️ ⚠️

If you feel like this is a bug, or want help figuring out what could be happening, please open an issue at
https://github.com/muter-mutation-testing/muter/issues

muter.conf.json:

{
  "arguments" : [
    "-project",
    "ExampleMacOSApp.xcodeproj",
    "-scheme",
    "ExampleMacOSApp",
    "test"
  ],
  "exclude" : [

  ],
  "excludeCalls" : [

  ],
  "executable" : "\/usr\/bin\/xcodebuild"
}

had to fix code signing to run tests:

diff --git a/Repositories/ExampleMacOSApp/ExampleMacOSApp.xcodeproj/project.pbxproj b/Repositories/ExampleMacOSApp/ExampleMacOSApp.xcodeproj/project.pbxproj
index b2f49f4..a1223a6 100644
--- a/Repositories/ExampleMacOSApp/ExampleMacOSApp.xcodeproj/project.pbxproj
+++ b/Repositories/ExampleMacOSApp/ExampleMacOSApp.xcodeproj/project.pbxproj
@@ -346,9 +346,10 @@
            buildSettings = {
                ASSETCATALOG_COMPILER_APPICON_NAME = AppIcon;
                CODE_SIGN_ENTITLEMENTS = ExampleMacOSApp/ExampleMacOSApp.entitlements;
+               CODE_SIGN_IDENTITY = "-";
                CODE_SIGN_STYLE = Automatic;
                COMBINE_HIDPI_IMAGES = YES;
-               DEVELOPMENT_TEAM = H2983HKDVJ;
+               DEVELOPMENT_TEAM = "";
                INFOPLIST_FILE = ExampleMacOSApp/Info.plist;
                LD_RUNPATH_SEARCH_PATHS = (
                    "$(inherited)",
@@ -365,9 +366,10 @@
            buildSettings = {
                ASSETCATALOG_COMPILER_APPICON_NAME = AppIcon;
                CODE_SIGN_ENTITLEMENTS = ExampleMacOSApp/ExampleMacOSApp.entitlements;
+               CODE_SIGN_IDENTITY = "-";
                CODE_SIGN_STYLE = Automatic;
                COMBINE_HIDPI_IMAGES = YES;
-               DEVELOPMENT_TEAM = H2983HKDVJ;
+               DEVELOPMENT_TEAM = "";
                INFOPLIST_FILE = ExampleMacOSApp/Info.plist;
                LD_RUNPATH_SEARCH_PATHS = (
                    "$(inherited)",
@@ -384,9 +386,10 @@
            buildSettings = {
                ALWAYS_EMBED_SWIFT_STANDARD_LIBRARIES = YES;
                BUNDLE_LOADER = "$(TEST_HOST)";
+               CODE_SIGN_IDENTITY = "-";
                CODE_SIGN_STYLE = Automatic;
                COMBINE_HIDPI_IMAGES = YES;
-               DEVELOPMENT_TEAM = H2983HKDVJ;
+               DEVELOPMENT_TEAM = "";
                INFOPLIST_FILE = ExampleMacOSAppTests/Info.plist;
                LD_RUNPATH_SEARCH_PATHS = (
                    "$(inherited)",
@@ -405,9 +408,10 @@
            buildSettings = {
                ALWAYS_EMBED_SWIFT_STANDARD_LIBRARIES = YES;
                BUNDLE_LOADER = "$(TEST_HOST)";
+               CODE_SIGN_IDENTITY = "-";
                CODE_SIGN_STYLE = Automatic;
                COMBINE_HIDPI_IMAGES = YES;
-               DEVELOPMENT_TEAM = H2983HKDVJ;
+               DEVELOPMENT_TEAM = "";
                INFOPLIST_FILE = ExampleMacOSAppTests/Info.plist;
                LD_RUNPATH_SEARCH_PATHS = (
                    "$(inherited)",

is there a better example project to validate muter works?

realtaraharris commented 3 years ago

I'm having the exact same problem on XCode 12.5.1 (12E507) and macOS 11.2.2 (20D80)

rakaramos commented 3 years ago

Hi folks, thanks for reporting this issue! We are having a couple of issues on ci while trying to update some dependencies.

Maybe it's related to that, but I'll double check anyway and let you folks know of any news.

realtaraharris commented 3 years ago

What is the latest version of XCode that works properly with muter? I switched back to 12.1 (12A7403), and that works, but at the price of the many annoying bugs in the IDE.

I think it would be helpful to have this information in the README so that people know not to try using this with the latest XCode...

rakaramos commented 3 years ago

Hi @maxharris9, thanks for diving into this issue! I'm swamped at work in last couple of months so I had no time to dedicate to Muter. Feel free to issue an PR with these changes, I'll be more than happy to merge it in!

ws233 commented 3 years ago

Any updates on this bug? Muter doesn't work with Xcode 12.3 either.

itsmeabhinay commented 3 years ago

Any updates on this bug, I downgraded to Xcode 12.1 but still see the same issue.

2021-10-20 08:46:40.578 xcodebuild[82723:8863454] [MT] IDETestOperationsObserverDebug: (C69EE629-B584-4A08-9126-F8CAF6A7E0E9) Finished requesting crash reports. Continuing with testing.
2021-10-20 08:46:44.786 xcodebuild[82723:8863454] [MT] IDETestOperationsObserverDebug: 93.055 elapsed -- Testing started completed.
2021-10-20 08:46:44.786 xcodebuild[82723:8863454] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start
2021-10-20 08:46:44.786 xcodebuild[82723:8863454] [MT] IDETestOperationsObserverDebug: 93.055 sec, +93.055 sec -- end
Gathering coverage failed.
Proceeding with mutation testing anyway.
Pass --skip-coverage argument to disable this step
+-----------------+
Discovering Swift files which Muter will analyze...
In total, Muter discovered 1347 Swift files

+-----------------+
Analyzing source files to find mutants which can be inserted into your project...
In total, Muter discovered 0 mutants in 0 files

⚠️ ⚠️ ⚠️ ⚠️ ⚠️  Muter has encountered an error  ⚠️ ⚠️ ⚠️ ⚠️ ⚠️
Muter wasn't able to discover any code it could mutation test.

This is likely caused by misconfiguring Muter, usually by excluding a directory that contains your code.

⚠️ ⚠️ ⚠️ ⚠️ ⚠️  See the Muter error log above this line  ⚠️ ⚠️ ⚠️ ⚠️ ⚠️

If you feel like this is a bug, or want help figuring out what could be happening, please open an issue at
https://github.com/muter-mutation-testing/muter/issues
ws233 commented 3 years ago

@itsmeabhinay It seems like you've still been using debug tools from Xcode 13. Pls, check what xcode-select -p points to. If it points to your Xcode 13, pls use xcode-select -s <path/to/your/Xcode 12.1>

itsmeabhinay commented 3 years ago

@itsmeabhinay It seems like you've still been using debug tools from Xcode 13. Pls, check what xcode-select -p points to. If it points to your Xcode 13, pls use xcode-select -s <path/to/your/Xcode 12.1>

Do we know when Muter will be compatible with latest Xcode versions?

rakaramos commented 3 years ago

@itsmeabhinay I've just updated a handful of dependencies, could you folks please check if its working?

itsmeabhinay commented 2 years ago

Thank you @rakaramos. Also, is there anyway to add a new argument include similar to exclude. This will help us test only specific files related to a specific feature.

rakaramos commented 2 years ago

hi @itsmeabhinay, maybe you are looking for this?

itsmeabhinay commented 2 years ago

@rakaramos, not exactly. I am looking to generate a mutation score only for specific files. I see we have exclude in the Configuration Options, was wondering if we can add include so that I can just run it on some files. I thought including 20 files would be easier than excluding 200 files.

YuriiSeals commented 2 years ago

@rakaramos are you planning to update the project for latest XCode?

ZevEisenberg commented 2 years ago

We were just talking about finding some time to update it. No promises, but we'd certainly like to!

grvermeulen commented 2 years ago

That would be excellent! Really would like to use it with the latest Xcode! Thanks for the effort!

ArthurBrum commented 2 years ago

I don't know why, but for me I found this error only when I installed muter using brew. When I retried installing with git clone and make method, it worked! Let me know if this work for you guys :)

rakaramos commented 2 years ago

thanks for letting us know @ArthurBrum, we'll try to fix that on the next release!