search

kateinoigakukun / mull-xctest

Experimental mutation testing tool for Swift and XCTest powered by mull
MIT License
45 stars 3 forks source link

`No mutants found. Mutation score: infinitely high` error #7

Closed ws233 closed 3 years ago

ws233 commented 3 years ago

I have a swift packet with UIKit import. Unfortunately there is no way to build it using swift build. That's why tried 3 different ways to use mull-xctests.

  1. I generate a project from a SPM with swift package generate-xcodeproj. Then I build it with xcodebuild build-for-testing -project PSBCore.xcodeproj -scheme PSBCore-Package -destination "platform=iOS Simulator,name=iPhone 11 Pro" OTHER_SWIFT_FLAGS="\$(inherited) -embed-bitcode" OTHER_CFLAGS="\$(inherited) -fembed-bitcode" LD="$(which mull-ld)" OTHER_LDFLAGS="\$(inherited) -Xmull --linker=clang -Xmull --linker-flavor=clang". It succeed. Then when I run mull-xctestrun -Xxcodebuild -destination -Xxcodebuild "platform=iOS Simulator,name=iPhone 11 Pro" --test-target PSBCoreTests /path/to/Library/Developer/Xcode/DerivedData/PSBCore-btiytljcqmsisacjtlzuldouejsz/Build/Products/PSBCore-Package_iphonesimulator15.0-x86_64.xctestrun and get [info] No mutants found. Mutation score: infinitely high
  2. I even tried to test the project if the command similar to above, but if test instead of building-for-testing. After that I tried mull-xctest /path/to/Library/Developer/Xcode/DerivedData/PSBCore-btiytljcqmsisacjtlzuldouejsz/Build/Products/Debug-iphonesimulator/PSBCoreTests.xctest. It also gives me following: 
    [info] Warm up run (threads: 1)
   [################################] 1/1. Finished in 45ms
[info] Baseline run (threads: 1)
   [################################] 1/1. Finished in 41ms
[info] No mutants found. Mutation score: infinitely high
  3. I've also tried to build a SPM with the command from the 1st, but without generating a Project file. This gives me an error: 
    x86_64-apple-ios11.0-simulator: No such file or directory
Command Ld failed with a nonzero exit code

    why is there x86_64-apple-ios11.0-simulator? I'm using Xcode 13.0, which doesn't seem to have ios11 simulator.

What I'm doing wrong? Pls, help me.

kateinoigakukun commented 3 years ago

I can't advise you more without seeing the actual project, but if your project doesn't include any primitive operations, like + for Int, your project doesn't have any mutable points.

And also make sure that your testing-target is linked into the test target statically.

ws233 commented 3 years ago

It definitely has primitive operations. I used https://github.com/muter-mutation-testing/muter this tools for checking.

It seems it's linked dynamically. Thanks.

How can I link it statically from SPM? Could you propose you test example, which was used to record demo video. StubKitTests.

kateinoigakukun commented 3 years ago

The demo used https://github.com/kateinoigakukun/StubKit

ws233 commented 3 years ago

I've tried with your test project and got the same result

swift build --build-tests \
    -Xswiftc -embed-bitcode \
    -Xswiftc -tools-directory -Xswiftc /usr/local/opt/mull-xctest/bin/mull-tools
<unknown>:0: error: error opening input file '/var/folders/wg/4pfx8wbs12n8kqk1pzsrzgr00000gn/T/TemporaryDirectory.l3yvXI/BuiltinStubProvider-1.bc' (No such file or directory)
<unknown>:0: error: error opening input file '/var/folders/wg/4pfx8wbs12n8kqk1pzsrzgr00000gn/T/TemporaryDirectory.l3yvXI/EnumStubProvider-1.bc' (No such file or directory)
<unknown>:0: error: error opening input file '/var/folders/wg/4pfx8wbs12n8kqk1pzsrzgr00000gn/T/TemporaryDirectory.l3yvXI/StubbableProvider-1.bc' (No such file or directory)
<unknown>:0: error: error opening input file '/var/folders/wg/4pfx8wbs12n8kqk1pzsrzgr00000gn/T/TemporaryDirectory.l3yvXI/StubKit-1.bc' (No such file or directory)
<unknown>:0: error: error opening input file '/var/folders/wg/4pfx8wbs12n8kqk1pzsrzgr00000gn/T/TemporaryDirectory.l3yvXI/StubProvider-1.bc' (No such file or directory)
<unknown>:0: error: error opening input file '/var/folders/wg/4pfx8wbs12n8kqk1pzsrzgr00000gn/T/TemporaryDirectory.l3yvXI/Stubbable-1.bc' (No such file or directory)
<unknown>:0: error: error opening input file '/var/folders/wg/4pfx8wbs12n8kqk1pzsrzgr00000gn/T/TemporaryDirectory.l3yvXI/Number-1.bc' (No such file or directory)
<unknown>:0: error: error opening input file '/var/folders/wg/4pfx8wbs12n8kqk1pzsrzgr00000gn/T/TemporaryDirectory.l3yvXI/Injector-1.bc' (No such file or directory)
<unknown>:0: error: error opening input file '/var/folders/wg/4pfx8wbs12n8kqk1pzsrzgr00000gn/T/TemporaryDirectory.l3yvXI/Decoders-1.bc' (No such file or directory)
<unknown>:0: error: error opening input file '/var/folders/wg/4pfx8wbs12n8kqk1pzsrzgr00000gn/T/TemporaryDirectory.l3yvXI/Misc-1.bc' (No such file or directory)
<unknown>:0: error: error opening input file '/var/folders/wg/4pfx8wbs12n8kqk1pzsrzgr00000gn/T/TemporaryDirectory.l3yvXI/String-1.bc' (No such file or directory)
[23/24] Merging module StubKit

mull-xctest .build/debug/StubKitPackageTests.xctest
[info] Warm up run (threads: 1)
       [################################] 1/1. Finished in 178ms
[info] Baseline run (threads: 1)
       [################################] 1/1. Finished in 36ms
[info] No mutants found. Mutation score: infinitely high

Do you have any ideas what could be wrong?

ws233 commented 3 years ago

Furthermore I've found the way to build SPM with UIKit using swift build with the following command:

swift build --build-tests -Xswiftc -embed-bitcode -Xswiftc -tools-directory -Xswiftc /usr/local/opt/mull-xctest/bin/mull-tools  -Xswiftc "-sdk" -Xswiftc "`xcrun --sdk iphonesimulator --show-sdk-path`" -Xswiftc "-target" -Xswiftc "x86_64-apple-ios14.0-simulator"

and it also produces a lot of error output of *.bc files

<unknown>:0: error: error opening input file '/var/folders/wg/4pfx8wbs12n8kqk1pzsrzgr00000gn/T/TemporaryDirectory.LqHETI/FileProviderProtocol-1.bc' (No such file or directory)
<unknown>:0: error: error opening input file '/var/folders/wg/4pfx8wbs12n8kqk1pzsrzgr00000gn/T/TemporaryDirectory.LqHETI/TempFileProvider-1.bc' (No such file or directory)
<unknown>:0: error: error opening input file '/var/folders/wg/4pfx8wbs12n8kqk1pzsrzgr00000gn/T/TemporaryDirectory.LqHETI/KeychainDataStoring-1.bc' (No such file or directory)
<unknown>:0: error: error opening input file '/var/folders/wg/4pfx8wbs12n8kqk1pzsrzgr00000gn/T/TemporaryDirectory.LqHETI/KeychainStorageProtocol-1.bc' (No such file or directory)
<unknown>:0: error: error opening input file '/var/folders/wg/4pfx8wbs12n8kqk1pzsrzgr00000gn/T/TemporaryDirectory.LqHETI/Array+SafeIndex-1.bc' (No such file or directory)
<unknown>:0: error: error opening input file '/var/folders/wg/4pfx8wbs12n8kqk1pzsrzgr00000gn/T/TemporaryDirectory.LqHETI/ArrayExtensions-1.bc' (No such file or directory)
<unknown>:0: error: error opening input file '/var/folders/wg/4pfx8wbs12n8kqk1pzsrzgr00000gn/T/TemporaryDirectory.LqHETI/CharacterSetExtension-1.bc' (No such file or directory)
<unknown>:0: error: error opening input file '/var/folders/wg/4pfx8wbs12n8kqk1pzsrzgr00000gn/T/TemporaryDirectory.LqHETI/CollectionExtension-1.bc' (No such file or directory)

and in the end there is [249/251] Merging module PSBCore

kateinoigakukun commented 3 years ago

Thank you for reporting. From my investigation, the latest swift frontend changed bitcode output file path, and it broke the mull-xctest assumption. Xcode 12.4's toolchain 5.3.2 is compatible with the latest mull-xctest, but Swift 5.5 is still incompatible. If I can take time, I'll repair this.

ws233 commented 3 years ago

Thanks for your quick answer. Finally I could succeed with Xcode 12.1.