search

muter-mutation-testing / muter

🔎 Automated mutation testing for Swift 🕳️
MIT License
496 stars 39 forks source link

AcceptanceTests fail locally when using Sonoma 14.4.1 + Xcode 15.2 #277

Closed aim2120 closed 5 months ago

aim2120 commented 6 months ago

Hi all, just wanted to report that when running the AcceptanceTests using make acceptance-test as described on the master branch, I get quite a few failures. I'm running using Sonoma 14.4.1 and Xcode 15.2.

I've discovered the following issues that occur while running muter on the ExampleApp project:

  1. Error Domain=NSCocoaErrorDomain Code=260 "The file “Products” couldn’t be opened because there is no such file."
  2. Swift/RangeReplaceableCollection.swift:867: Fatal error: Can't remove last element from an empty collection

The first error occurs when running off master as-is. It seems the that the Products directory doesn't exists when at the point when it's expected to.

The second error occurs when I make some adjustments to get past the first error. It seems that when attempting to get coverage using llvm-cov, the --ignore-filename-regex='.build|Tests' flag happens matches all file paths, because they're located in path that contains AcceptanceTests. Since no percentages are listed, the line percents.removeLast() crashes due to being an empty array.

I'm guessing these errors are the result of new behavior in Xcode 15+, since there seem to be no existing issues related to these errors. I'd be more than happy to open a PR to fix both of these issues. Let me know if you have any comments/suggestions.

aim2120 commented 6 months ago

Actually, turns out the missing Products issue was related to which simulator was used.

I've opened PR #278 to fix this issue.