Identified issues

[felipecp]

Main problems

We started implementing the mutation tests on a specific iOS module in our project. Tried running for the whole project and a small subset of files to analyze performance.

High disk usage Since this tool makes mutations and runs all the tests for the result, it makes copies of the project for every mutation during its run. This way, disk usage frequently got filled up with temp files. The run that executed the tests for the whole module, used around 200gb of hard drive.

Time to run is too long The best time was 4 hours and 27 minutes for the mutation test running on specific module repository. As this is not acceptable to run on each CI validation, our plan was to schedule a pipeline to run weekly and improve our test codes later based on mutation scores.

The compatibility is not good depending on Xcode version Some users reported problems with different versions of xcode and maintainers oriented them to run with a specific older version (12.1) but even doing so, users reported errors. This issue was created in JUL/2021 and so far it is still not solved. We managed to make it run for Xcode 13.2.1, but the risk here is that we may sometime in the future use a more updated version and mutation tests can end up impossible to run.

Have you guys identified these issues? Do you have some suggestions to fix them?

[rakaramos]

Hi @felipecp! I'll try to answer your questions

High disk usage This happens because we basically need to create copies of file for each mutant. To fix that we need to discover mutants dynamically with subsumption relations, which we, unfortunately don't have at the moment.

Time to run is too long Mutation tests are known for being slow. One thing you could do is filter (using --files-to-mutate or skip branches of code. This PR can also improve the overall run time.

The compatibility is not good depending on Xcode version It happens because we use a weak link to the SwiftSyntax dylib, that is bundle within Xcode. If that changes, it will break muter.

[ZevEisenberg]

We could probably delete copies of the project after we're done with them, right? I forget whether each mutation is a full new copy of the project, or whether it's more efficient than that. Also, note that APFS means that you're likely not actually incurring all that disk space, since you're only paying the cost of new files for ones that you duplicate. (For a large project with many small files, the cost of the directory structure is probably nontrivial…but probably not 200GB either.)

[rakaramos]

I need to double check, but I think we apply the mutation, run the tests and undo it.

@felipecp do you happen to know if that's on the temp folder? Or maybe it's the logs that are generated on each mutation round?

[ArthurBrum]

Hi there, I'm with @felipecp. In fact they are in a temp folder, but I cannot find out why muter (or xcode) is not deleting them.

The folders that occupied these 200gb are inside this path: /var/folders/k2

Muter found only 366 mutants but I see about 13.000 folders like:

(...)
com.apple.CoreSimulator.SimDevice.C41A0642-BD5D-435A-96B5-7E4A5D60DCE9.Standalone.FFFFFFFFF-2C5B-46DD-9D04-B7642C705017
com.apple.CoreSimulator.SimDevice.C41A0642-BD5D-435A-96B5-7E4A5D60DCE9.Standalone.FFFFFFFFF-3265-40BC-9A40-0001643B451E
com.apple.CoreSimulator.SimDevice.C41A0642-BD5D-435A-96B5-7E4A5D60DCE9.Standalone.FFFFFFFFF-A582-4633-9E1A-AA2DC73371A0
com.apple.CoreSimulator.SimDevice.C41A0642-BD5D-435A-96B5-7E4A5D60DCE9.Standalone.FFFFFFFFF-B6DD-4A6B-BC8A-EC74AA88544F
(...)

I'm not sure if these are only outputs of compilation but wouldn't you expect xcode to clean them? or should muter code call for this clean manually? (and btw, just so you can estimate the size of the project to see if these 200GB makes any sense... I'm already ignoring some parts of the source code and yet it took muter 4h27min to run all mutations)

[ZevEisenberg]

I believe those folders are cleaned up by the OS on reboot, but muter should totally do a better job of cleaning up after itself. It could even potentially do it in the background so it doesn't interfere with the currently running tests.

Alternatively, it could do all mutations and reversions in-place instead of making a new copy for each change. I feel like it maybe even used to do this, and was removed for some reason? Or at least we had talked about it.

[ArthurBrum]

Hi @ZevEisenberg and @rakaramos

Do you think I can create any kind of shell script that could delete these folders once they are done being used by muter? The main problem I see is that I don't know how to make sure which ones xcode is still using and which ones can be deleted...

Ideas I have: 1) Each 10 minutes the service would delete the folders in this path older than 2 min 2) The service would somehow track xcode new builds (from logs or any other instruments tool) and delete olders once a new one is generated

Problems I see in these solutions: 1) Will this path be the same for all the runs of muter? Is 2 minutes really enough for muter to conclude it's work, I mean, how to make sure I'm not deleting files that are still being used? 2) No idea how to track this

[ZevEisenberg]

I think the right fix for this will be to update muter itself to delete old files, possibly with a flag to keep them around for debugging purposes. The folders are generated by the OS via a call to NSTemporaryDirectory() (I think), and it would be hard to know outside of muter which ones are no longer in use.