Add a feature with mutating code changed by a patch

[ligurio]

Currently there is only a one supported scenario to run mutation testing with Mull -- run it on a whole source codebase. It's nice for case when we want to do it once to know how good code coverage in a project. For regular use it is not practical and would be better to support mutations limited by source code in patch. Imagine you are a developer in a project and you made a patch with some functionality and tests that covers that functionality. You want to know how good these tests covers your patch.

[stanislaw]

@ligurio just a small update here: we are working on this feature with mid priority. Alex and I need to hammer out all the details: how exactly this integrates to the existing code and how the CLI interface should look like. The estimated time: 2-4 more weekends :)

Regarding the CLI interface: do you have immediate thoughts on how the corresponding CLI command would look like to tell Mull to only work against a diff changeset? From our point of view, a single CLI argument something like -git-diff=git-branch-or-hash would be enough to make Mull work only on the diff between the current project's git state and the git-branch-or-hash (branch or a hashtag can be passed, like with the normal git diff command).

[ligurio]

Alex and I need to hammer out all the details: how exactly this integrates to the existing code and how the CLI interface should look like. The estimated time: 2-4 more weekends :)

Perfect! Thanks for working on this!

From our point of view, a single CLI argument something like -git-diff=git-branch-or-hash would be enough to make Mull work only on the diff between the current project's git state and the git-branch-or-hash (branch or a hashtag can be passed, like with the normal git diff command).

I would not rely on any VCS. I know that Git is mainstream and widely used, but there are other VCS, like Mercurial, Fossil SCM, Subversion, CVS (!) etc and some projects still use them. Moreover you may use source code without using VCS at all and this scenario should be supported too.

AFAIK every VCS can produce a patch for changes (for Git are: git show <hash> or git diff hash1 hash2 and so on) and format of patch is a thing that we can rely on. So option may look like --diff = <path-to-a-file-with-patch or '-' to read patch from stdin>. With such option one could use Mull with any VCS and even with patches generated manually using diff(1).

[stanislaw]

From our point of view, a single CLI argument something like -git-diff=git-branch-or-hash would be enough to make Mull work only on the diff between the current project's git state and the git-branch-or-hash (branch or a hashtag can be passed, like with the normal git diff command).

I would not rely on any VCS. I know that Git is mainstream and widely used, but there are other VCS, like Mercurial, Fossil SCM, Subversion, CVS (!) etc and some projects still use them. Moreover you may use source code without using VCS at all and this scenario should be supported too.

The WIP work is being tracked here: https://github.com/mull-project/mull/pull/833.

I think at this point, we want to start with Git support to cover the majority of cases. Later, when the end-to-end implementation for git diff-based filtering is in place, we could consider supporting other VCS.

AFAIK every VCS can produce a patch for changes (for Git are: git show <hash> or git diff hash1 hash2 and so on) and format of patch is a thing that we can rely on. So option may look like --diff = <path-to-a-file-with-patch or '-' to read patch from stdin>. With such option one could use Mull with any VCS and even with patches generated manually using diff(1).

In the quoted PR, Mull does git diff -U0 under the hood to get a diff from a project's Git root.

We have also discussed the option of reading from files and optionally from stdin (- is a good option to indicate that actually!) and this still has not been decided yet.

[ligurio]

Is it available in nightly builds?

I think at this point, we want to start with Git support to cover the majority of cases. Later, when the end-to-end implementation for git diff-based filtering is in place, we could consider supporting other VCS.

Got it.

[stanislaw]

@ligurio the work is still WIP at the moment, so there's nothing to try just yet.

There is one thing that we have been discussing with Alex:

Let's say you have a function with 50 lines, and you are introducing one new line in that function. The diff is then only one line, and we wonder:

1) Should Mull find mutations only in that single line or 2) It should take the whole function and show all mutations in that function from its 50 lines?

The option 1 seems to be the easiest to implement, the only question: if seeing the mutations of the whole function would not make sense either.

That said, we are going to implement the option 1 but let us know if you have any thoughts on this topic.

[ligurio]

The initial description of the feature was exactly about mutations in source code changed by a patch. So I would limit mutations in added or changed lines only. But I'm afraid that I may miss something. Does changed/added lines in a patch may affect mutations in non-touched source lines? From first sight the answer is no. If you think so too then it's better to mutate only lines not an changed function(s).

[stanislaw]

Hi again,

(yes, we go with the option 1 and see if we need to extend anything in the future)

The branch with this work has been merged to the master branch today from #833. The first documentation is here: https://mull.readthedocs.io/en/latest/IncrementalMutationTesting.html.

Alex has migrated Mull's distribution to Cloudsmith and the nightly builds should be available here: https://cloudsmith.io/~mull-project/repos/mull-nightly/packages/.

If you start testing this, please expect that things go wrong since the feature have received only limited testing so far. I would appreciate critical feedback.

[ligurio]

The branch with this work has been merged to the master branch today from #833. The first documentation is here: mull.readthedocs.io/en/latest/IncrementalMutationTesting.html.

Thanks! I'll to evaluate it on next weekend.

[ligurio]

Seems feature is implemented and issue can be closed

[stanislaw]

Thanks for the heads-up! I still have to add a more detailed description of the git diff options. I will try to do it today, and then we will close it.

By the way, we are looking into adding another mutation engine to work on the AST level instead of IR level. The Clang API ist not very friendly for mutating AST out of the box, but it seems doable. If this proof of concept works, we should be able to support a wider set of mutation operators including those that you suggested in your tickets.

[stanislaw]

I have added the description here: https://github.com/mull-project/mull/pull/860. Let's consider this feature to be ready :)