search

trailofbits / necessist

A mutation-based tool for finding bugs in tests
https://crates.io/crates/necessist
GNU Affero General Public License v3.0
109 stars 10 forks source link

Incremental Testing based on `git-diff` #958

Open ASuciuX opened 8 months ago

ASuciuX commented 8 months ago

Hello @smoelius, @moodmosaic told me about this and found it quite intriguing and a great complement to the concept of mutants!

Currently, I'm focusing on incorporating mutation testing into Stacks. This led me to ponder how we could integrate this into our entire workflow. Similar to our approach with mutants, I see two primary applications for this in a large-scale project:

  1. Ensuring that all new tests are thoroughly vetted before their integration into the production environment.
  2. Identifying and improving existing tests that aren't up to the mark.

Given the extensive size of the stacks repository, we've created a specific process that runs with each pull request (PR) to indicate if the functions are inadequately tested, utilizing cargo-mutants. It's noteworthy that cargo-mutants already includes certain custom features for incremental runs, such as testing only those mutants associated with functions modified in a git-diff.

I'm curious if a similar approach could be applied to necessist, performing the operations only on modified tests based on a git-diff.

I'd greatly appreciate your thoughts on this and any feedback you might have. Also, thanks for your contributions to the field!

ASuciuX commented 8 months ago

The requirement can be divided into two distinct tasks:

  1. Execute a specific subset of tests from a test file, identified by their unique test names. 
    
necessist stackslib/src/chainstate/stacks/boot/pox_4_tests.rs \ 
-- --package stackslib chainstate::stacks::boot::pox_4_tests \
[list of test names]

2. Extract the list of test names from the changes highlighted in the `git-diff` file.

Are there any particular details or considerations related to these tasks?
smoelius commented 8 months ago

Hi, @ASuciuX.

I'm curious if a similar approach could be applied to necessist, performing the operations only on modified tests based on a git-diff.

Necessist doesn't currently have this functionality. But I agree it could be worthwhile to have something like this built in.

It sounds like you are considering scripting something together to do this. If you do, I'd be curious to hear about your experience.

Just to state the obvious: a change to a function called by a test could affect Necessist's results. The described approach wouldn't catch such changes. But I don't think that detracts from the idea's merits.

Sorry for being slow to reply. Your question got me thinking about how best to incorporate Necessist into a CI workflow, generally. I don't currently have a good answer to that. But that fact should not have delayed my response. Apologies.