stryker-mutator / stryker-net

Mutation testing for .NET core and .NET framework!
https://stryker-mutator.io
Apache License 2.0
1.75k stars 176 forks source link

Exclude folder path from mutation in stryker-config #2795

Closed gentcod closed 1 month ago

gentcod commented 7 months ago

I am trying to exclude DB migrations from mutation.

The code for the DB migrations is auto generated by Entity Framework Core and it makes no sense to include them in the mutation as there are a lot of hard coded string values. It gets the mutants that are not covered.

This is a sample of code generated that needs to be excluded: migrationBuilder.CreateTable( name: "RecipeRatings", columns: table => new { Id = table.Column<int>(type: "INTEGER", nullable: false) .Annotation("Sqlite:Autoincrement", true), RecipeId = table.Column<int>(type: "INTEGER", nullable: false), RatingNum = table.Column<int>(type: "INTEGER", nullable: false), Comment = table.Column<string>(type: "TEXT", nullable: true) },

rouke-broersma commented 7 months ago

Did you try the mutate option?

gentcod commented 7 months ago

I excluded the methods using "ignore-methods" but the strings and boolean values are being highlighted for mutation, categorized under No Coverage

I have about 100+ mutants being highlighted from just that file.

rouke-broersma commented 7 months ago

Ignore methods is not the same as mutate. Try the mutate option. https://stryker-mutator.io/docs/stryker-net/configuration/#mutate-glob

dupdob commented 7 months ago

I am not familiar with EF inner workings, but Stryker normally does not mutate generated code. To be specific, it does not mutate build time generated code, plus a bunch of well known patterns.

gentcod commented 7 months ago

It doesn't but instead of ignoring the contents, it categorizes it as mutants that are not covered.

Not Covered mutants fall are regarded as survived mutants, so still affecting the mutation score

I noticed it ignored 2 other files in the same directory though; EF Core generates 3 files when a migration is initialized.

rouke-broersma commented 7 months ago

I am not familiar with EF inner workings, but Stryker normally does not mutate generated code. To be specific, it does not mutate build time generated code, plus a bunch of well known patterns.

Iirc ef doesn't consider it generated code because you should always manually check and modify migrations, so they refuse to place the generated code attribute.

rouke-broersma commented 7 months ago

It doesn't but instead of ignoring the contents, it categorizes it as mutants that are not covered.

Not Covered mutants fall are regarded as survived mutants, so still affecting the mutation score

I noticed it ignored 2 other files in the same directory though; EF Core generates 3 files when a migration is initialized.

It should be marked as ignored when excluding using the mutate option. Could you share your full configuration or your debug log file?

Are you using since or baseline?

gentcod commented 7 months ago

I am not familiar with EF inner workings, but Stryker normally does not mutate generated code. To be specific, it does not mutate build time generated code, plus a bunch of well known patterns.

Iirc ef doesn't consider it generated code because you should always manually check and modify migrations, so they refuse to place the generated code attribute.

Well, yeah I understand but there are still a lot of hard coded values that could be mutated.

gentcod commented 7 months ago

This is the configuration I have used so far: { "stryker-config": { "ignore-methods": [ "*migrationBuilder.CreateTable", "*migrationBuilder.CreateIndex", "*migrationBuilder.DropTable", "*Column", "*Annotation", "*PrimaryKey", "*ForeignKey" ], "thresholds": { "high": 80, "low": 60, "break": 0 }, "reporters": [ "progress", "html", "cleartext" ] } }

richardwerkman commented 7 months ago

@gentcod The option you are using won't work in this case. ignore-methods will only ignore the invocations of those methods. Not the declarations. So the behaviour you're seeing is by design. In this case you could use stryker comments or the mutate option.

See: https://stryker-mutator.io/docs/stryker-net/ignore-mutations/ for an extensive guide on how to ignore mutations.

gentcod commented 7 months ago

@richardwerkman oh I get but in cases where there are strings in other test suites, that implies that they would be overlooked unless there is a way for me to ignore specific strings. If there is, please do let me know, thank you.

gentcod commented 7 months ago

I read the documentation and saw exclude-mutation referenced on the page for ignore-mutations docs but it's not available on the configuration page.

rouke-broersma commented 7 months ago

I read the documentation and saw exclude-mutation referenced on the page for ignore-mutations docs but it's not available on the configuration page.

Could you link to these docs?

gentcod commented 7 months ago

https://stryker-mutator.io/docs/stryker-net/mutations/

rouke-broersma commented 1 month ago

I'm updating those docs, it should not be called exclude-mutations.

As for your original issue, the method to use to ignore those declarations is to use stryker comments. See: https://stryker-mutator.io/docs/stryker-net/ignore-mutations/#stryker-comments