search

stryker-mutator / stryker-net

Mutation testing for .NET core and .NET framework!
https://stryker-mutator.io
Apache License 2.0
1.76k stars 175 forks source link

Stryker does not work with ASP.NET Core Integration Tests In-memory Test Server #1021

Closed hannahchan closed 4 years ago

hannahchan commented 4 years ago

I'm currently experimenting with Stryker on an ASP.NET Core WebAPI app which has both Unit Tests and Integration Tests in separate projects. The Integration Tests are written using the In-memory TestServer class as outlined here; https://docs.microsoft.com/en-us/aspnet/core/test/integration-tests.

When I run dotnet stryker against the Integration Test project and tell it to mutate my WebAPI project, I get the warning; It looks like all non-excluded mutants are not covered by a test. Go add some tests! with a final mutation score of 0.00%.

When I run dotnet stryker --coverage-analysis=off against the Integration Test project and tell it to mutate my WebAPI project, I get a final mutation score of 100.00%. When I glance over the mutation report, there's at least one mutant that I know of that can't have been possibly killed because I know there's no test for it.

I don't know how the inner working of Stryker works, so to me this looks like a bug.

Stryker seems to work ok for all my Unit Test projects. The version of Stryker that I am using is 0.17.1.

I'm also looking forward to the ability to mutate multiple projects in one run.

How to replicate

The ASP.NET Core WebAPI that I am experimenting with is located here; https://github.com/hannahchan/Scaffold.WebApi.

The project under test is located in Scaffold/Sources/Scaffold.WebApi.

The test project is located in Scaffold/Tests/IntegrationTests/Scaffold.WebApi.IntegrationTests.

To replicate, simply set your working directory to the test project and run dotnet stryker.

richardwerkman commented 4 years ago

I think this might be a duplicate of #591. When I run stryker on your project with debug logging I see the following for each test output: System.InvalidOperationException : The provided Type 'Startup' does not belong to an assembly with an entry point. A common cause for this error is providing a Type from a class library.

This happened before in MVC projects. We are not sure how to set the Startup type in Roslyn when compiling.

This seems to cause both the 0% coverage and 100% score when turning coverage off. As the tests cannot be started no mutant is covered by a test. And if you turn coverage off you get 100% score since every test fails...

I'm also looking forward to the ability to mutate multiple projects in one run.

We do already support multiple test projects! See: https://github.com/stryker-mutator/stryker-net/blob/master/docs/Configuration.md#test-projects

dupdob commented 4 years ago

For information, I am looking into this. It looks like this issue stacks up the problems we face building a project without being able to use MsBuild indeed. Already got past the entry point issue (this one is easy). Currently dealing with having wrong paths, probably due to a loss of automatic ressource generation or something to that effect.

dupdob commented 4 years ago

quick update: I hard coded some changes in Stryker + manually created the directory expected by the testing infrastructure and I have been able to generate a realistic looking mutation report!!. Capture d’écran 2020-04-06 à 17 29 29

Getting rid of the hard coded part is simple. I still have to figure out why a specific folder is expected only when testing the mutated assembly.

dupdob commented 4 years ago

see PR #1028 for what the code looks like. Progress report! Overall: semi functioning (see below) I discovered that projects using Microsoft.Web.Sdk are build as executable dll. Which requires setting the output type to consoleapp and forcing the modulename (otherwise you have non valid assembly). Then testing fails en masse due to a missing folder (that is not required for the original assembly). Creating the expected folder allowed to run the tests properly, except for one failing test (not failing with the original assembly).

Trying to debug the encountered issues is very difficult due to dynamic loading of many assemblies. I have not been able to understand the cause of theses (issues).

I also work by performing manual diff analysis of the assembly. That is how I understood the initial 'missing entry point' issue.

I opened this PR in the hope that someone with experience in web projects can help me make progress.