Stryker fails when running against .NET 5 Azure Function

[richparker288]

Describe the bug When running Stryker against a .NET 5 Azure Function with the Microsoft.Azure.Functions.Worker.Sdk package referenced we get the following exception:

Unhandled exception. System.IO.DirectoryNotFoundException: Could not find a part of the path 'C:\XXXXX\XXXX.XXXXX\XXXXX\StrykerAzFunctionRepro\StrykerAzFunctionRepro\obj\Release\netcoreapp3.1\WorkerExtensions.AssemblyInfo.cs'.

Version: 1.0.1

[14:15:10 INF] Stryker will use a max of 4 parallel testsessions.
[14:15:10 INF] Identifying project to mutate.
A new version of Stryker.NET (1.2.2) is available. Please consider upgrading using `dotnet tool update -g dotnet-stryker`

A preview version of Stryker.NET (1.2.2) is available.
If you would like to try out this preview version you can install it with `dotnet tool update -g dotnet-stryker --version 1.2.2`
Since this is a preview feature things might not work as expected! Please report any findings on GitHub!

[14:15:12 INF] The project C:\XXXX\XXXX.XXXXX\XXXXX\StrykerAzFunctionRepro\StrykerAzFunctionRepro\StrykerAzFunctionRepro.csproj will be mutated.
[14:15:14 INF] Time Elapsed 00:00:04.0508342
Unhandled exception. System.IO.DirectoryNotFoundException: Could not find a part of the path 'C:\XXXXX\XXXX.XXXXX\XXXXX\StrykerAzFunctionRepro\StrykerAzFunctionRepro\obj\Release\netcoreapp3.1\WorkerExtensions.AssemblyInfo.cs'.
at System.IO.FileStream.ValidateFileHandle(SafeFileHandle fileHandle)
at System.IO.FileStream.CreateFileOpenHandle(FileMode mode, FileShare share, FileOptions options)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
at System.IO.StreamReader.ValidateArgsAndOpenPath(String path, Encoding encoding, Int32 bufferSize)
at System.IO.StreamReader..ctor(String path, Encoding encoding, Boolean detectEncodingFromByteOrderMarks)
at System.IO.File.InternalReadAllText(String path, Encoding encoding)
at System.IO.File.ReadAllText(String path)
at System.IO.Abstractions.FileWrapper.ReadAllText(String path)
at Stryker.Core.Initialisation.CsharpProjectComponentsBuilder.FindProjectFilesUsingBuildalyzer(IAnalyzerResult analyzerResult, StrykerOptions options) in /_/src/Stryker.Core/Stryker.Core/Initialisation/CsharpProjectComponentsBuilder.cs:line 99
at Stryker.Core.Initialisation.CsharpProjectComponentsBuilder.Build() in /_/src/Stryker.Core/Stryker.Core/Initialisation/CsharpProjectComponentsBuilder.cs:line 40
at Stryker.Core.Initialisation.InputFileResolver.ResolveInput(StrykerOptions options) in /_/src/Stryker.Core/Stryker.Core/Initialisation/InputFileResolver.cs:line 77
at Stryker.Core.Initialisation.InitialisationProcess.Initialize(StrykerOptions options) in /_/src/Stryker.Core/Stryker.Core/Initialisation/InitialisationProcess.cs:line 64
at Stryker.Core.Initialisation.ProjectMutator.MutateProject(StrykerOptions options, IReporter reporters) in /_/src/Stryker.Core/Stryker.Core/Initialisation/ProjectMutator.cs:line 30
at Stryker.Core.Initialisation.ProjectOrchestrator.MutateProjects(StrykerOptions options, IReporter reporters)+MoveNext() in /_/src/Stryker.Core/Stryker.Core/Initialisation/ProjectOrchestrator.cs:line 61
at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
at Stryker.Core.StrykerRunner.RunMutationTest(IStrykerInputs inputs, ILoggerFactory loggerFactory, IProjectOrchestrator projectOrchestrator) in /_/src/Stryker.Core/Stryker.Core/StrykerRunner.cs:line
59
at Stryker.CLI.StrykerCli.RunStryker(IStrykerInputs inputs) in /_/src/Stryker.CLI/Stryker.CLI/StrykerCLI.cs:line 90
at Stryker.CLI.StrykerCli.<>c__DisplayClass9_0.<Run>b__0() in /_/src/Stryker.CLI/Stryker.CLI/StrykerCLI.cs:line 65
at McMaster.Extensions.CommandLineUtils.CommandLineApplication.<>c__DisplayClass146_0.<OnExecute>b__0(CancellationToken _)
at McMaster.Extensions.CommandLineUtils.CommandLineApplication.ExecuteAsync(String[] args, CancellationToken cancellationToken)
at McMaster.Extensions.CommandLineUtils.CommandLineApplication.Execute(String[] args)
at Stryker.CLI.StrykerCli.Run(String[] args) in /_/src/Stryker.CLI/Stryker.CLI/StrykerCLI.cs:line 71
at Stryker.CLI.Program.Main(String[] args) in /_/src/Stryker.CLI/Stryker.CLI/Program.cs:line 15

Logs log-20220112.txt

Expected behavior Stryker tests are executed

Desktop (please complete the following information):

• OS: Windows
• Type of project: Azure Function
• Framework Version: .NET 5
• Stryker Version: 1.0.1 (same error occurs with version 1.2.2)

Additional Context As soon as the package Microsoft.Azure.Functions.Worker.Sdk is removed, the tests run fine.

[rouke-broersma]

Looks like a generated file WorkerExtensions.AssemblyInfo.cs is available during buildalyzer analysis but missing afterwards causing us to not be able to read this file. It might be enough to skip *.AssemblyInfo.cs files if they cannot be found on disk. If that does not work this might have to be fixed in upstreams buildalyzer.

[mlouage]

I'm having the same issue with .NET 6 and isolated functions. How can I configure it to skip that WorkerExtensions.AssemblyInfo.cs file?

[JJ-288]

Is there an update to this or are there any workarounds we can use in the meantime? Thanks.

[dupdob]

Is the problem still present with the latest version? Is there any chance we can get a project to reproduce this issue ?

[JJ-288]

Is the problem still present with the latest version? Is there any chance we can get a project to reproduce this issue ?

Apologies for the delayed response. Yes, this was tested against the latest version. Will try and find time to get a sanitized project to replicate the problem.

[rodchenkov]

Hi, I have the same issue with .NET 6.0 isolated function

function.csproj

`

net6.0 v4 Exe AllEnabledByDefault eb4984ff-b68e-46de-91ab-cda4f2e350a4

`

Can you please check how to solve it as even exclude of WorkerExtensions.AssemblyInfo.cs not working for me or I'm doing it not right.

Thanks!

[JJ-288]

StrykerReplication.zip

Attached a dummy project with test - enough to replicate the issue on 2.0.0

[dupdob]

thanks for the project. I did reproduce this issue. The missing WorkerExtensions.AssemblyInfo.cs file is just the tip of the iceberg: it seems that Buildalyzer completely fails for this project. Indeed, WorkerExtensions.AssemblyInfo.cs is the ONLY found source file (according to Buildalyzer).

[dupdob]

@richardwerkman : I see you are the author of those lines https://github.com/stryker-mutator/stryker-net/blob/f67f96fbd413b6f62c0c34d68107b4cbbc0e79de/src/Stryker.Core/Stryker.Core/Initialisation/ProjectFileReader.cs#L69-L72 but with this project here, we clearly have a true failure... do you remember when/why Stryker assumes everything will work even if Buildalyzer fails.

[richardwerkman]

Yes. Buildalyzer seemed to return success false sometimes even though it provided all project info we needed and our intitial build step succeeded. So it seemed like a bug in Buildalyzer to me. Since stryker had no trouble continuing the run I decided to log a debug statement but ignore the failed Buildalyzer step.

That was quite some time ago, so things might have changed since then.

[dupdob]

I have some progress, but I fear I am in a dead end: I have been able to properly extract the source files (out the AnalyzerResults), but I still have troubles with the project references which are not resolved. I may be able to hack around this. But this appears to be a Buildalyzer issue/limitations. I am afraid the problem needs to be addressed there.

[JJ-288]

I have some progress, but I fear I am in a dead end: I have been able to properly extract the source files (out the AnalyzerResults), but I still have troubles with the project references which are not resolved. I may be able to hack around this. But this appears to be a Buildalyzer issue/limitations. I am afraid the problem needs to be addressed there.

Did you have any luck with a potential work around or does this need to be raised separately?

[Cosmin-Apopei]

Has an issue been raised on the Buildalyzer side? If so, could it be linked here?

[dupdob]

@JJ-288 : I am afraid I stopped pursuing this strategy. I realized I was working around one of BuildAlyzer's issues and this was turning in a wild goose chase: everytime I fixed a problem, another one appeared. @Cosmin-Apopei : not that I am aware of. Most of my current Stryker devoted time is spent on a new feature, so I did not have the opportunity to create a reproducing project for the BuildAlyzer team to work upon.

looking at azure functions project, I realized those rely heavily on custom build steps. As such, there is a high risk that Stryker will not be able to handle those even if Buildalyzer's issue(s?) is fixed.

[bonner-earle]

I have created an issue in the Buildalyzer project https://github.com/daveaglick/Buildalyzer/issues/210 with an example project based on the example project for this bug

[daveaglick]

looking at azure functions project, I realized those rely heavily on custom build steps

This is the key. Buildalyzer runs MSBuild (I.e. dotnet build) while instrumenting the build with a custom logger that pipes events back to Buildalyzer for analysis. Those events usually follow a known pattern which means we can use them to pick up on things like which files were involved in the build (in that case by looking for the call MSBuild makes to the .NET compiler and parsing the command line involved).

Basically, all bets are off if someone instruments MSBuild in a way that it deviates too much from expectations. MSBuild itself is a "program" and so creating a tool that can process all possible MSBuild behaviors is impossible, because the range of possible behaviors is infinite.

That said, we may be able to special-case the tasks and targets that Azure Functions adds to continue reporting source files, assuming (and this is a big assumption) they're in the logs at all. I've seen other MSBuild tasks that "eat" things like source files because they go off and do their own compilation or processing totally outside the normal MSBuild flow of calling csc. Still looking at this specific case in https://github.com/daveaglick/Buildalyzer/issues/210 to see if Buildalyzer can support it - but no promises at this point.

[jakubtrebacz-dev]

I believe I had similar problem in DevOps pipeline. It's a .NET 6 WebApp project. I build the solution with dotnet build - This produces the right DLLs Property TargetFramework: net6.0 Property MSBuildSemanticVersion=17.3.2+561848881 Property CompilerApiVersion=roslyn4.3

Then I run the dotnet Stryker which runs Buildalyzer I guess? And the DLLs seem to disappear? (it uses the same settings as dotnet build task)

✅ Here is what helped me in the end!

{
    "stryker-config":
    {
        ~"solution": "../../My.Solution.sln",
        "project": "MyProject.csproj",
        "verbosity": "trace"
    }
}

All I had to do is remove the reference to the solution file and it started working 😃 For some reason when it was building it through the solution file it lost the references and didn't build the whole project? Pointing it at the project alone seemed to resolve the right references and build everything.

[mjthompson2107]

Has anyone found a solution to this? I have the same problem, but the post above did not fix it

[AnonymousRetard]

I've had the same issue as well since 2022 and the above did not fix it for me either. I have unfortunately not found any solution.

Edit: I have been subscribed to this thread for a long time just waiting for an update without giving stryker a try again but now I did and it worked. Not sure what has changed. Are all your Microsoft.Azure.Functions packages at their latest versions?

[dupdob]

Some update on this issue.

TLDR; still no official support for AzureFunction, but things are improving

Details

1. Stryker (still) relies on Buildalyzer to reverse engineer the way to build the project properly. Buildalyzer relies on an instrumented MsBuild build to extract information. Buildalyzer relies on known pattern and structure, and there is no strong guarantee it can properly analyze exotic C# projects such as AzureFunction.
2. Stryker project analysis and build steps have been improved along the years. Especially Stryker provides support for more build customization scenario

What's next

The repro project provided in the conversation does not work at all with Stryker; I am currently fixing associated issues. it should increase support further.

[dupdob]

Update:

1. I found only 1 possible improvement in Stryker (increase odd of analyze success)
2. I am now looking into Buildalyzer: I may have a found an issue or a limitation in buildalyzer: it assumes the C# compiler will be called only once, hence it only stores the last call details. For this project, csc is called twice, and the second time is only used to generate an empty assembly.

[dupdob]

Update: Diagnostic is that Buildalyzer current version does not not support AzureFunction project. I opened PR # 280 on Buildalyzer with a fix for this problem. Next steps are:

• [x] 1. Buildalyzer PR to be validated and merged
• [ ] 2. Buildalyzer version with the fix to be released
• [ ] 3. Stryker with the fixed Builealyzer version to be released

I am afraid this is going to take time. Not that I can guarantee yet that it will works, I can only guarantee it works on the sample project provided in the conversation.

I will update this issue as the fix progresses though the pipeline

[dupdob]

Update: my BuildAlyzer PR has been validated and merged. Now we need to wait for it to be released