Stryker Logs Warnings For .NET 5.0 Analyzers

[pinkfloydx33]

Describe the bug When running Stryker against a .NET 5.0 project, errors/FileNotFoundExceptions are logged to the console as warnings. These appear to be related to built-in analyzers automatically added when targeting .NET 5.0.

This behavior is exhibited in 0.22.9 but not in 0.22.6

Logs

2021-08-05T09:02:44.0586863-04:00 [DBG] 206 mutants created (edec1dee) 2021-08-05T09:02:44.1083293-04:00 [WRN] Analyzer assembly /usr/share/dotnet/sdk/5.0.300/Sdks/Microsoft.NET.Sdk/targets/../analyzers/Microsoft.CodeAnalysis.CSharp.NetAnalyzers.dll could not be loaded. Generated source code may be missing. System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.CodeAnalysis.NetAnalyzers, Version=5.0.4.22903, Culture=neutral, PublicKeyToken=31bf3856ad364e35'. The system cannot find the file specified.

File name: 'Microsoft.CodeAnalysis.NetAnalyzers, Version=5.0.4.22903, Culture=neutral, PublicKeyToken=31bf3856ad364e35' at System.Reflection.RuntimeAssembly.GetExportedTypes() at System.Reflection.Assembly.get_ExportedTypes() at Stryker.Core.Initialisation.Buildalyzer.IAnalyzerResultExtensions.GetSourceGenerators(IAnalyzerResult analyzerResult, ILogger logger)

[10:31:46 WRN] Analyzer assembly /usr/share/dotnet/sdk/5.0.300/Sdks/Microsoft.NET.Sdk/targets/../analyzers/Microsoft.CodeAnalysis.NetAnalyzers.dll could not be loaded. Generated source code may be missing. System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.CodeAnalysis.Workspaces, Version=3.3.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'. The system cannot find the file specified.

File name: 'Microsoft.CodeAnalysis.Workspaces, Version=3.3.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' at System.Reflection.RuntimeAssembly.GetExportedTypes() at System.Reflection.Assembly.get_ExportedTypes() at Stryker.Core.Initialisation.Buildalyzer.IAnalyzerResultExtensions.GetSourceGenerators(IAnalyzerResult analyzerResult, ILogger logger)

2021-08-05T09:02:44.1122101-04:00 [DBG] Trying compilation for the first time. (8be7d951)

Expected behavior No exceptions are logged

Desktop (please complete the following information):

• OS: Windows 10 and Ubuntu 18.04
• Type of project: core
• Framework Version: net5.0
• Stryker Version: 0.22.9 beta

Additional context I see the errors whether my project targets only .NET 5.0 or if it's multi-targeted.

[rouke-broersma]

In 0.22.8 we added support for c# source analyzers. It looks like this causes problems when you don't use source analyzers. From the logs it looks like Stryker handles this gracefully is that correct?

[pinkfloydx33]

So far as I can tell that is correct. It just blurts out the warnings to the console but otherwise seems to run the same as before.

[psfinaki]

Yeah I can confirm this noise has appeared. Nothing bad happens really but those messages probably shouldn't be there or at least should be "quieter". @JohnMcGlynnMSFT FYI.

[dupdob]

For clarifications, this messages are here on purpose: I asked for them. They come from the support of Roslyn source code generator and they signal Stryker failed to load an assembly that may contain a generator; a situation that would likely lead to failed build. In this case, this is clearly an analyzer that failed to load (based on the assembly name), so it may be seen as noise. Sadly, the situation is quite complex:

1. Generators are declared as Analyzers, hence Stryker cannot distinguish them without loading them, and when one failed to load, there is no way to tell if it was a generator or an analyzer, or even both. hence the error message
2. There is little to none available documentation regarding Analyzer infrastructure, that is why we cannot reliably load them (yet).
3. It appears most analyzers failed to load while generators seem to work fine. As of know, we do not know if this is structural or random.
4. There is a sensible request to add support for Analyzers (#1534).

In short, we clearly need more testing and feedback from users to decide how to proceed with this. If those messages were quieter, it may lead to unexplained failure from Stryker.

[rouke-broersma]

Thank you for the clarification @dupdob. I agree that we should keep this logging. Perhaps we could downgrade the exception to debug logging but keep the warning message above it as warning so we don't alarm users as much as we currently do. And also explain in the logging that if the warning is about an expected Analyzer that it can be ignored and if it's referencing a source generator that it could be a problem?

[pinkfloydx33]

That all makes sense to me.

Would it make sense (or even be possible) to automatically exclude the framework analyzers by name, ie. the ones that come just from using net5.0/net6.0 and aspnetcore? Those are never going to be "user" code so loading them seems unnecessary. That would prevent the warning and exception logged for those cases but keep them for others, without changing the log level.

Obviously that wouldn't cover analyzers from other referenced projects, which I think is OK. These specific ones are going to start appearing in projects by default meaning pretty much every run of Stryker for net5+ will spam the output.

[rouke-broersma]

I do wonder what these built in analyzers.. Do exactly. Perhaps they do stuff with editorconfig and such and then it still can explain certain behaviors when they don't work.

[pinkfloydx33]

The add a lot of built in code fixes, new configurable diagnostics, usage/best practices tweaks and the like. A lot of stuff requested by the community that doesn't need to be a part of the actual language spec. They're adding more and more of them with each release.

[dupdob]

Priority is to have analyzers working 😉. Every analyzer I tested exhibits this very problem: being unable to resolve basic analyzer dependencies, so fixing it once should fix it everywhere. As a workaround, we can treat specifically assemblies that fail to load due to Microsoft.CodeAnalysis.Workspaces

[p-bojkowski]

Related to this issue: https://github.com/JosefPihrt/Roslynator/issues/833#issuecomment-894577987

[JohnMcGlynnMSFT]

Give this a whirl: https://github.com/stryker-mutator/stryker-net/pull/1702