Inject/Forge Diagnostics or run "foreign" Analyzers in CodeFix-Test

[LukasGelke]

I'm currently writing a CodeFixProvider which will provide possible fixes to existing Diagnostics based on pre-existing company guidelines/codebase. But I cannot test it, as the original Diagnostic (e.g. CA1062) is never triggered, even when adding the Package Microsoft.CodeAnalysis.NetAnalyzers to the TestState.ReferenceAssemblies and adding an .editorconfig (with that Analyzer on Warning) as an AdditionalFile.

My Test

using ValidatePublicParameterWithKomsaAssertVerifier = Komsa.CodeAnalyzers.Test.Internal.KomsaCodeFixVerifier<
  Komsa.CodeAnalyzers.CodeFixProviders.ValidatePublicParameterWithKomsaAssertCodeFixProvider>;

public async Task FindAndAddAssertsAsyncTest(string folder, string code, string fix, string key)
  {
    ReferenceAssemblies assemblies = ReferenceAssemblies.Net.Net60
      .AddPackages(ImmutableArray.Create<PackageIdentity>(item: new("Microsoft.CodeAnalysis.NetAnalyzers", "6.0.0")));
    var addtional = (".editorconfig", @"
[*.cs]

# CA1062: Validate arguments of public methods
dotnet_diagnostic.CA1062.severity = warning
");
    ValidatePublicParameterWithKomsaAssertVerifier.Test test = new()
    {
      TestState =
      {
        Sources = { CodeAnalyzerTestHelper.GetSourceCode(Rule, folder, code) },
        AdditionalReferences = { typeof(KomsaAssert).Assembly },
        AdditionalFiles = { addtional },
        ReferenceAssemblies = assemblies,
      },
      FixedState =
      {
        Sources = { CodeAnalyzerTestHelper.GetSourceCode(Rule, folder, fix) },
        AdditionalReferences = { typeof(KomsaAssert).Assembly },
        AdditionalFiles = { addtional },
        ReferenceAssemblies = assemblies,
      },
      CodeActionEquivalenceKey = "ValidatePublicParameterWithKomsaAssertCodeFixProvider:CA1062:" + key,
    };
    await test.RunAsync().ConfigureAwait(false);
  }

TestState.Source

public sealed class MyClass
{
  public void DoStuff(object o)
  {
    _ = {|CA1062:o|}.ToString();
  }
}

internal sealed class KomsaCodeFixVerifier<TCodeFix>
  : KomsaCodeFixVerifier<EmptyDiagnosticAnalyzer, TCodeFix> {...}
  //... and corresponding classes exist, to add company-defaults

ValidatePublicParameterWithKomsaAssertCodeFixProvider.FixableDiagnosticIds = { "CA1062" }

Assert.AreEqual failed. Expected:<1>. Actual:<0>. Context: Diagnostics of test state Mismatch between number of diagnostics returned, expected "1" actual "0"

Removing the markup doesn't raise this Exception, but I obviously have no Diagnostic for my CodeFix.

How can I produce a "usable" Diagnostic? For this test, i would only need the Span, so the markup would be sufficient. But to "future-proof" the CodeFix (and others), it would be "more stable" to run the original Analyzer. But one can't reference an Analyzer from a NuGet-Package, and use it in the Verifier.

[LukasGelke]

oh, and it might also be used for testing Suppressors

[sharwell]

You'll want to change EmptyDiagnosticAnalyzer to Microsoft.CodeQuality.Analyzers.QualityGuidelines.ValidateArgumentsOfPublicMethods, which is the analyzer responsible for reporting CA1062.

[LukasGelke]

this is what i thought of first too, but: a) which NuGet/Reference do i need? and more importantly b) how would I get to that in code? Assuming I can't consume an Analyzer Package normally and access it's containing Analyzers, the way it's done with 'lib' Packages

[LukasGelke]

ok. So some Analyzers can be used like this

  <ItemGroup>
    <PackageReference Update="Microsoft.CodeAnalysis.NetAnalyzers" GeneratePathProperty="true" />
    <Reference Include="Microsoft.CodeAnalysis.NetAnalyzers">
      <HintPath>$(PkgMicrosoft_CodeAnalysis_NetAnalyzers)\analyzers\dotnet\cs\Microsoft.CodeAnalysis.NetAnalyzers.dll</HintPath>
      <CopyLocal>true</CopyLocal>
      <Private>True</Private>
    </Reference>
  </ItemGroup>

it seams hacky, but it works.

Now though, we would like to test stuff regarding some 'IDE'-Diagnostics. For instance IDE0002. But that's in dotnet/roslyn CSharpSimplifyTypeNamesDiagnosticAnalyzer which is internal and inaccessible through normal means. But I've hit a roadblock: https://github.com/dotnet/roslyn/blob/0b24f6bf0bc243a102248ca501e35f2d85326212/src/Analyzers/Core/Analyzers/EnforceOnBuildValues.cs#L123 suggests that IDE0002 cannot be used during build in the first place.

But how/where would one enable these Analyzers in the Testing Environment? Because https://learn.microsoft.com/en-us/dotnet/fundamentals/code-analysis/overview?tabs=net-7#enable-on-build says to first "Set the MSBuild property EnforceCodeStyleInBuild to true". Now I'm not sure if a Test would "qualify as a build" at all, and how to set it

[sharwell]

If you want to manually add diagnostics to a test, it's pretty trivial. Here's a test analyzer that reports a diagnostic for literal integers less than 5: https://github.com/dotnet/roslyn-sdk/blob/f9932c818a30eedd0715df2971c340a695d393af/tests/Microsoft.CodeAnalysis.Testing/Microsoft.CodeAnalysis.Testing.Utilities/TestAnalyzers/LiteralUnderFiveAnalyzer.cs

[LukasGelke]

Hm. Yes, that would be possible, but I'd like the "real analyzer" better.

I need to refer to my previous comment. Our test works with Microsoft.CodeAnalysis.CSharp.Workspaces Version 4.4.0 but in trying to upgrade to version 4.5.0 there are no more diagnostics reported by Microsoft.CodeQuality.Analyzers.QualityGuidelines.ValidateArgumentsOfPublicMethods