Inconsistent behaviour with source generator results between unit testing and launch profile

[papafe]

Version Used: Microsoft.CodeAnalysis.CSharp 4.2.0 Microsoft.CodeAnalysis.Analyzers 3.3.3 Microsoft.CodeAnalysis.CSharp.SourceGenerators.Testing.NUnit 1.1.1

Steps to Reproduce:

1. Create net6.0 library project (Project A) with an attribute:

   [AttributeUsage(AttributeTargets.Class, AllowMultiple = false)]
   public sealed class TestAttAttribute : Attribute
   {
   internal string Name { get; }
   
   public TestAttAttribute(string name)
   {
       Name = name;
   }
   }

2. Create another net6.0 project (Project B) with a class using the attribute:

   
   [TestAtt("testString")]
   public class Class1
   {

}

3. Create a source generator project in which we try to extract the attribute argument:
```csharp
public void Execute(GeneratorExecutionContext context)
{
    // ....
    var attributeName = "TestAttAttribute";
    var attribute = classSymbol.GetAttributes().FirstOrDefault(a => a.AttributeClass.Name == attributeName);
    var attributeArgument = attribute?.ConstructorArguments[0].Value;  // "testString" 
}

4. Create a test project that follows the recommended testing approach in the roslyn source generators cookbook. The unit test takes as an input only the file in Project B where Class1 is defined. The only difference with the cookbook is adding a reference in the test state to Project A, where the attribute resides:

   TestState.AdditionalReferences.Add(typeof(TestAttAttribute).Assembly.Location);

5. Setup "Roslyn Component" launch profile on the source generator project using Project B as the target project, in order to be able to debug the generation

Expected Behavior:

Debugging the source generator, both with the launch profile and the unit test (passing only the file with Class1 as input), I expect the variable attributeArgument to be "testString"

Actual Behavior:

When debugging with the launch profile attributeArgument = "testString" as expected. From the Immediate window we can see that ConstructorArguments has a length of 1:

When debugging the unit test, instead, ConstructorArguments is empty. From the Immediate window we can see that ConstructorArguments has a length of 0:

Still while debugging the unit test it seems that the assembly of the attribute class is correctly recognised:

Is there anything that could cause this? Maybe I'm using the testing framework wrong? I have created a solution that reproduces the issue, I can eventually post it here if necessary.

Additional note I've noticed this while working on a much bigger source generator project. The result is the same, but in my original project when debugging the unit test I can see that the AttributeClass is of ErrorType, so I thought that this was the cause of the missing constructor arguments. This is the reason why I've made a separate solution trying to reproduce the issue. The issue is the same, but in this case AttributeClass is of type NamedType, so that seems to be correct.

[jaredpar]

Do you have a repro project published anywhere that we can look at?

[papafe]

@jaredpar sure, I've just created a repo: https://github.com/papafe/source-gen-attribute. To reproduce, add a breakpoint here:

https://github.com/papafe/source-gen-attribute/blob/6505190f8509187461cb3dd7e843e850214d87c3/SourceGenerator/ClassGenerator.cs#L23

If you debug the single unit test in UnitTest1.cs (project TestProject), you can see that that line will throw an exception. Instead if you debug the source generator using the "Roslyn component" profile (called Profile1), there's no exception.

The solution should be working as-is, but let me know if you need additional info

[Youssef1313]

@papafe It looks like you're not providing the source code of the attribute into the test input. I see now you do it in CSharpSourceGeneratorVerifier, sorry.

[papafe]

@Youssef1313 I am adding a reference to the assembly containing the attribute in the CSharpSourceGeneratorVerifier here:

https://github.com/papafe/source-gen-attribute/blob/6505190f8509187461cb3dd7e843e850214d87c3/TestProject/CSharpSourceGeneratorVerifier.cs#L23

[Youssef1313]

@papafe You need to add ReferenceAssemblies = ReferenceAssemblies.Net.Net60; in Test constructor (below or above the line TestState.AdditionalReferences.Add(typeof(TestAttAttribute).Assembly.Location);)

The assembly that contains TestAttAttribute that you're adding is .NET 6. So, previously the compilation had this error:

error CS1705: Assembly 'MainLibrary' with identity 'MainLibrary, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' uses 'System.Runtime, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' which has a higher version than referenced assembly 'System.Runtime' with identity 'System.Runtime, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'

[papafe]

@Youssef1313 It works, thanks! For future reference, what do you mean by "previously the compilation had this error"? Is there a way to access the error from the verifier?

[Youssef1313]

@papafe While debugging, you can see compilation errors via context.Compilation.GetDiagnostics() (context is the GeneratorExecutionContext in Execute method).

[papafe]

@Youssef1313 Thank you! That's definitely good to know 😄

[jaredpar]

Thanks for the detailed repro @papafe. I hadn't realized @Youssef1313 had already solved the issue and was debugging through your repro this morning. I came to the same conclusion as @youssef1313 did.

I went a slightly different route though. Generally when there is inconsistency like this it's usually due to differing reference sets. I looked at context.Compilation.References under the debugger for both scenarios. Noticed in the unit tests that the main set of references were for netcoreapp3.1 while your lib was compiled for net6.0. As an experiment I changed your lib to target netcoreapp3.1 and verified that the error was fixed.

[papafe]

@jaredpar Thanks for the tip! It's definitely good to know