Source Generators and Analyzers extensibility

[pedro-gilmora]

Motivation

I'm exploring a scenario where the initialization context of a source generator (SGen) can subscribe to a delegate from an SGen engine host, provided by a method within the SourceProducerContext. This would enable core source generator authors to:

• Check the state of user-defined metadata
• Allow third-party or the same authors to generate code based on the information shared among authors of custom data through the proposed provider.
• Integrate or extend existing generated code by creating an execution mechanism, (could evolving into a tree-execution-queue).

However, this approach comes with performance concerns, such as potential deadlocks, especially when relying on extension SGen authors’ implementations. To mitigate these issues, one solution could involve propagating the cancel token and execute 3rd party SGens with some predefined timeout.

Key-points

• Coordination of 3rd party SGens, though can they cause waiting subsequent dependent (also the reason of the cancellation token and timeout, something you the team can consider to be taken differently)
• Avoid implementation collisions (namings and duplicate implementations)
• Obviously, compilation contexts (as well as nodes and syntaxes) can't be shared or accessed by different authors. (only metadata, not generation context)
• Subsequent generators must generate their own code only based on the given metadata shared by core SGens authors and can be combined with other providers
• Shared metadata should be an intermediary assembly not being the same core SGens assembly (not a good idea I guess)

---

Proposed API

To address the outlined scenario, the following API can be introduced:

public sealed class AnalyzerExtensibilityProvider
{
    static IncrementalValueProvider<T> Publish<T>(T exchangedData, CancellationToken cancelToken);
    static IncrementalValueProvider<AnalyzerSubscription<T>> SubscribeTo<T>(Func<T, CancellationToken, bool> exchangedData);
}

public sealed class AnalyzerSubscription<T>
{
    public T State { get; }
    // Report back to the publisher the current state changes
    public void Return(CancellationToken token);
}

Names are proposal. Maybe you can provide better ones according to their usage context

• Publish<T>: Allows the core source generator to publish data to third-party consumers.
• SubscribeTo<T>: Allows third-party authors to subscribe to the data published by the core source generator.
• AnalyzerSubscription<T>: Represents the subscription, allowing subscribers to return the state changes to the publisher.

---

Usage Sample

Core Source Generator Author Example

var compilationProvider = incrementalGeneratorInitializationContext.CompilationProvider;

// Usage of AnalyzerExtensibilityProvider as a property
var extensibilityProvider = incrementalGeneratorInitializationContext.AnalyzerExtensibilityProvider;

var collectedServiceContainerSymbols = context.SyntaxProvider
    .ForAttributeWithMetadataName(ServiceContainerAttributeFQName,
        (node, a) => true,
        (t, c) => (t.SemanticModel, Class: (INamedTypeSymbol)t.TargetSymbol)) 
    .Collect();

incrementalGeneratorInitializationContext.RegisterSourceOutput(
    compilationProvider
        .Combine(extensibilityProvider)
        .Combine(collectedServiceContainerSymbols), 
    (sourceProducer, ((compilation, interop), collectedServiceContainer), cancellationToken) => {

        HashSet<string> namesSet = new(NameUniquenessComparer); //ensure file name uniquness name

        foreach (var (semanticModel, serviceContainerSymbol) in collectedServiceContainer)
        {
            if(ServiceContainerGenerator
                .DiscoverServicesFor(
                    compilation, 
                    semanticModel,
                    serviceContainerSymbol,
                    namesSet,
                    unresolvedDependencyHandler: (ServiceMetadata serviceDesc) => {
                                                  // ServiceMetadata type is hosted at some core-author assembly 
                                                  // to exchange custom metadata info with other interested SGen writers
                        interop.Publish(serviceDesc, cancellationToken);
                    })
                .TryBuild(out string fileName, out string code))
            {
                sourceProducer.AddSource(fileName, code);
            };
        }
    });

Third-Party Source Generator Author Example

var compilationProvider = incrementalGeneratorInitializationContext.CompilationProvider;

// Usage of AnalyzerExtensibilityProvider as a property
var emittedLoggerServices = incrementalGeneratorInitializationContext.AnalyzerExtensibilityProvider
    .SubscribeTo<ServiceMetadata>((serviceMetadataSub, cancelToken) => 
    {
        if (IsLogger(serviceMetadataSub.State))
        {
            serviceMetadataSubscription.State.Resolved = true;
            serviceMetadataSubscription.State.Name = serviceMetadataSubscription.;
            // Reports changes made to the current shared state for this SGen 
            // and no more changes can be reported in order to avoid delay on core-sgen publisher of this metadata
            serviceMetadataSubscription.Return(cancelToken); 

            return true;
        }
        return false;
    })
    .Collect();

incrementalGeneratorInitializationContext.RegisterPostInitializationOutput(CreateLoggerFactory);

incrementalGeneratorInitializationContext.RegisterSourceOutput(
    compilationProvider.Combine(emittedLoggerServices), 
    (sourceProducer, info, cancellationToken) => {
        var (compilation, emittedLoggerServicesData) = info;
        // Proceed to generate third-party code, with the help of the current Compilation context 
    });

---

Risks

• Performance Concerns: The proposed mechanism could lead to potential deadlocks, particularly when extension SGen authors' implementations are not well-coordinated.
• Broadcasting Solution: As a potential mitigation, broadcasting shared types might alleviate some of these risks, but this too introduces its own challenges in managing the broadcast and ensuring consistent state across participants.

@sharwell ... At your considerations

[sharwell]

Issue #57239 involves a proposal for an overlapping set of problems.

The main problem with the current proposal is it does not include any concrete motivating examples (detailed descriptions of source generator solutions which cannot currently be implemented, but could be implemented if the proposed API was available). Once these examples are provided, the next step is evaluating related proposals (e.g. #57239) to determine which proposals provide a solution to which specific problems.

[pedro-gilmora]

I see, you're asking for real world examples for it.

The one that made me do this proposal was a Dependency Injection resolver at compile time. I thought it to make able others SGen and analyzer writers to use this one as basis for their own code analysis and generations.

Since it's not possible to share data among generators, I took the following approach

Make others generators aware of the existence of mine. And then, they should implement based on their own attributes inheriting mine. So I can detect those ones and be aware whenever I find them.

Eg:

public sealed class JsonConfigurationAttribute(
    string fileName = "appsettings", 
    bool optional = true, 
    bool reloadOnChange = true) : SingletonAttribute(factory: nameof(ConfigurationResolver.GetConfiguration));

public sealed class JsonSettingAttribute(string key) : SingletonAttribute(factory: nameof(ConfigurationResolver.GetSetting));

But that won't avoid collisions and possible ambiguities or implementations when doing such parallel process by possible multiple authors trying to integrate their packages by generating their code variant.

So, this is the main part. Ensuring other generator to no waste their time resolving what's already taken for some primary work.

Same for runtime existing variant it might lead to resolving exceptions

Abstracting this issue sounds like this:

What if I want to others use my conventions to generate code and provide more functionalities to an existing one? What would be the best approach to conciliate in compile time, such that others want to provide to an initial generated implementation by generating or not their own partialization of it?

My first idea sharing data through a delegate didn't work at release because of the difference of execution contexts. So, I can't bring that to here I guess