Source Generators: Questions about files on disk and (build) options

[ericwj]

I have questions and a general call to simplify Source Generators

Version Used:

<PackageReference Include="Microsoft.CodeAnalysis.CSharp.Workspaces" Version="3.7.0-2.final" PrivateAssets="all" />
<PackageReference Include="Microsoft.CodeAnalysis.Analyzers" Version="3.0.0" PrivateAssets="all" />

Questions I read about obtaining settings from SourceGeneratorContext but afai can see AnalyzerConfigOptions and GlobalOptions as documented here have completely vanished - I tried installing older NuGets and still they're not there.

• What is the current status of obtaining build options and to opt into receiving them in a source generator?
• How to create/obtain logs and error information like stack traces short of CS8785 and writing diagnostics? Especially while working on the source generator. I now misuse Warning just because anything less won't show up in VS.

I might have misunderstood the blog and other communications about this feature, but I would really like and had expected Source Generators to be able to write Source.g.cs or Source.Variant.g.cs next besides Source.cs in the project folder if so designed, or if desired/configured by the developer consuming the source generator.

• Is this planned? How to get files on disk that can be checked in?

The source generator happens to be randomly in either $(ProjectDir) or C:\....\MSBuild\Current\Bin\Roslyn, hence what I do now is use environment variables and not call AddSource, but that is obviously not quite completely as desired. It does however provide me the complete view of the code being generated, the file in the proper location and the source able to be checked in.

• Isn't it much more practical and quite a lot more intuitive than Guids and very long paths to use $(ProjectDir)\HintFile.g.cs if the source generator isn't overwriting anything when calling AddSource. A simple annotation (not a guid) in a comment at the top of the file or something basic like that would tell the compiler who might overwrite the file.

I don't think diagnostics like the one below are useful. For one, clicking on it doesn't do anything because the file does not exist.

• How to develop a source generator that is sizeable and generates lots of code and figure out errors like duplicate overloads? I mean before writing tests to figure out what tests to write if at all.

However as I find if the project doesn't build, the source generator doesn't run. So I now open the generated file and just Ctrl+A, Delete all code before building so the generator will run.

• Hence imo Roslyn should be able to link files to a source generator and run the generator if it finds (only) files generated have errors in them.

Also, the AnalyzerReleases.*.md files are very picky. Can these be slightly better documented or more flexibly parsed please? I tried writing one myself but I kept complaints until I copied one from github. If the column widths must be exact (why?) please document that.

Severity    Code    Description Project File    Line    Suppression State
Error   CS0102  The type 'x' already contains a definition for 'y'  Full.Namespace (net5.0) $(SolutionDir)\src\Project\acb8fc70-56e1-457d-afcf-b7c374e503ba_GeneratorNamespace.SourceGenerator_HintFilename.cs  14  Active

[sharwell]

What is the current status of obtaining build options and to opt into receiving them in a source generator?

It works starting with compiler 3.7.0-4.20351.2.

Is this planned? How to get files on disk that can be checked in?

It is planned for the compiler to have an option of writing generated files to disk so their content can be reviewed in an editor. The feature is designed from the start to actively prevent these files from getting checked in. Many of the early known use cases for the feature involve the elimination of .g.cs and .Designer.cs files from source control.

Isn't it much more practical and quite a lot more intuitive than Guids and very long paths to use $(ProjectDir)\HintFile.g.cs if the source generator isn't overwriting anything when calling AddSource.

The exact filename isn't really important for this, since the files will never be shared or in source control. Regardless of name, we want the IDE to allow navigating to these files are part of a seamless editor experience that includes generated content. For an example of a scenario that we want to work, see #45449.

How to develop a source generator that is sizeable and generates lots of code and figure out errors like duplicate overloads? I mean before writing tests to figure out what tests to write if at all.

A concrete example here would help me better understand the question. Part of the reason Source Generators were released using the early experimental API was to develop and understand the limits and best practices by starting to use them in real-world scenarios. We might not have all the answers yet, and are looking to community source generator authors to help fill in some of those gaps.

So I now open the generated file and ... delete all code ... so the generator will run ...

Source generators do not write any content to disk that is ever included in the subsequent compilation, so I don't understand how this sequence can be occurring.

Also, the AnalyzerReleases.*.md files are very picky. Can these be slightly better documented or more flexibly parsed please?

I believe @mavasani can help out with this?

[ericwj]

actively prevent these files from getting checked in

This is contrary what I understood from source-generators.md#output-files.

Why would the compiler care what I check in? Not writing to disk or writing to GeneratedFiles will be a suitable default which can then be .gitignored and that probably has your requirements covered.

• My first generator is one that by design isn't even supposed to be part of any other project than the one it is being written for - it just fits the scenario of writing lots of tedious code for one particular purpose. The resulting code could be a NuGet package, not the source generator.
• My second one will be to replace again project specific stuff that I now have as MSBuild steps. But I might not want on any particular day to work on the source generator and if it breaks I might choose to comment it out from the project file until whomever is going to fix it is going to do it. So I need those files on disk persisted and preferably checked in, because there is no telling what kind of edit will break the generator and when.

So, all I'm asking is to consider helping out a generator like mine which I definitely want to have writing files to $(ProjectDir) and you could help me do that in two or three ways.

1. First by allowing me to configure it to write to disk at a location specified or relative to the source file(s) that it is generating from (still with suitable collision detection),
2. second by preventing overwriting user sources or sources from other generators by a) writing to disk instead of me doing it anyway and b) identifying generated sources and which generator they came from, by ... idk ... something simple as to write /// <autogenerated generator="..."/> at the top of the file or something like that and parsing that back before considering to overwrite a file. That would also help track where the files came from such that a build can be clean short of brute forcing it by deleting GeneratedFiles\**\*.*.
3. Third - well, I'm not 100% sure but having spent a few days on this I didn't come accross any way of knowing which source file I am looking at in my generator/syntax receiver. Writing source.g.cs if I wanted to I would need to know that the file is source.cs in the first place.

I don't understand how this sequence can be occurring.

So I just call File.WriteAllText because csc starts generating these silly paths and refuses to let me see what was generated, not even if I doubleclick an error in the generated source from the Error Window. So afai can see these files were generated then vanished into thin air instantly. Sure that is because VS isn't enlightened about this yet, but it is silly useless nonetheless.

[CyrusNajmabadi]

Why would the compiler care what I check in? Not writing to disk or writing to GeneratedFiles will be a suitable default which can then be .gitignored and that probably has your requirements covered.

Our design may not include even running source generators in an environment that has a disk. i.e. we could easily opt for a system where SGs run in an extremely constrained sandbox where the SG only gets compilation-in and can only provide strings-out. The sandbox host would then manage everything from that point. So expecting or depending on the disk being present would explicitly put you in an unsupported place that could easily break at any point in the future (including post v1 launch).

[mavasani]

Also, the AnalyzerReleases.*.md files are very picky. Can these be slightly better documented or more flexibly parsed please? I tried writing one myself but I kept complaints until I copied one from github. If the column widths must be exact (why?) please document that.

Couple of points to note:

1. You are not supposed to manually edit the AnalyzerReleases.*.md files - we have a code fix that will do this correctly for you.
2. If you still prefer manually editing these and want to make the parsing more flexible, please file an issue at https://github.com/dotnet/roslyn-analyzers/issues.

[ericwj]

we have a code fix

Thanks, that appears to work now that I have them in the first place. Not sure what the problem was but if I rename them I can pretty much create new ones using Ctrl+. indeed.

How is the Notes column used? I put my diagnostics in Diagnostics.cs and that appears to cause the Notes column to be set to Diagnostics which would be wrong if that is shown to the user somehow.

Also the link that appears in the generated file shows [Documentation](CA4000_Documentation_Link). Is CA4000_Documentation_Link just a placeholder for a regular http:// link or is there still some additional magic to generate one somehow? It would be useful to generate them with some standard format, I can imagine.

[mavasani]

How is the Notes column used

That is just for your own reference - you should be able to put any custom text inside it. The code fix defaults the notes to the name of the containing type, as most analyzer authors would put their descriptor instantiations within the analyzer itself. Feel free to update it as appropriate for your case.

Also the link that appears in the generated file shows Documentation. Is CA4000_Documentation_Link just a placeholder for a regular http:// link or is there still some additional magic to generate one somehow? It would be useful to generate them with some standard format, I can imagine.

The code fixer uses the help link that is provided to the DiagnosticDescriptor constructor. So, whichever link you provide will be emitted in the notes field.

[ericwj]

Thanks @mavasani

[sharwell]

So I just call File.WriteAllText

I was afraid this might be the case. You need to make sure the files are not written to a location where the build will include them. Prior to the compiler doing this automatically, you can follow the approach of https://github.com/dotnet/roslyn-sdk/pull/553.

[ericwj]

Our design may not include even running source generators in an environment that has a disk

All the more reason to design some configuration to the ends that I tried to describe. I wouldn't care so much how as long as I can make it eventually do what I want, even through a combination of configuration provided by the source generator and the consuming project.

I can imagine why you would want to guarantee trustworthiness about the code running inside the compiler at who knows how many machines across the world and not provide an attack vector for malicious people.

But please while you're at it, the scenario's I outlined aren't hard to enable even if the code runs in some sandbox. If so configured it would just appear to do the most intuitive and simple thing and be hackable e.g. by making that XML annotation invalid for example.

the approach of #553

Yes, but VS doesn't understand the situation and my 9k of generated code doesn't compile - the Error Window does not really help me at all figuring out how to fix it. If these files were tagged somehow, csc would be fine with them having errors and recreate them on build -- unless I comment out the source generator from the .csproj in which case the project will keep building if it built when the source was generated.

[sharwell]

Yes, but VS doesn't understand the situation

If you follow the approach of dotnet/roslyn-sdk#553, Visual Studio will not be aware of the generated files at all, so they will not be able to cause build errors. The location where the files get saved is key to the approach working.

[ericwj]

Visual Studio will not be aware...they will not be able to cause build errors

And I can't see what is wrong with them.

So there are two things in my perspective.

• VS will in the near future be made aware of source generators and that will give me the insight into the code that is generated by being able to double-click the Error Window and just see the code. Alas, that doesn't work today, so I hack the system to get around that. The key being that VS does see the code and shows me what is wrong with it. If these files were tagged and the compiler understood those, this scenario would already work without any integration with VS (or whatever editor we'd like to use).
• Not all of us write massive test suites 10 times the size of the code generator and release them to millions of people. The whole system can be made much more usable without massive test suites and for use by all sorts of people for all sorts of projects of all sizes if it is able to tag files, write to disk and be generally hackable if so desired - in a secure way, if so configured.

I sense that your perspective is solidly from an enterprise perspective, trusting solidly that any code generator will always be (made) perfect, whatever the cost, but it would be such a pity for you to spend all this effort to build a great feature only to make it a pain to use on smaller scales because the few knobs I ask for aren't there.

Did I mention you wouldn't have to do much to make source generators integrate with any editor whatsoever (notepad) if you implement tagging and write files to disk?

[CyrusNajmabadi]

Yes, but VS doesn't understand the situation \

No matter what we do, we'll have to make VS understand. That's a given in teh design. The absence in understanding currently though is not a reason to then design a hacky way to make this work. :)

[sharwell]

I sense that your perspective is solidly from an enterprise perspective, trusting solidly that any code generator will always be (made) perfect, whatever the cost, but it would be such a pity for you to spend all this effort to build a great feature only to make it a pain to use on smaller scales because the few knobs I ask for aren't there.

You couldn't be further from the truth. The team is well aware of my concerns related to the difficulty of understanding source generator outputs. 😄 Really I want #45449 to work, where it doesn't matter if/where/how the files appear on disk, because you can always get to them like normal in the IDE.

[ericwj]

it doesn't matter if/where/how the files appear on disk

Sure that is a great vision (the crash is spectacular too, VS just vanishes, no exception whatsoever). It is also a suitable default to not write to disk and have a complex and involved IDE like VS do it like that. But yeah the merits of what I'm asking are that you don't need any IDE and can still get to the code and that the generator can be commented out if required. Also I don't need half a page of code to

design a hacky way

I'd rather have you do it for me once, forever proper.

I already have MSBuild steps that I regularly comment out for whatever reason temporarily and these are much simpler - usually they just write #define X and leave the rest of the 'code generation' to my own hand-typed #ifdefs.

[quinmars]

I also think that in most cases you do not want to see the output of source generators, and even worse include them to version control. For example, some extra functionality that is generated by an attribute or code parts of a XAML file. But there are cases where looking on the input file gives you no clue how the output will look like. Take for example a cryptical XML file, that is taken as input, to generate a readable C# class hierarchy or a data set that is generated from an excel file. In those cases I do want to include the generated file into version control, because than I can browse GitHub or Azure DevOps and inspect those file, without cloning the project, compile it with VS and look on it with IL spy, i.e., I can inspect the output without leaving the browser.

[ericwj]

I might have misunderstood the blog and other communications about this feature, but I would really like and had expected Source Generators to be able to write Source.g.cs or Source.Variant.g.cs next besides Source.cs in the project folder if so designed, or if desired/configured by the developer consuming the source generator.

I'm not entirely sure of the status of this request. How do I generate Namespace\HelloWorld.g.cs as a nested file of Namespace\HelloWorld.cs from which it was generated?

EDIT: Writing to $(MSBuildThisFileDirectory) generates the following error, so making generated files nested files doesn't appear to be possible for sure.

CSC : error CS2021: File name 'C:\Repos\Solution\src\HelloWorld\' is empty, contains invalid characters, has a drive specification without an absolute path, or is too long [C:\Repos\Solution\src\HelloWorld\HelloWorld.csproj]

By now I am able to use

<PropertyGroup>
  <TargetFramework>net6.0</TargetFramework>
  <EmitCompilerGeneratedFiles>true</EmitCompilerGeneratedFiles>
  <CompilerGeneratedFilesOutputPath>$(MSBuildThisFileDirectory)\Generated</CompilerGeneratedFilesOutputPath>
</PropertyGroup>

in the HelloWorld.csproj project that references my source generator, but it doesn't quite work, since the files are actually written to the folder (pseudo code)

$(MSBuildThisFileDirectory)\Generated\$(ProjectName)\$(SourceGeneratorProjectName)

and I cannot include a subfolder in the hintName:

context.AddSource("Namespace\\HelloWorld.g.cs", SourceText.From(sourceBuilder.ToString(), Encoding.UTF8));

CSC : warning CS8785: Generator 'HelloWorldGenerator' failed to generate source. It will not contribute to the output and compilation errors may occur as a result. Exception was of type 'ArgumentException' with message 'The hintName contains an invalid character '\' at position 9. (Parameter 'hintName')' [HelloWorld.csproj]

I consider this just about as bad as putting some GUID in the path of the generated file.

[sharwell]

How do I generate Namespace\HelloWorld.g.cs as a nested file of Namespace\HelloWorld.cs from which it was generated?

Currently, you don't. Source generators do not generate content on disk, so there are no files to show in solution explorer.

The request is interesting though. Currently the generated content is only shown under the Analyzers node, but it would be interesting to indicate that one or more generated files should be shown under specific other files.

[ericwj]

Source generators do not generate content on disk

I have some success using the MSBuild properties above which cause files to be written.

In fact if I set EmitCompilerGeneratedFiles to false, Visual Studio 16.11 preview 2 will not play along and show CS0103 for the HelloWorld class copied from the samples repo, even after nudging it to build successfully with dotnet build first and then restarting Visual Studio. It appears to be flaky without files on disk, seeing I spend yesterday and today while writing this quite some time trying to get IntelliSense to work and right now it just magically works; first in 17 preview 1.1 and then somehow it also starts to work reopening the unmodified solution in 16.11 preview 2...

But yes, having to hunt for some reference and pressing F12 is a very poor experience compared to just having the file right there, clickable to open it, on top of arguments that I have previously shared like wanting the ability to temporarily or semi-permanently disable the source generator completely.

[pr8x]

Yep, this entire Source Generator experience is extremely flaky - to say the least.

• IntelliSense doesn't work correctly. Partial class compiles fine, but IS reports errors.
• Generated sources don't appear in Solution Explorer (as implemented in some other PR, that claims it's shipping in VS 16.9). I am on 16.10 and haven't seen anything like that - even after hours of trying.
• Errors and warnings that you can't click (and see why/where it fails)

Sorry, but all in all really immature feature that I cannot recommend to anyone right now....

[CyrusNajmabadi]

@pr8x IDE support is still a WIP. We haven't finished any of the work there.

[ericwj]

The picture at the top of that PR has me seriously worried what it'll look like when it's not WIP anymore.

Mildly interesting to see which source generated generated those files, but it's hardly at all interesting (actually, it should usually be totally obvious) during work. So if those FIVE levels of nesting is all of the story then the inner loop takes a massive hit especially for source generators starting out with 99.9% handwritten code merely copied to be compiled more than once, or for thousands of lines of code that are being generated very tediously just that once from some data source with custom made format, etc.

[sharwell]

For 16.10 and newer, the only known cases where IDE support is incomplete are:

1. Diagnostics reported by a source generator only appear on build
2. Incorrectly authored source generator packages (even if the incorrect portion is accidental, it can cause the package to break in one or more scenarios)
3. Use of unsupported functionality (e.g. MSBuild targets customizations or file I/O within a source generator with the intent to bypass one or more limitations of current source generator APIs)

There are a few additional edge cases that are not handled, but users typically are able to avoid problems. Issues like the ones described in https://github.com/dotnet/roslyn/issues/45645#issuecomment-878281907 are typically caused by (2) or (3). We're happy to help if we can get a complete repro case as a separate issue.

The picture at the top of that PR has me seriously worried what it'll look like when it's not WIP anymore.

I'm not sure what the scenario/question here is. I'm happy to answer any specific questions you have about the current or future state, particularly in relation to IDE performance.

[ericwj]

For 16.10 and newer, the only known cases where IDE support is incomplete are: 1. 2. 3.

Does this mean IntelliSense should be working as expected as well? I copied a sample and it's flaky. Is the current state documented somewhere up-to-date? Including like a matrix of VS, TFM, package and compiler versions?

When I enable EmitCompilerGeneratedFiles I now get CS0101 and CS0111 for HelloWorld and SayHello respectively upon rebuild. Or is this considered an example of 3. above?

I'm not sure what the scenario/question here is.

Sorry that probably sounded a bit negative. I think the location of generated sources in the picture, nested 5 levels below Dependencies, is a location where you put code that you don't plan to ever look at. Maybe that's the case with Razor, because you have that same code somewhere else, just written differently. And for XAML, which just reflects all the work done in a visual designer.

But consider you might not like to look at, however you have to unless the original format is very well defined and the source generator is very high quality - or if it isn't, it'll shortly be because a very large company is backing that promise.

This issue was opened a year ago in hopes of keeping in your minds the situation where all these assumptions are false:

• few people or just one write the source generator, intermittantly,
• and it is not the product,
• the source format or data used is a work in progress
• perhaps perpetually so because if it doesn't change we don't work on the source generator,
• or the source format is the code being generated (using #if) and the whole point is getting that to compile multiple times to get all the grayed out blocks in the original, editable file to light up, syntax checked and compiled.
• etc, and combinations of all this.

Now probably you agree with me burrying that code 5 levels deep is not very productive, because there is no context between source file and generated file and the space between my source files and the related generated files is much too large. Just imagine to end up in some generated code file by pressing F12 somewhere, but that file is read only, and in some completely other part of solution explorer - how will I end up editing the original to fix a bug? Just keep scrolling up and down?

So, the questions are,

• How will both source generator authors and consuming developers be able to improve accessibility to generated files?
• And if so desired (how) will that work seamlessly and free of danger to overwrite user code when it is desired to write files to disk?

I don't think the MSBuild properties previously mentioned and the way they work today improve much on the picture in the PR and how Visual Studio works today. It's still 3 levels of nesting and it's only one global setting for all source generators. And, well, it doesn't seem to work the 2nd build.

[sharwell]

Does this mean IntelliSense should be working as expected as well?

Yes, it should be working. The cases where I'm aware of remaining "hiccups" are:

• If the source generator is defined in the same solution as the consuming project (as opposed to referencing a NuGet package), there are two limitations:

  1. You have to build the project once before the generator starts working in the IDE
  2. You have to build the source generator and restart the IDE after each change to the source generator so it will take effect in the IDE

  Both of these limitations tend to work themselves out because the source generator is not often edited after it is initially created, and as long as you don't do a full solution clean it typically keeps working.

• When you change solution configuration or platform, it can take a bit for the IDE to refresh its understanding of source generators, but it will typically resolve automatically aside from the same limitations as previous.

Is the current state documented somewhere up-to-date? Including like a matrix of VS, TFM, package and compiler versions?

Not that I'm aware of. I expect to see a lot of the remaining issues resolved in the first few minor releases of Visual Studio 2022 (dev17), at which point the majority of source generators should see long periods of consistent stability.

When I enable EmitCompilerGeneratedFiles I now get CS0101 and CS0111 for HelloWorld and SayHello respectively upon rebuild. Or is this considered an example of 3. above?

I would need to see the configuration to be sure. EmitCompilerGeneratedFiles isn't a problem by itself, but it can be a problem in combination with a customized value for CompilerGeneratedFilesOutputPath.

How will both source generator authors and consuming developers be able to improve accessibility to generated files?

I like the idea of being able to indicate that a source generator produces output from a single specific input, and in this case show the generated file as a nested file from it in Solution Explorer. Can you file a separate request for this specifically? It might be a while before we get to it, but it's a good idea.

And if so desired (how) will that work seamlessly and free of danger to overwrite user code when it is desired to write files to disk?

I didn't understand this question.

[ericwj]

Is the current state documented somewhere up-to-date? Including like a matrix of VS, TFM, package and compiler versions?

The samples use some msbuild or arcade magic to obtain the version of the compiler to reference. I guess the most important question right now is when to use 3.11.0-1.final and 4.0.0-1.final depending on a) whether I am on net5.0 or net6.0 and b) whether the VS used is 16 or 17 and c) something I overlooked?

I would need to see the configuration to be sure.

Yes, this is with <CompilerGeneratedFilesOutputPath>$(MSBuildThisFileDirectory)\Generated</CompilerGeneratedFilesOutputPath>.

I like the idea of being able to indicate that a source generator produces output from a single specific input

The relationship between the nested file and its parent should be logical - the source generator may still use any inputs it has available, not just that one file. I'll write something up.

And if so desired (how) will that work seamlessly and free of danger to overwrite user code when it is desired to write files to disk?

I guess I was just trying to tease you into designing something ;) I don't think it takes a lot of complexity neither for you to build nor for consumption, to make those guarantees mentioned and a few others as part of the larger vision in which source generators are at least able to completely align with existing code organization, greatly improving discoverability and navigation, even to integrate their source tree right at the root of the existing project.

But that requires letting go at least partially of this ideal world you are building where generated code is being buried. I can work out that vision at least partially in the issue to be filed regarding nested files, unless I didn't quite convince you of the merits of even going in that direction just yet. Maybe it deserves its own issue with a proper title to collect support of the idea. It appears I am about the only person in the world interested in this right now. For one because this issue still looks like a question.

[juancastrodlc]

Why would the compiler care what I check in? Not writing to disk or writing to GeneratedFiles will be a suitable default which can then be .gitignored and that probably has your requirements covered.

Our design may not include even running source generators in an environment that has a disk. i.e. we could easily opt for a system where SGs run in an extremely constrained sandbox where the SG only gets compilation-in and can only provide strings-out. The sandbox host would then manage everything from that point. So expecting or depending on the disk being present would explicitly put you in an unsupported place that could easily break at any point in the future (including post v1 launch).

This design decision even sounds contrary to the concept of Source Generator. It should write whatever output it generates somewhere unless instructed not to. These options should be accessible and easy to configure or even handled as part of the generation process. As it stands today there is nothing easy or obvious or afaik documented as to how to enable writing the generated code anywhere, and having to reach into the Analyzer module is cumbersome and un-intuitive. I had to read this whole thread to get to the answer as to where and how to get output of the generated code anywhere else. @ericwj You are not the only person in the world who would care about this. How do we get this issue out of question mode?

[CyrusNajmabadi]

How do we get this issue out of question mode?

To be clear, the above answer I gave is the design of it. Literally the concept of a 'disk' may not exist when running source generators.

This design decision even sounds contrary to the concept of Source Generator.

It isn't. The concept is that you can produce code and artifacts and the compilation and build system can interact with those. This is important as well as arbitrary disk access cannot be tracked by a build system, which breaks all sorts of things like tracking what may or may not need to be rebuilt.

[ericwj]

How do we get this issue out of question mode?

I was going to write a proposal for file nesting, but I haven't gotten to it yet.

For one because I try to play with all of this and learn, but the end result is quite thoroughly useless. I have no way whatsoever to see any generated code, worse, the error list fills up when I navigate to code referencing generated code, even though my over one-year old generator has been made to build with dotnet and in Visual Studio 17.

coffee it artifacts

What?

we m which breaks all sorts of things like tracking what may or may not need to be rebuilt

Isn't a simple <Compile Exclude> glob about half what is needed to work around this problem.

Literally the concept of a 'disk' may not exist when running source generators.

It is a bit tiresome to hear arguments for the design limitations of Roslyn as if it is a standalone system that apparently lacks proper abstractions to fluently enable the scenario being asked for where it applies - which is bound to be 99.99% of the time - and an engineer with the imagination to dream them up to work in both situations.

We are asking for a holistic solution which includes working with the compiler but not by itself. The build system is obviously there most of the time, operating from disk, on the command line, in a dumb IDE, as well as in Visual Studio when it doesn't work, which is right now in my experience permanently and more thoroughly so than 2 months ago.

[CyrusNajmabadi]

Isn't a simple glob about half what is needed to work around this problem.

No. A build system needs to know what was actually generated and what was actually consumed. Having arbitrary IO escape that violates those invariants. It's not acceptable to have a solution that says "the violation can be papered over later like so", rather the violation should not be allowed in the first place.

It is a bit tiresome to hear arguments for the design limitations of Roslyn

I don't know what to tell you. It's not an argument. I'm explaining a core part of the design here. Arbitrary IO is not part of the design of the system, and that will likely remain the case for the forseeable future.

We are asking for a holistic solution

Sure, i'm not opposed to a solution which addresses teh need, but doesn't go against this design constraint. My posts have been specifically about addressing the particular questions around writing things arbitrarily.

[ericwj]

Arbitrary IO

All of us are completely happy to stick to AddSource and let you figure out when, where and how to do I/O that doesn't trip up the build system. Afaik existing source code generators already use the .g.cs convention to distinguish between manually authored sources and generated ones and it entirely works beautifully.

But again you might not be able to do that reasoning from the compiler alone - the build system must cooperate or be made aware unless like you said nothing is ever written.

The need is to get those files right next to what they were generated from. Like I said, my generator compiles, loads and produces sources; the project referencing the project that uses the generator compiles using generated code, but since your artificial limitation precludes the simplest, most straightforward way of getting those sources on disk, I'm stuck with an IDE that actually breaks my project and zero visibility into what was generated.

Good job.

[CyrusNajmabadi]

and zero visibility into what was generated.

If you want visibility into what was generated i recommend two approaches:

1. (Most ideal): Just unit test it.
2. Pass generatedfilesout to the compiler. You can then examine a dump of those files for visibility purposes.

I'm stuck with an IDE that actually breaks my project

Please file any IDE issues you're running into. I'm unaware of any problems currently in this space, so we'll need good actionable reports that we can fix to address anything you're running into.

[ericwj]

Apart from building what I have I cannot realistically work on this project because as soon as I start editing, the error list fills up and it is like the generator hadn't ran at all.

1. (Most ideal): Just unit test it.

Isn't ideal at all. For one, I generate 10k lines and I have no clue what to unit test. Otherwise, even if all unit tests succeed but this is the result when doing it for real, I'm still unable to work on the project.

2. Pass generatedfilesout to the compiler.

I set EmitCompilerGeneratedFiles in the project file, which works, but isn't visible from Visual Studio except if I Show All Files and expand a whopping 7 nodes, filling up my solution explorer for half the height of a full 2180 pixels I have with noise. But still the code referencing definitions in those files is showing errors and IntelliSense isn't working.

The culprit is most likely that, although I have went through the loathsome, backwards and tedious process of downgrading my projects to netstandard2.0 (can we get rid of that, please...when?; why is this? you're telling me Roslyn is on netstandard2.0?!), as soon as I open files using generated code I get CS8032 complaining it cannot load System.Runtime, Version=6.0.0.0, .... I read through this and this issue, added SetTargetFramework, GeneratePathProperty, defined target GetDependencyTargetPaths, hacked the path together for the local project reference, closed the solution, deleted the .vs folder and reopened it, with the exact same result.

Again, the project builds, both on the command line and in Visual Studio, so I must've done all that correctly because I don't have errors anymore in the build output window complaining the source generator could not contribute source code or could not be loaded as I did before. Only the error list shows that CS8032 warning.

I am referencing Microsoft.CodeAnalysis.CSharp.Workspaces, Version=3.11.0 and Microsoft.CodeAnalysis.Analyzers, Version=3.3.2, Microsoft.Bcl.HashCode, Version=1.1.1 and a local project reference using that.

[CyrusNajmabadi]

But still the code referencing definitions in those files is showing errors and IntelliSense isn't working.

As mentioned, please file issues on these IDE problems you're having. We'll need those use cases so we can fix any of those things.

[CyrusNajmabadi]

For one, I generate 10k lines and I have no clue what to unit test.

Unit test the output of your generator. You'll not only get to see exactly waht you generate, but if you have issues, you can write tests and ensure no regressions as you make changes.

I set EmitCompilerGeneratedFiles in the project file, which works,

Great. :)

but isn't visible from Visual Studio except if I Show All Files and expand a whopping 7 nodes,

You're conflating issues. No matter what solution we came up with, we'd have to address the IDE side of it. If there's an existing solution, but the IDE side needs work, then we should fix that rather than introduce a new duplicative approach that will still need IDE work.

[ericwj]

You're conflating issues.

Maybe. I am still firmly of opinion it's harder than absolutely required right now. Maybe the issue is msbuild and add to that how Visual Studio uses that, but the end result is the same and the road to get there is very time consuming and awkward.

I take you didn't immediately spot any issues with what I wrote above. I managed to get rid of CS8032. Now all it does is complain it can't find the correct (generated) overload with CS0815 and CS1503.

Not sure how to report that short of copy/pasting the whole solution and stripping out what it actually does and posting that on Github. Should I use Report a problem for that? Feels like a detour.

[CyrusNajmabadi]

I take you didn't immediately spot any issues

Best would be to file an issue with a repro so someone can validate things directly.

[CyrusNajmabadi]

Not sure how to report that short of copy/pasting the whole solution

Whole solution is a fine starting place.

[CyrusNajmabadi]

but the end result is the same and the road to get there is very time consuming and awkward.

Fixing the existing capabilities sound happen first before inventing a duplicate way to do something that should already be there. Otherwise we'll just have multiple duplicated features, potentially all with the same issue your having with the existing feature

I am still firmly of opinion it's harder than absolutely required right now

Sure. But adding other mechanisms doesn't solve that. It may make things worse (for reasons I've outlined)

[ericwj]

Check this. You or whomever should see this repo referenced in a Developer Community issue. It again gives me CS8032 although the intent was to only remove stuff and not change anything from the solution this was created from, but alas. Remember the original solution doesn't do that. This one complains the [Generator] type cannot be loaded, although it still builds fine and apparently gives me IntelliSense in VSCode.

[CyrusNajmabadi]

@ericwj Can you please file an issue about whatever problem you are seeing? Thanks!

[ericwj]

You should see this repo referenced in a Developer Community issue.

Already did. Plus, clone this and try to fix RS1024. Nasty bug. I reported that, too.

[znakeeye]

How do I generate Namespace\HelloWorld.g.cs as a nested file of Namespace\HelloWorld.cs from which it was generated?

Currently, you don't. Source generators do not generate content on disk, so there are no files to show in solution explorer.

The request is interesting though. Currently the generated content is only shown under the Analyzers node, but it would be interesting to indicate that one or more generated files should be shown under specific other files.

I'm working on a pretty awesome partial class generator where I need this exact feature. E.g. FooClass.cs implies a partial implementation FooClass.g.cs. It's there in memory, and using <EmitCompilerGeneratedFiles>true</EmitCompilerGeneratedFiles> I'm able to see those generated files on the file system. Nice. But now I want to nest them. Are you saying this is impossible?

[Eli-Black-Work]

@znakeeye My understanding is that at the current time, this is not supported. It's possible to try to hack around this by having the source generator write files to disk, but this is explicitly not supported, and likely will break.

Do you have a specific reason for needing the files to be emitted to disk?

We have a similar source generator and are accessing the files via [project name] -> Dependencies -> Analyzers -> [your_analyzer_name], which works fairly well 🙂

[znakeeye]

Yes, I want to group the files using DependentUpon. For partial classes, this makes perfect sence. The desired output looks like this:

I already managed to get hold of the files. I just need to copy them to their right destination folders (where the corresponding non-generated file resides), otherwise DependentUpon will not work.

<PropertyGroup>
    <EmitCompilerGeneratedFiles>true</EmitCompilerGeneratedFiles>
</PropertyGroup>
<ItemGroup>
    <None Include="$(CompilerGeneratedFilesOutputPath)\**\*.g.cs"
          CopyToOutputDirectory="PreserveNewest"
          LinkBase="Generated\" />
</ItemGroup>