Champion "Replace/original and code generation extensions"

[gafter]

• [ ] Proposal added
• [x] Discussed in LDM
• [ ] Decision in LDM
• [ ] Finalized (done, rejected, inactive)
• [ ] Spec'ed

See also

• https://github.com/dotnet/roslyn/issues/5292 https://github.com/dotnet/roslyn/issues/5561
• An implementation branch is at https://github.com/dotnet/roslyn/tree/features/source-generators
• Use-cases by @alrz: Discussion: Code Generator Catalog
• Spec: https://github.com/dotnet/roslyn/blob/master/docs/features/generators.md

[MgSam]

Spec: https://github.com/dotnet/roslyn/blob/master/docs/features/generators.md

[MgSam]

@mattwar Are you guys still actively working on this feature or has it been tabled for now due to the implementation complexity?

[asdfgasdfsafgsdfa]

No news on this? Would be really helpful for a ton of things...

[MgSam]

I just read @MadsTorgersen 's design notes that said this feature is blocked from somewhere else not on the language end.

Could you guys please elaborate? Is the VS tooling the roadblock here now? In my view, this feature is far and away the most useful one out of the entire backlog for C#.

I know the brass likes to dedicate resources to issues that community puts a lot of votes on, so I've made a UserVoice issue as well. Please vote for this feature here and on UserVoice!

[CyrusNajmabadi]

Yes. VS tooling is the roadblock. This work will require substantial resources to get through the tooling side in any sort of sensible and efficient manner. It requires coordination between many different teams as well. As such, while still super great and valuable, it's unclear what the right timeline/delivery avenue would be for it.

[asdfgasdfsafgsdfa]

Sorry if this is a dumb question but what exactly does tooling mean here? The "debugger UI" in Visual Studio? What parts would need to get changed? Isn't this just a case of simply generating different sequence points in the pdb or am I completely off the track here?

Sounds like there's nothing the community can do, or is there?

[CyrusNajmabadi]

but what exactly does tooling mean here?

Everything about the experience in a product like VS. That includes, but is not limited to:

1. IntelliSense. Clearly people will want to generate items that then show up in IntelliSense. How does that happen efficiently?
2. Navigation. People will want to navigate to generated code. How do we do that, and what's the right experience overall?
3. Debugging. How do you debug through the code that is generated.
4. Project-System. How do you present generated code in the project system?
5. Refactoring. How do refactorings work properly in a context where some of the code they're analyzing was generated by previous code, but the changes they make may cause more transformations to happen?

etc. etc.

[iam3yal]

@CyrusNajmabadi Just for clarification it won't be strictly specific to Visual Studio, right?

[CyrusNajmabadi]

I'm not sure i understand your question @eyalsk . Can you clarify?

[iam3yal]

@CyrusNajmabadi I'm asking whether these features you mentioned like IntelliSense, Navigation, Debugging are going to be available in say Visual Studio Code or any other editor? as opposed to being available in Visual Studio only.

[CyrusNajmabadi]

The owners of those respective products would have to make a decision on that. :)

[iam3yal]

@CyrusNajmabadi Got'cha, thanks. 😉

[MgSam]

Why couldn't you guys just add the feature to the compiler and the IDE support will come whenever it comes? This is certainly the strategy TypeScript takes, where language/compiler features sometimes don't get VS support for months/years after they're introduced.

[CyrusNajmabadi]

Why couldn't you guys just add the feature to the compiler and the IDE support will come whenever it comes?

Because the IDE experience won't meet the user experience bar we've set for language changes.

Plus, it will just be a terrible experience. Think about it this way, if there's no IDE support, and you have a code generator that generates symbols that you try to reference, then you'll get tons of error squiggles that never go away as you're trying to use the IDE.

[MgSam]

What doesn't make sense is the arbitrarily different bar for different MS products:

• T4 has shipped with VS forever and IDE support for it is awful. Yet it is widely used both in and outside of Microsoft.
• As I said before, TypeScript regularly ships features with zero VS support until an indeterminate time later. VS isn't even the lead platform.

Basically, my point is C# language evolution shouldn't be constantly hamstrung by Microsoft's internal politics. The language team should be able to add features and let IDE support catch up (as it should be). That's how nearly every other open source language works.

Put the feature behind an opt-in flag until you're comfortable the VS support is up to snuff. In the meantime, JetBrains will probably have R# and Rider fully supporting within 3 months. If the feature is forced to wait for the stars to align inside Microsoft, it'll likely be a minimum 5-10 years before it actually ships.

[CyrusNajmabadi]

Basically, my point is C# language evolution shouldn't be constantly hamstrung by Microsoft's internal politics

It's not internal politics. We are the C# team. We decide what we feel is best for the language. And that includes insuring that it has a great development experience out of the box.

And there's a good reason for that. We don't want to ship the language changes too early, only to find out later that we royally screwed things up once we try to then get proper IDE support for it. SourceGenerators are a massively complex undertaking. We could try to get something out without considering and attempting to prove it would work in the IDE. But then it's highly likely we would get it very wrong.

Even our preliminary efforts here showed us immediately how our early source-generator designs simply would not be suitable to providing a good IDE experience.

[MgSam]

And that includes insuring that it has a great development experience out of the box.

As you say this you guys have just finished shipping a feature which is functionally broken out-of-the-box - tuples. It requires you to know that you need to go to NuGet and search for a specific package in order to use it. The VS feature which suggest packages to download is turned off by default.

Dustin was demoing replace/original last year, so clearly some form of it is possible to ship without the sky falling.

It's also ironic that you're the one making this argument, Cyrus, when not long ago you were on the TypeScript team helping to ship new features every couple months.

[iam3yal]

@MgSam I agree with you on some of your points but I think that they probably don't want to make the same mistake twice and the experience I had with T4 from what I can remember was pretty awful to the point I removed the extension I wrote from the VS gallery. :)

[iam3yal]

@MgSam

As you say this you guys have just finished shipping a feature which is functionally broken out-of-the-box - tuples. It requires you to know that you need to go to NuGet and search for a specific package in order to use it. The VS feature which suggest packages to download is turned off by default.

Didn't they fix this for RTM?

[CyrusNajmabadi]

Dustin was demoing replace/original last year, so clearly some form of it is possible to ship without the sky falling.

Yes. That was a demo. It was woefully incomplete. It worked in extraordinarily small scenarios. It had serious issue that we did not (and still do not) have solutions for real projects. We do not ship prototypes just because they're cool and they demo'ed well. A lot of real effort went into exploring this feature. Even just the language feature alone had a whole host of issues that we did not know how to resolve. And that was before the very first problem of: how will this even hook up into the IDE compilation flow model?

[CyrusNajmabadi]

Didn't they fix this for RTM?

No. But we're working on it. The core problem is that there have been extremely hard perf goals to meet, such that even loading a single extra dll is not allowed. That's problematic in the context of any sort of extensibility story. We made a hard decision to ship in that state, even though we would have preferred it be better.**

And that goes to my point. That was something we did not want to do, and it was only because the scope of the problem was so limited. i.e. you would get an error about tuples, but at least you could solve it.

That doesn't even begin to compare to the issues you'd get with source generators. Again, imagine it was like tuples, and you got lots of errors in your IDE while trying to work. Except now there wouldn't be anything you could do about that. Also consider that just typing things like "replace" and whatnot would simply throw of the IDE parser completely, leading to code that was nearly impossible to edit.

Seriously, if you want to get a feel for what it would be like, go back to something like 2013 and try to type modern C# in it and see how quickly the IDE experience goes off the rails. Just basic editing will fall to the floor because we won't understand teh code you're writing on a syntactic and semantic level. And, currently, our IDE is based around the idea that we will understand at least that much.

--

** We've won back enough performance across the board though that we feel lke we can get an exception here and that this feature can be enabled soon. If/when that happens, it will mean there was a short window of time when we had a not-great experience. That's not at all comparable to the types of timeframes we're talking about for something like SourceGenerators.

And again, it's not like we can "just ship source generators now and just make the IDE work later". We have to understand the needs of hte IDE so that we can properly design source generators in the first place. We know this first hand from other work we've done in the past, including work in analyzers. The original analyzer work did not think closely enough about the compiler/IDE interaction, and as such had absolutely unacceptable performance characteristics. By designing things together, we were able to come up with something that met our feature goals while also having acceptable performance.

[CyrusNajmabadi]

It's also ironic that you're the one making this argument, Cyrus, when not long ago you were on the TypeScript team helping to ship new features every couple months.

There's no irony here. Nor is this an accurate representation of what actually happens in TypeScript. In TypeScript features are shipped and the TS team themselves does update the LS side of things at the same time as they're doing the compiler work**. It is the case that they can release a compiler that people can use before an official VS build is made available, but that doesn't mean that they're shipping language features that they then might not have any clue how to the IDE work for.

That's what's different here. IT's not like we know what source-generators are at a language level, could ship that, and could then just "do the IDE work". We do not know what source-generators are precisely because the design of the language feature itself is so informed by the tooling that will consume it.

--

** I know. I was involved with it for several years, and i still work closely with them as they're an adopter of Roslyn and an early adopter of lots of the improvements we make. I wrote a ton of both the LS features for TS as well as the majority of the LS implementation in VS. I also wrote the extensibility points in Roslyn to allow TS to become a Roslyn language.

[MgSam]

Yes. That was a demo. It was woefully incomplete. It worked in extraordinarily small scenarios. It had serious issue that we did not (and still do not) have solutions for real projects.

Totally fair. So there's a lot more work to be done. I just don't buy the argument that it's impossible to do the work unless 6 other MS teams first jump on board. Good software/API design doesn't require explicit buy in and support from every user before you proceed. If it did, no software would ever get shipped.

I use PostSharp- which accomplishes many of the same things as this feature. In the first versions had nearly zero IDE support and the support still sucks pretty bad. That doesn't mean it's still not super useful.

Like I said, do your best effort, put it behind a feature flag and let your partners implement IDE support themselves. I still haven't heard a reason why making an optional feature that's turned off by default is problematic. Again, you guys do this constantly in TypeScript.

Seriously, if you want to get a feel for what it would be like, go back to something like 2013 and try to type modern C# in it and see how quickly the IDE experience goes off the rails.

Except if I'm using R#, in which case it'll probably work perfectly. Again, my point that language design shouldn't be hampered by MS politics.

TS team themselves does update the LS side of things at the same time as they're doing the compiler work

This is completely off on a tangent, but non-nullable types were released in TypeScript 2.0 in September. The --strictNullChecks flag still isn't supported in VS 2017, over 6 months later.

When TypeScript first came out, you had Mads Kristensen adding capabilities on his own into VS (like compile on save) because no one else was doing it. The TypeScript-VS team coordination has always been dysfunctional. Yet TypeScript has been a huge success nonetheless. And you guys should be applauded for pushing the language forward without waiting for VS.

---

I don't mean to get into an extended back-and-forth over this, but it's super-frustrating to see useful features like this continually delayed indefinitely so that the team can instead turn to syntax sugar features that have little real-world impact on code quality or productivity (like the entire C# 7.0 release). And the reason is always the same- "this feature requires *** team inside Microsoft to cooperate so let's delay it indefinitely"

Seriously, you guys do a better job cooperating with the Angular team on TypeScript then you do cooperating with other teams inside of Microsoft.

So, in my opinion, the C# team needs to stop gating progress on other MS teams and move ahead doing your best to design features and APIs. If it ends up needing changes for native VS support to work later on so be it- that's agile software development.

[CyrusNajmabadi]

I still haven't heard a reason why making an optional feature that's turned off by default is problematic.

I have given the reason:

Because we literally don't know how the feature should work. We don't know what the right design is for it. How do we even create it if we can't figure out these questions :)

I just don't buy the argument that it's impossible to do the work unless 6 other MS teams first jump on board.

Good thing i didn't make that argument ;-)

The argument is that the C# team itself (i.e. us) are not willing to ship the feature when we can't deliver even a basic experience that is satisfactory. Nor are we going to just ship the language/compiler feature if we don't even have an idea for a design that we think could be sufficient.

and let your partners implement IDE support themselves.

And when the partners (i.e. the C# team) says to the C# team: "uh... there's no way we can get any sort of acceptable experience here with the design you shipped to customers..." then we're in a big heap of trouble. :)

And you guys should be applauded for pushing the language forward without waiting for VS.

Again, TS does not create language/compiler features that they don't know how they'll integrate into their LS. I'm not sure how much more plainly i can state that.

And the reason is always the same- "this feature requires *** team inside Microsoft to cooperate so let's delay it indefinitely"

Please stop saying "** team" :). The reason here is that the C# team itself does not know what the right design is for thsi feature. We don't know what it is and even as we experimented and prototyped many options here we ran into enormous issues (even at the language/compiler level) that we simply had no solution for. On top of that, the C# team itself does not want to ever ship language/compiler features that it doesn't know if it will be able to deliver a suitable developer experience around. That is a core goal of our language, and we believe it is very important. Right now we could come up with literally zero designs that we could feel confident that we'd be able to provide even a basic experience around, let alone a good experience. It would be enormously* irresponsible of us to ship something like that at the language level, especially with the expectations people have had from us for the last decade+.

Seriously, you guys do a better job cooperating with the Angular team on TypeScript then you do cooperating with other teams inside of Microsoft.

It's not other teams. We are the C# team. The C# team has not been able to come up with a satisfactory design for this feature, and also feels that there are enormous complexities and problems that would be extremely difficult to deal with for the developer experience.

--

I feel like you're thinking the situation is one of: "Oh man... we have this amazing feature. And we totally know how to do it. But those pesky devs over there are preventing it from going out! :( ".

That's not what is happening here. We simply do not have a design that we feel actually delivers the right feature to users (let alone an implementation to go along with that design). And we are never going to ship something that we can't even come up with a design for.

--

If it ends up needing changes for native VS support to work later on so be it- that's agile software development.

Sorry, the rules of C# and the promises we've made over the years still apply. Backcompat is still a thing. We're not going to just throw this feature out there, then realize it is full of problems, and go "sorry, gotta break everyone, that's agile software development". We are the language and platform. We get that this would be hugely useful. But that's all the more reason to try to get something good that can last before shipping things out to people.

[CyrusNajmabadi]

Except if I'm using R#, in which case it'll probably work perfectly.

No, it won't. I've tried future language features in older resharper products, and it doesn't work well.

Again, my point that language design shouldn't be hampered by MS politics.

The C# team sets its direction here. We have decided as a team that we did not think we had a good enough design for this feature. And we believe that we should not be shipping features that don't have good designs. That is not a political decision. That is a decision borne of our team principles.

--

Note: TypeScript has similar principles, and this is not something they are exploring either precisely because they do not have any sort of design that they think is good enough for this sort of scenario.

[mattwar]

@CyrusNajmabadi is correct but leaving you guys with the wrong impression. There are serious problems to solve before source generators can be a feature. However, these are not intractable problems. The C# and related teams just needs to make it a priority for it to happen. There are many other important things that are consuming their time right now.

[vbcodec]

@CyrusNajmabadi

There are no tooling roadblocks.

You can implement replace / original without source generators. Third party generators that use replace / original will appear pretty quickly.

If you want to implement source generators, you must lower your requirements and apply some simplifications. Let developer decide for which files generated code will land. Generators do not need to be fired on every keystroke. It may be fired only for adding / removing / changing attributes, and for other purposes they may be fired manually, by pressing some button or combination of keys. You may also set the rule, that any changes to generated code won't fire generators. There is also no need for any highlighting files with generated sources in Solution Explorer. All of this will be fine for generators V1. You may gradually extend this in subsequent versions.

You missed the point that developers want that auto generated code (hidden and non modified in most cases), and not superior and highly polished tooling.

[asdfgasdfsafgsdfa]

Cyrus didn't say that they're looking for some super highly polished tooling. He said that there are very basic problems that still need to be solved, and that without them the feature would be almost (or completely) unusable.

Frankly I sometimes disagree with Cyrus but you gotta admit he makes some valid points here that can't be easily dismissed and that he puts in some real effort to make us understand.

Seeing this feature get delayed (again/indefinitely) is really sad, but I think the problems can at least be approached if a few knowledgeable people come together and brainstorm about different approaches.

Maybe re-imagining how parts of this feature would work (maybe only trigger generators on a full rebuild)...

Only one thing is really clear here there is still a ton of conceptual work to be done, ideas and solutions to be found - and if the team wants to bring this feature to life at some point I'm the future, they have to commit a little time every week to thinking about the still open problems.

Anyway, thanks for the explanations. I am sure I speak for all when I say we really do appreciate it even if we often don't show it and keep arguing 😊

[agocke]

To add to what @mattwar is saying -- it's not my highest priority but I am still actively thinking and planning about how compiler+IDE support can be made to work for this feature while still meeting all of our standards for high quality tooling.

I have solutions for some problems, but they involve limiting the feature in ways that some people may find overly restrictive. For instance, I think one of the constraints that would really help the compiler + IDE from the performance perspective is making generation triggers entirely syntax-based. This forces every generation point to be explicitly annotated, probably with a property. In addition, if the scope of generated files is limited to members on that class, we can have more optimization opportunities on 1) when generators need to be run and 2) what needs to be reevaluated when they are run.

If we were to release this feature earlier this is the kind of approach I would suggest -- provide support for a limited number of well-tested scenarios and make sure that end-to-end design is wired up ahead of time. The problem is that the feature was previously spec'd so broadly that this tends to disappoint a lot of people when they figure out it's not flexible enough to meet their scenario, at which point they'd rather we have worked on a different feature or spent more time to cover more scenarios.

In short, it's just like any other software development project with a lot to cover and not enough time to do it all. 😉

[axel-habermaier]

@agocke: I understand the difficulties, still it's disappointing to see that source generators do not have priority at the moment. I think this is the feature that could easily have the biggest impact on my daily work with C# compared to anything else currently planned!

[Athari]

This discussion is a little too generic:

"There're problems." "No, there're not."

Let's be practical and discuss real problems rather than priorities, teams and other less relevent things.

So, based on discussion here, I know about these real problems:

1. IntelliSense. Clearly people will want to generate items that then show up in IntelliSense. How does that happen efficiently?
2. Navigation. People will want to navigate to generated code. How do we do that, and what's the right experience overall?
3. Debugging. How do you debug through the code that is generated.
4. Project-System. How do you present generated code in the project system?
5. Refactoring. How do refactorings work properly in a context where some of the code they're analyzing was generated by previous code, but the changes they make may cause more transformations to happen?

Let's remember that source generators already exist. XAML generates control fields (btnOK etc.), their initialization (InitializeComponent) and x:Bind bindings in UWP version (UpdateBindings or whatever it's called and all related crazy code). All this code is put into a separate source file inside preliminary output folder. Besides XAML, ANTLR comes to my mind which functions similarly. T4 generates files in a different way.

XAML and ANTLR are already broken. My experience is based on VS+R#, but I suspect VS tooling without R# isn't better.

1. IntelliSense. It works. For some time, ANTLR files in the output folder were ignored by R#, if I remember correctly, and files were full of "errors". It was fixed later.
2. Navigation. If developers know the "secret" names and press F12, they can navigate to the generated code. Otherwise it's hidden.
3. Debugging. XAML and ANTLR experience doesn't translate to replace/original experience, but there's another analogy: skipping auto-properties, skipping "not my code" in debugger options. I suspect that "skip replace code" option would be appropriate and not too hard to implement. Otherwise debugger will jump like crazy between parts, not display all locals and whatever - it can be improved later, but I can survive even if it never works.
4. Project-System. Generated files are hidden in the project. Imperfect, but acceptable. The way T4 works with generated files attached to the input files is better though.
5. Refactoring. Refactoring is already broken in case of T4. If you refactor something generated, T4 file is left untouched. Running it results in broken code. I guess the sequence "refactor, save original, regenerate replace" should be fine. If it results in broken code or partial refactoring, I'm fine with that and can fix the rest manually.

Overall, I experience most of these problems already, granted at lesser scale. Replace/original would give me more power at expense of even more problems. I can accept that. Every developer can choose whether to use replace/original for reliable and well tested simple scenarios like INPC and logging based on source generators written by MS and community, or try to do crazy things themselves, and receiving all sorts of problems in return.

Sorry if I oversimplify problems. I've never installed alpha versions of VS and haven't even been part of the original replace/original discussion.

[jnm2]

@Athari thanks for helping the discussion get back on track. The best thing for the community to be involved in at this stage is finding solutions to these difficult problems.

Re: project system. For the use cases I have in mind, I'd like the option of not emitting generated files to the file system at all. The reason for this is source control noise. Rather, syntax trees would be created in memory as the compilation happens. Debugging/go to definition would create the text on demand, just like I'm used to ReSharper doing when debugging/F12ing 3rd party library code to decompile. How feasible is this?

[vbcodec]

Maybe language is wrong place to implement AOP-like facility. Better solution is to extend Type class and allow to register handlers that are fired when some action is performed: new instance of class is created, before calling function etc. This will give also possibility to AOP-ing compiled libraries. Disadvantage is that this will be pretty raw, without any tooling support and performance may be bit slower than compiled code.

[CyrusNajmabadi]

You missed the point that developers want that auto generated code (hidden and non modified in most cases), and not superior and highly polished tooling.

That's not the argument i made. I made the argument that in order for us to ship something we'd have to at least address some very real problems.

A good example of a very real problem is the "diagnostics problem". If source generators are not involved in generation in the IDE (because you just shipped them but did nothing to hook up the tooling), then now you have the issue that you may get literally thousands of errors showing up because the tooling doesn't realize code exists which will exist at runtime.

Another example of a very real problem is that if you just naively enable source-generators in the IDE then you dramatically change the performance characteristics of all features. For example, today in many IDE features we code as if we getting semantic information is cheap (because it is in Roslyn's current design). As such features can depend on it being available, both:

1. immediately and
2. accurately.

The moment source-generators get added then that equation goes out the window unless you place very significant constraints on generators (which is possibly what we need to do). You can no longer get information immediately, as source-generators may take unbounded time to run. Or, if you want immediacy, you may not be able to get information accurately as you won't have the information from the source generator.

You've now changed a core design principle that every IDE semantic feature has been designed and implemented based on. That's not something trivial that can be ignored. Many customers may not like the hit in semantic accuracy that comes wit providing immediate results. And many customers may not like the perf hit that comes with trying to get correct semantic data.

As i mentioned before, we may be able to address this by putting in appropriate constraints within the core design of SourceGenerators. But that supports precisely the argument i've been making from the beginning: that we do not want to just create something and throw it out there without strong and careful consideration of the tooling impact. Instead, we want to consider what design choices we could make so that we don't end up situations where the tooling has not capability to provide even a basic experience because the core design makes that too difficult.

[Athari]

For the use cases I have in mind, I'd like the option of not emitting generated files to the file system at all. The reason for this is source control noise.

@jnm2 If files are written to output folder like in case of XAML and ANTLR, the files are generated but ignored by VCS. Less magic, already implemented, already works. The T4 way with generated files added to the project and VCS is clearer and I'm more used to it, but I see the obvious problem with it - ideally, generated code shouldn't be put into VCS as it's redundant. I guess I don't have a strong preference here.

Your approach with syntax trees in memory is interesting, but it would require so many changes across so many components that it increases the price of the feature too much and it's already high enough to be put on hold.

Or, if you want immediacy, you may not be able to get information accurately as you won't have the information from the source generator.

@CyrusNajmabadi Won't running source generators on file save just like T4 be an acceptable compromise? (T4 also runs on tab switch which may or may not be desired.) If a source generator is very popular, more sophisticated code completion support for it may be added later.

Let's take XAML for example. It took a decade for XAML designer and XAML code completion to approach usable state. Or tuples - while VS outputs a weird error about missing type during compilation, ReSharper suggests to download the NuGet package when it sees tuples in the code. And T4 is still not supported by VS. At all. After all these years. There're only commercial solutions for T4, one worse than another.

Maybe many customers do want perfect immediate code completion for source generator magic, but I'm pretty sure they can survive without it. Maybe many customers do want perfect performance, but there're enough developers who accept the price of having two syntax trees and analyzers runnings at the same time in the same process (Roslyn + ReSharper). Customers' expectations aren't that high.

[CyrusNajmabadi]

Won't running source generators on file save just like T4 be an acceptable compromise?

Don't know. We'd have to actually try it out and run through all our scenarios and work with lots of customers to actually see. :)

[iam3yal]

@Athari

Maybe many customers do want perfect immediate code completion for source generator magic, but I'm pretty sure they can survive without it.

Of course we want it to be perfect but no solution would be really perfect, there would be trade offs but the question is whether no code completion now, means no code completion tomorrow... what if they settled on a design where some of their goals were missed or some scenarios weren't covered? what then? what do you do? go back to square one and start over?

We survive without source generator today, we will probably survive without it for few more years... I prefer to to wait and let them do it right then having a rushed solution and do it wrong.

[CyrusNajmabadi]

Also, thinking about "Won't running source generators on file save", i'm not sure that would be ok. So someone uses source generators, and makes a change. They then want the change to be visible in intellisense. So they save. And now save take 20 seconds. That seems super unpleasant.

[Athari]

We survive without source generator today, we will probably survive without it for few more years... I prefer to to wait and let them do it right then having a rushed solution and do it wrong.

Let's be realistic. Non-nullable reference types were planned for C# 7. Didn't happen. There were issues related to generics, weren't there? I didn't follow closely.

Record types and primary constructors were planned for C# 6. Didn't happen. Didn't happen in C# 7 either. If I remember correctly, C# 6 turned out to be syntax sugar-only release, and record types were seen as part of pattern matching which was too serious for C# 6.

Pattern matching was planned for C# 7. Or was it for C# 6? Either way, pretty much didn't happen. Out variables and old switch upgrade can be seen as syntax sugar at this point.

So, just from major features which weren't delivered for C# 6 and C# 7, we have pattern matching, record types, non-nullable reference types... All of them seem to be much more clearly defined, they have less problems, they require less cooperation between teams and so on.

Judging by how things go since C# became open-source and its progress can be seen at all stages, these major features will occupy developers till around C# 12: delayed, split, cancelled etc. Judging by issue milestones, replace/original comes after them. So, it won't happen in the foreseeable future. Let's just move on.

[Athari]

@CyrusNajmabadi

Also, thinking about "Won't running source generators on file save", i'm not sure that would be ok. So someone uses source generators, and makes a change. They then want the change to be visible in intellisense. So they save. And now save take 20 seconds. That seems super unpleasant.

If a refactoring touched dozens of files then running source generators for 20 seconds is annoying but kinda acceptable. The refactoring likely took more time.

If you mean a change of a single file of a sensible size can cause source generators to run for 20 seconds then I find it hard to imagine what these source generators do.

Then again, if it's actually needed and works that way in some cases, maybe it would be time to implement an option in VS on when to run source generators (and T4 too): on save, on tab switch, on timer, only manually. I'd love that option even now.

[CyrusNajmabadi]

can cause source generators to run for 20 seconds then I find it hard to imagine what these source generators do.

Right now (with the basic proposals we've had), they can do whatever they want. For example, a SourceGenerator might run any sort of complex analysis to determine what it should do. Someone could have a system whereby the SourceGenerator went and hit an internal service that responded with schemas that they then generated code from. They might go and talk to the Office Automation model to generate code based on business logic encoded in Office docs.

If you think any of this sounds "out there" then remember that a very similar system ("Type Providers") is in F# and people can and do use it for this sort of purpose. And there are issues with perf over there. :)

The idea of arbitrary codegen is very attractive. After all, you could literally solve any problem with such a flexible system. However, with that flexibility you do find that people end up doing these sorts of things, and then are unhappy when the surrounding tooling can't keep up.

[CyrusNajmabadi]

maybe it would be time to implement an option in VS on when to run source generators (and T4 too): on save, on tab switch, on timer, only manually. I'd love that option even now.

Yes. That is the sort of work that would have to happen in the tooling. These issues have to be examined and addressed. And it would not be appropriate to just create source generators without examining ane understanding what we would need to do in teh tooling to provide a good enough experience.

[m0sa]

Since it's not mentioned in this thread yet, and to add to @Athari's list of existing "source generators", we shouldn't forget about Razor. It takes .cshtml view files, and generates C# code that gets compiled either at runtime or at MVC pre-compilation time.

• The editing experience has been broken there for quite a while now. Find all references, refactorings etc, were very limited w/o the use of a 3rd party tool (e.g. R#).
• The fact that it also worked with dynamic models, view bags, etc didn't help either...
• Debugging worked by mapping the generated C# code to the originating .cshtml file via #line directives

IMHO a big issue with debugging and having more tools be able to layer on top of tools like Razor is the "source mapping" point. Having a bunch of #line directives inside the generated source is very cumbersome. We should be able to encode that information into the compilation itself / have structured source mapping data on the compilation or source text itself.

[paulomorgado]

@MgSam,

Except if I'm using R#, in which case it'll probably work perfectly.

Try counting the number of questions posted on StackOverflow by blind followers of the R# religion complaining about problems in Visual Studio 2017 that are actual caused by R# not understanding C# 7.

[gulshan]

Code generation in it's current form using "replace/original" -

• Does code generation in a separate step from compilation.
• Generate human readable source code files.
• Can alter pre-existing types.

All these adds pressure to the tooling. If we try to re-imagine the concept, how about explicit code generators, that take any type and programmatically generate a new type during compilation? This may look something like-

private class PersonBase(string Name, int Age);
public class Person = NotifyPropertyChanged<Log<PersonBase>>;

Looks like generics, which operates on types rather than objects. With this, I think the tooling issues related to generated source code will be gone. But it may feel like magic during debugging I guess, which have to be figured out. F# may do something similar- https://github.com/fsharp/fslang-design/blob/master/RFCs/FS-1023-type-providers-generate-types-from-types.md

[iam3yal]

@Athari I get where you're coming from and there are many people here that were pretty vocal and disappointed about the fact that features didn't make it into C# 6 and 7 and that's totally fine but I also understand the design team, they can't do everything, things that are high on the list are more important by virtue, things that are urgent are more important than others and major features usually pushed to major releases so if they can't finish something in the current major release it will get delayed to another major release, especially when it comes to features that require more iterations and design work.

Really though, isn't this what we do in our own applications? design, iterate, repeat until we come up with a better design and sometimes, we might end up trashing the idea/feature completely... :)

I understand that people want it asap because they think that this will solve some of their problems, the issue with this thought process is that if they would settle on a design that is only half good and/or cover half of the scenarios, it might not cover the scenarios you need and it might not be possible to change it later and finally the experience might be bad so they invested quite a bit of time on "nothing".

[vbcodec]

@eyalsk Demand for AOP features and generators is high. Postsharp, Fody and other tools aren't popular without reasons, and lot of devs cannot live without these services. Many others are making enormous amounts of boilerplate code manually, and desperately thinking 'why someone do not automate these tasks'. On the other hand, records and null-check are just minor improvement to what already exists, and. async streams do not seems to be highly demanded.

Definitely, team's priorities are not in-line with demand. I understand that they cannot rapidly switch to create new big feature. But they can move / hire more devs or outsource job, to make more features in parallel, especially that amount of rejected features are constantly piling up. It's kinda shame that company is making billions $'s profits quarterly, and don't spend more to 'most important company's language used by millions of developers'

[iam3yal]

@vbcodec

Demand for AOP features and generators is high. Postsharp, Fody and other tools aren't popular without reasons, and lot of devs cannot live without these services.

I will disappoint you with my answer but you're basically telling me that for some scenarios a solution exist and that source generators is not a deal breaker because devs use these services today and my guess is that they can continue to do so for few years more.

In fact, my guess is that many people wouldn't switch to source generators just because C# has this feature built-in and the reason for it has to do with benefit and costs, they might gradually and eventually replace it with their current tech but not necessarily.

Many others are making enormous amounts of boilerplate code manually, and desperately thinking 'why someone do not automate these tasks'.

So why don't they? making specialized solutions is a lot easier than making a general tool, I've written my own ORM before tools like NHibernate and EF existed... why people need to wait to automate things?

On the other hand, records and null-check are just minor improvement to what already exists, and. async streams do not seems to be highly demanded.

So now you get to decide what's more important? I don't understand how you can make that claim and based on what facts?

Definitely, team's priorities are not in-line with demand.

Yes, in your opinion, I don't have enough data to make this claim and even if I had I don't think that features are prioritized based on what the majority of people want but based on what they need, now, clearly it's needed and we're not the only ones to need it, there are various Microsoft tools and frameworks that can take advantage of this so I think they want it as much as we do.

I understand that they cannot rapidly switch to create new big feature. But they can move / hire more devs or outsource job, to make more features in parallel,

Do you understand what goes into hiring people into enterprise? more people doesn't necessarily mean more features and many times it's a process so it can take many months not to mention that they made it clear that the problem is with the design of the feature so hiring more people to implement nothing at this point wouldn't get anything done.

especially that amount of rejected features are constantly piling up.

So you expect them to accept most features? I mean, honestly, what do you expect?

It's kinda shame that company that is making billions $'s profits quarterly, and don't spend more to 'most important company's language used by millions of developers'

It's a shame that people make assumptions that don't help the discussion. :)

[CyrusNajmabadi]

The metadiscussion about team priorities is really offtopic and needs to end (or be taken elsewhere). We pick the things we think are important, and we do them when we think we can get the level of quality we think is necessary.

This will never satisfy all needs and demands of our entire ecosystem. That's why we continuously have version after version.

This is the last that needs to be said about this here in this thread.

Thanks :)

[amis92]

So back on track for the actual feature discussion: it seems to me that the biggest problem right now is tooling design around the generated files themselves. When we concentrate on just C# side of things, there are really just two things to do:

1. support two new contextual keywords original and replace
2. decide on implementation of replacement order

I really think generators could be a big feature, comparable to async, LINQ and others that define C# as it is today - but maybe taking it in steps could help deliver something a little sooner. So maybe if C# decides to support these two syntactical aspects a little in advance, the tooling will follow at some distance.

So that's my suggestion: focus on delivering support for the replace/original and replacement order. Split the feature from Source Generators.

I think the question that'd follow is "what scenarios does that help if there are no source generators to start with"? Well, those same scenarios we already need partial for today. It just increases power of tool-generated code, be it for Source Generators or other tools. EF, XAML, many others - they could definitely use enhanced scenario provided by replace/original. It also wouldn't be the first feature to be split: pattern matching was split and what we have right now is a very uninteresting shortcut for just a couple scenarios - but we can get used to it and we wait for the actual big thing (match expression).