Announcement: Razor Compiler API Breaking Changes

333fred commented 1 year ago

Over the past year, the Razor team has been on a journey to improve performance and reliability of the compiler, paying down technical debt, reorganizing infrastructure to enable performance tracking and faster shipping, and restructuring compiler internals to reduce complexity and improve compiler performance. While we've had a good amount of success in this area, architectural decisions made for the current version of the compiler and maintaining compat with these decisions are limiting our ability to make the even bigger improvements we need to make to deliver a good Razor experience. Therefore, we will be freezing our public APIs and the packages we publish for the compiler, and we will publish new packages in the future with public APIs we can support going forward.

High Level Changes Overview

Before we get into the details of why we need to make this change and the specifics of the timeline, we’d like to give a high level overview of the change:

We will be working on a new public API more closely resembling the Roslyn model, with goals to improve performance and enable new functionality such as analyzers and refactorings in Razor files. When we are confident in our public API for these scenarios, we will start publishing new Razor compiler packages.
Existing Razor compiler packages will not have new versions published. Existing versions will continue to exist and be serviced according to the standard .NET servicing guidelines. You can continue consuming these packages for your existing scenarios without any changes.
As part of this there are no changes to how, and where, we build the Razor compiler. The code remains open source, and we welcome community contributions in the form of issues, PRs, etc.

The rest of this issue will cover the specifics of why we’re doing this, the timeline of the change, and the exact packages this will affect.

Incompatible Design Goals

The modern Razor compiler was designed to be extremely extensible: users can substitute their own versions of many components of the compiler pipeline, using the Razor HTML templating syntax with their own embedded language and emitting to their own languages. While this was an important part of the initial vision of Razor, it does mean that the public API components of the compiler are extensive, and rearchitecting how compilation phases interact with each other, the information each phase is responsible for building, and the format of the output of each phase is potentially a breaking change. Some examples of where the existing structure impacts our experiences are:

Semantic information about the project the Razor file is embedded in is exposed in the syntax tree model itself. Syntax types such as MarkupTagHelperAttributeSyntax depend on knowing what tag helper types exist; this means that, with the current architecture, Razor files cannot be parsed without crawling the entire containing project for tag helper types. This then impacts our ability to give even basic syntax highlighting before a significant amount of expensive work on the containing project has been done.
Another parser-related architecture component, error recovery becomes incredibly complex (even more so than it already is). Because any language can be embedded within Razor's HTML markup, it limits our ability to solve issues like https://github.com/dotnet/razor/issues/7608, a top issue for Razor users. In order to handle error recovery for these cases, the HTML and C# parsers will need to work in tandem and be more tightly coupled than they are today.
The existing compiler phases have a number of structures embedded in the public API that have proven inefficient. Some of these could just be deprecated and eventually removed, but some of them, such as the structure of the tokenizer and passing of semantic information, have core issues and will need to be reworked to be usable.

Given these and other similar issues, we no longer believe that the original goal of allowing any embedded language to be used with the Razor HTML components is something that we can support going forward. Blazor support already started down this path, as it deeply integrates the Razor syntax with C#, and we think this provides a good experience for users. We will be focusing on making the Razor + C# experience as awesome as possible and addressing the above issues will be key to this effort.

New Design Goals

Despite this change, the Razor team does believe in the power of public compiler APIs and tooling based on them. Many of us that work on the Razor team also work on the C# compiler and tooling teams and have first-hand experience with awesome features that can be enabled by good public compiler APIs (indeed, many of us have been part of the design of the APIs that analyzers and source generators use today). We would like to enable a similar level of API for interacting with Razor files: being able to interact with a Razor syntax tree, query semantic information about the tree, and getting information about embedded C# as well. We hope to enable writing analyzers and fixers for Razor files, much like people can write analyzers and fixers for C# files with Roslyn today. It will take some time for us to get to this point; restructuring a hugely popular compiler without breaking users isn't an overnight task. But the new structure will be built with this goal in mind.

Timelines and Actions

Currently, the Razor compiler and supporting packages are published to Nuget.org, in addition to being included with the .NET SDK and VS. This will change:

VS 2022 Version 17.6 - We will not publish new versions of the existing Razor compiler and source generator packages on Nuget.org. They will still be present in VS and the SDK in the same shape as they are currently to support old editing experiences, but they will not receive new changes. You can continue using these packages for your existing scenarios, but they will not be receiving new Razor features. These packages are:
VS 2022 Version 17.7 - We will restructure the dlls that the compiler produces into new names, likely Microsoft.CodeAnalysis.Razor.Compiler, consolidating our different dlls. This new package will start receiving breaking changes as soon as our partner IDE and runtime compilation experiences have absorbed the changes. This package will not be published to Nuget.org yet, as we make major breaking changes to its internals.
VS 2022 Future - When we have restructured the compiler, worked out the kinks, and designed/vetted our public API structure, we will start publishing the new Razor compiler package to Nuget.org.

Of course, these packages will be available on our dev feeds, for adventurous users, but they will come with breaking changes, possibly even breaking new APIs as we continue restructuring.

For those who made it all the way to the end, thanks for reading. These changes, while big, will help us bring a better experience for Razor consumers, and a more usable public API.

gerneio commented 1 year ago

When you say:

...we no longer believe that the original goal of allowing any embedded language to be used with the Razor HTML components is something that we can support going forward.

Just to clarify, are you stating that the future razor compiler will NOT support embedded CSS/JS and ONLY support HTML & C#?

davidwengier commented 1 year ago

Just to clarify, are you stating that the future razor compiler will NOT support embedded CSS/JS and ONLY support HTML & C#?

A good question, but no, that's not what this means. To the Razor compiler right now, CSS and JS are completely opaque: They are just content that happens to be in a HTML tag, namely a <style> or <script> tag. Likewise inline styles or JavaScript happen to be content in an attribute value. Nothing about that is changing, so CSS and JS will continue to work in their current form.

On the tooling side we have to do work to extract those bits out and provide colorization, IntelliSense etc but the compiler simply sees them as opaque blobs of content.

danroth27 commented 1 year ago

@daveaglick Anything from how Statiq uses Razor that we should consider here?

daveaglick commented 1 year ago

Thanks for the tag @danroth27! As far as I know, Statiq is one of a handful of third-party projects that are currently using Razor directly outside of the in-the-box scenarios. Everything above seems extremely positive to me in terms of overall goals. One of the biggest challenges I've had with the recent versions of Razor is figuring out how and where to extend and hook into it. That's understandable given the existing primary design goals didn't explicitly include opening up to outside consumers, and at least there are some publicly exposed APIs, but it did require a lot of reverse-engineering and even some reflection-based cracking of internal/protected code.

In general I think comparing to Roslyn is the exact correct mindset for the next generation of Razor APIs. We've seen how quickly Roslyn enabled lots of scenarios that were unheard of or extremely challenging before. I suspect that once Razor is easier to use by consumers (and such use is well publicized and documented), we'll see an equally large adoption of Razor by third-party consumers for all kinds of interesting scenarios.

I don't actually think the Statiq use cases are that exotic, and are probably indicative of what most external consumers would want to use Razor for. Here's what it needs to do:

Send content to the Razor compiler to generate Razor assemblies (which can either be saved/cached and/or immediately executed). I.e. the "here's a string, give me an executable Razor assembly" scenario.
Similarly, allow saving/caching compiled layouts, partials, etc. along with the actual pages. I currently do this by creating a custom view compiler that intercepts compilation calls and delegates to the in-box caching compiler (which requires a little reflection magic currently). This is one area I'd love to see more extensibility.
Hook into the Razor file system so that layouts, partials, etc. can be provided using hooks rather than files on disk (which may end up actually loading content from disk for those things, but may provide it from somewhere else too).
Allow consumers easily use to all the current conventional Razor features such as layouts, partials, tag helpers, etc. This is one area I think could use improvement over the existing APIs. The underlying Razor engine right now is certainly usable if you're willing to do some reverse-engineering and learning, but getting access to the conventional features that ride on top of that and a lot of developers think are "part" of Razor is trickier. I.e. getting tag helper support on top of generic Razor engine use requires some hacking currently. It would be nice if those "optional" features were more plug-and-play for external consumers.

I'll add to this list if any other scenarios come to me, and hopefully that all made sense. I'm happy to continue the dialog when/if it would be helpful!

seclerp commented 1 year ago

Hi folks, do you have any updates or ETA?

333fred commented 1 year ago

Hi @seclerp, what part in particular are you looking for an ETA on?

seclerp commented 1 year ago

I'm very interested in the new pluggable Razor compiler that will have the ability to modify its execution pipeline completely. The current solution is full of "internal" assemblies and types, making it very tricky to customize. In particular, my goal is to make my own compilation backend that differs from HTML.

333fred commented 1 year ago

Unfortunately, I think you may have misinterpreted what we plan for the new API surface area. Making the compilation process itself (meaning the pipeline from parsing to emitting C# code) pluggable is the source of the issues that we are seeking to address with this change. Replacing the C# code that is emitted by the Razor compiler with a different format is not a use case we will be looking to support.

seclerp commented 1 year ago

The C# code at the end is OK for me, but I want to modify its shape. From what I see now it generates C# that itself coupled on HTML tags, am I right?

meaning the pipeline from parsing to emitting C# code

Does it mean that I will be able to replace any step, including a couple of final ones? Currently, I need to go deep inside the forked repository and patch things to have access to these steps.

Also, I've tried making my own tag helpers, but it's not enough for me, unfortunately.

333fred commented 1 year ago

Does it mean that I will be able to replace any step, including a couple of final ones that I need to replace?

No. It means the exact opposite. We do not plan to allow users to replace any step of the razor compiler, just as we don't allow users to replace any step of the Roslyn compiler. It may someday be possible to write pre-steps (a la source generators for Roslyn) or analysis steps (a la analyzers for Roslyn), but if you need to tweak the internals of how the razor compiler generates code, you will need to fork it as you are doing today.

seclerp commented 1 year ago

Very sorry to hear that. Anyway, thank you for the quick response.

tbasallo commented 1 year ago

@333fred will these changes address the issues found in #7616 #7608 and others? And if so, can we get an update as far as timeline?

I was about to post on one of the other issues and found this one - which seems like progress is being made, but not sure.

333fred commented 1 year ago

These changes will help us with those problems, yes, but we will update the relevant bugs directly when we make progress on issues. We understand that they are problems and cause pain, but I don't have any specific timelines for when we get them fixed.

dotnet / razor