Discussion: Razor.Evolution and the future of Razor extensibility

[rynowak]

Opening this issue at the suggestion of @daveaglick and @guardrex to explain a little more about our future plans.

What is Razor.Evolution?

We're current undertaking an effort as part of the Razor Pages effort to improve the fundamentals, and extensibility of Razor. Much of the Razor codebase was ported as-is from the original implementation and hasn't had a fresh coat of paint in a long time. Our intention here is to start fresh with APIs for using Razor and build a set APIs that meet our extensibility goals and can be stable for the future.

This is taking place in the Microsoft.AspNetCore.Razor.Evolution project. We don't currently have a release date/version that we can share publicly for this work.

You shouldn't expect to see any Razor language changes as part of this work. This isn't a revision of what Razor is or how to write view code, this is as all happening in the implementation of the compiler and language services.

Tooling

One of the primary outcomes we're hoping to support is better support for design time tooling (Visual Studio, VS Code, others?). Right now Visual Studio is hardcoded to use MVC's version of Razor, and in fact it's bundled with Visual Studio so it always has a copy. We want to be able to use the version of Razor that your project depends on directly.

We've also got a lot of great ideas about how to make the tooling experience well rounded and more complete. I'm not going to share any of that publicly right now because it's a ways off. What we're planning right now is a prerequisite.

There's a significant amount of work going in to this repository to improve the design and fundamentals of how Razor language support works. An equivalent amount of work is going to go into Visual Studio as well for now. We'll be pulling more of those things out into reusable packages as appropriate.

Extensibility

As we do this work we're optimizing for the kinds of extensibility commonly asked for. This includes things like customizing code generation for a different framework and adding new directives with meanings that you define.

If you've done Razor extensibility in the past, imagine that things were less monolithic. Rather than replacing the default functionality, you will add a code generation pass, or add a directive parser.

[guardrex]

Thanks for opening this one @rynowak --

Not necessarily a "Razor thing" as it might be more on the MVC side, but having a way to tap the final generated markup after Razor processing but before markup is returned to the client would be helpful for markup minification. @NTaylorMullen told me about a spot in MVC where "lambda expressions are hoisted" after the Razor engine is finished that might be the right spot to get a crack at the markup before being returned.

The reason that minification was difficult for me within the Razor engine was that Razor chunks break up the markup in ways that make the algorithm for minification difficult to pull off. Any minification algorithm will find itself having to work across Razor chunks in a complex (and easy to mess up) way. E.g., https://github.com/GuardRex/GuardRex.MinifyingMvcRazorHost (ugh ... hacks ... orf! ... :poop: )

Another possibility is to have a spot on the front-end of the Razor engine where the incoming View file can be minified before the Razor engine even sees it. I used to do something like this with a folder of Default_Views and then using Gulp (with a watch on that folder) to pre-minify those into our normal Views folder that Razor would then act upon. Maybe there is a front-end spot on Razor that we could tap into for it?

The backlog issue for minification is at https://github.com/aspnet/Razor/issues/444, and I definitely went TL;DR on it (oops), but the thing to just mention here is that if Razor.Evolution has anything possibly to do with a good spot for minification processing, then I hope it will remain under consideration and come off the backlog at some point.

[rynowak]

Thanks @guardrex!

Part of this work for us to improve chunks into a better IR. This includes including both more information and less information 😆 (depending on the specifics).

One of the top level goals here is that doing something like what you're doing would just be adding a pass. In theory all of the code in https://github.com/GuardRex/GuardRex.MinifyingMvcRazorHost/blob/master/ChunkVisitorMinifyingMvcRazorHost.cs that does something other than modify a chunk would just melt away!

Given that any Razor function is a series logical "HTML" writes interspersed with logical "dynamic" writes I'm curious how you'd want us to represent that information for you? I think that this will inherently have some challenges.

If you have ideas let me know. If you want to plug in before the parser, that's something that would be possible.

[guardrex]

that does something other than modify a chunk would just melt away!

Yes, indeed, that would be helpful. @Anderman did it in a matter of hours for me, and it certainly would have taken a small-brained T-Rex a few days just to get the Razor engine replacement setup.

The challenge doing it that way was that Razor chunks are really tough to minifiy around. I favored either a front-end or back-end approach to get out of having to deal with Razor chunks.

Front-end

If Razor.Evolution has a feature where devs can get access to the whole View file ... do the minification (or whatever other mods they want to make at compile time) ... then Razor goes to work on it and all of the Razor language bits don't break in the process, that would be good.

The nightmare here is NOT breaking the Razor language ... @statement and @{...} bits mixed with markup. Ugh.

Back-end

I have a real lack of understanding on the other end what's happening between what Razor puts out that MVC gets back and what goes on within MVC (e.g., "raising lambdas," etc.) before the final markup is rendered and sent to the client. As far as I know, that :sparkles: just works by magic :sparkles:.

My nightmare on the back-end is that compiled, Razor-processed Views just don't exist in a form that makes minification any easier (or even possible) given that we do want minification to happen at compile-time and not runtime.

If you tell me that its :turtle: "chunks all the way down" :turtle: or :turtle: "funky compiled code mixed with markup all the way down" :turtle:, then a back-end approach might not even be possible ... or at least no easier to deal with than Razor chunks.

TL;DR

Yes, front-end then if that seems easiest. I just need a crack at the View file just before the Razor engine goes to work. I'll just look for the first < angle bracket, go work, and try to avoid all of the Razor language bits, possibly even pulling Razor constructs, putting markers in their place, minfiying the markup, and then returning the Razor constructs.

[Bartmax]

I'm not sure if you opening this to let us know, asking for suggestions or what. Don't even understand the whole picture of where this is going, so sorry is this misplaced.

I want to tell my current pain points when using razor.

1 - Collections There is no easy/intuitive way to deal with collections on views. So a model with IEnumerable is not "razor friendly" like other types. Html does have a notion of collections but creating forms for them rely heavily on javascript. I think razor, templates, modelbinder and javascript? can be improved in such a way that collection are first class citizen. Model Validation plays a key role on this story.

2 - Consuming Model Metadata from Javascript. Right now we have a great experience with data validation with unobstrusive javascript but as soon as we want to do something outside of that, there's no built in way to consume such metadata.

3 - Spa is not a full in or out. I believe that current MVC can be much more powerful with the only addition of data binding on client side. While a whole admin site might as well wants or requires a full spa framework, I see a lot of value of include and embrace data binding only into the framework, for simple crud or to add some magic. There's already a great library (knockout) that tackles this problem, but it's not close to the tooling or framework as one would expect, like manual serialization, build custom elements tags, etc. There was an approach (previous to knockout) on MVC with Html Helpers like AjaxForm that started a work on this direction.

4 - Cache There's a great story on cache right now, one key feature I'm missing is "clear this cache part when X happen". Let's say I have a list of movies on a view surrounded via tag. I would love to have some easy mechanism to say at (for example) Post new movie, delete the cache. Something like:

View:

<cache id="MovieListCache">...</cache>

Controller:

ViewCache.Clear("MovieListCache")

5 - @@addTagHelper directive this is more of a tooling thing, but it's like working with string. Strong type all the things :)

6 - Dependency Injection on client. This may be more a template thing: Make it so you only include a bootstraper.js file and then use your own site.js with it with require("jQuery")

7 - ViewComponents as render parts There could be a hook into mvc so that one can call a ViewComponent from ajax and update it content seamslessly.

8 - Minification. This is similar to what @GuardRex just said, I think there are 2 things going on on minification of razor. a) Minification code, stuff that you write. Like visual studio creates a file.min.js when you save file.js, there could be a similar mechanism on razor so you get Index.min.cshtml when you save Index.cshtml with the corresponding tooling to minify views, while i must be done at the parser/compilation step, this I think will also work. b) then you have dynamic data, like stuff coming from code (aka from database). Sometimes you want to minify that too. Right now you read the output, minify and write it again or you deal with razor chunks. I super easy extensibility point here, like a ChunkTransformer that takes an input and returns an output would be awesome. Or anything along the lines with including something in the pipelining.

TL;DR:

• Collections as first class citizen on razor
• Model Metadata available on client-side scripting
• Two way client data binding for MVC.
• Enhance cache manipulation
• Strongly type @addTagHelper directive
• Add dependency injection to Client-side libraries
• ViewComponents as client-side consumables
• Easy extensibility for any razor processing. (like minification, or style inlining, possibilities are endless)

I know these the things can be done already. Maybe some can just be better. Again, sorry if this is out of place.

[tuespetre]

This will be great! I've thought the extensibility was bearable so far though some freshening up could help. The design time host extensibility will open up :door: :door: :door:

[rynowak]

Thanks @Bartmax - this issue is specifically about the extensibility and implementation details of Razor. We are not making changes to the Razor Language as part of this work.

However I do think this is a good list of pain points with current MVC and Razor, some of these points we've heard pretty often. I'll pass on your feedback. As I alluded to earlier, we're doing this work because we think there's a lot more power in Razor that we haven't unlocked yet.

I super easy extensibility point here, like a ChunkTransformer that takes an input and returns an output would be awesome. Or anything along the lines with including something in the pipelining.

Yes, this is something we're targeting to make easier. I still think that writing a minifier based on Razor will have some inherent challenges, but if you look at the link provided @GuardRex - I think we can eliminate 95% of that code that isn't actually doing minification.

[rynowak]

The design time host extensibility will open up 🚪 🚪 🚪

This is really important to keeping Razor relevant for other .NET web frameworks without dirty dirty hacks.

In fact one of the cases we ran into this recently was adding support for View Components as TagHelpers in 1.1.0-preview. We needed to extend the way we 'find' TagHelpers and there was just no great way to do it, but ultimately we can because we control what goes into Visual Studio.

[atrauzzi]

I'm not sure if this is better mentioned in a different issue, but comparing razor to other templating languages, there is clearly a lot of ground to make up.

When you look at some of the features and syntaxes on offer in other environments, razor seems quite "last-gen": https://laravel.com/docs/5.3/blade

Just visit that link and read the page top-to bottom. Rather than having nice syntax, razor seems to force people to write what almost looks like defensive code everywhere. Extending or overriding sections is built into the templating language itself. You have niceties like forelse which looks a lot better than if checks and then render blocks nested underneath (code complexity in templates should be avoided at all costs).

I guess my comment overall is: Razor itself is already dated, why not look at creating a new engine or a razor 2.0? Then you can hit all your goals in this ticket while also improving the ecosystem.

[tuespetre]

@atrauzzi: doesn't look too different to me. A 'foreach' block followed by an 'if (!items.Any())' block is just as straightforward, and terse enough to boot. We have the coalescing operator built into C# as well for the whole 'isset(variable) ? ...' thing in the Laravel docs. Not to knock on Laravel; they had to make up for stuff that wasn't as nice about PHP in order to bring the spirit of Razor to PHP :wink:

[frankabbruzzese]

@Bartmax , Collections can be handled with Razor and with ModelBinding sice the early versions of Asp.net Mvc as explained here http://haacked.com/archive/2008/10/23/model-binding-to-a-list.aspx/. Of course additions of elements and deletes MUST use either JavaScript or server roundtrips. Since you must change your client side Html, either you do it on the client side with JavaScript, or you go back to the server to render the pagae again. Since server roundtrips would be very painful, the only option is Javascript.

A lot of the stuff you are asking for would imply a kinda automatic js generation to handle client side stuffs in a completely transparent way. However this out of Razor scope, you should use Razor as it is to implement TagHelpers or HtmlHelpers that rely on page JavaScript files to do some Magic.

Availability of Metadata on the client stuff would be great! However, it's more a kinda compilation of C# classes int JavaScript (or better TypeScript) than a Server Templating stuff.

[atrauzzi]

@tuespetre - There's more than just forelse. The section extending and overriding is also a lot nicer.

[Bartmax]

@frankabbruzzese yes exactly that! :)

[tuespetre]

@atrauzzi:

Blade

<html>
    <head>
        <title>App Name - @yield('title')</title>
    </head>
    <body>
        @section('sidebar')
            This is the master sidebar.
        @show

        <div class="container">
            @yield('content')
        </div>
    </body>
</html>

@extends('layouts.app')

@section('title', 'Page Title')

@section('sidebar')
    @parent

    <p>This is appended to the master sidebar.</p>
@endsection

@section('content')
    <p>This is my body content.</p>
@endsection

Razor

<html>
    <head>
        <title>App Name - @ViewBag.Title</title>
    </head>
    <body>
        This is the master sidebar.
        @RenderSection("sidebar")

        <div class="container">
            @RenderBody()
        </div>
    </body>
</html>

@{ 
    Layout = "_Layout"; 
    ViewBag.Title = "Page Title";
}

@section sidebar {
    <p>This is appended to the master sidebar.</p>
}

<p>This is my body content.</p>

Thoughts

I see that Blade seems to offer a construct for section overriding that Razor doesn't (at least not without some weird calls to IsSectionDefined or whatever.) Razor could be extended with such a construct, though.

What I don't see is how any of this shows that Razor is 'dated' or 'doesn't have nice syntax' or needs a '2.0'.

[atrauzzi]

Whether you see it or not, the experience in razor falls short of what other templating languages offer.

The whole point of my bringing it up here is to see whether razor can be extended. So in some weird way, you're hijacking my point and using it as a complaint against itself. (To that: Your tone seems to be angling for an internet-fight, but I think overall you've proven my point.)

So yes, let's get razor extended with new types of blocks. That's exactly what I said.

[frankabbruzzese]

@Bartmax , give a look to my Open Source that contains several already defined advanced TagHelpers(immediate grid, batch grids, etc). At moment it contains just one provider that relies on server side templates, but in the future I'll add providers for generating automatically code from the most famous client frameworks (Angular, knockout.js, etc.) from the TagHeper settings, and a compiler to generate TypeScipt classes and metadata from C# classes.

[am11]

This evolutionary direction is very welcome. 🎉

Is the purpose of this new architecture (with explicit extension points and decoupling of concerns) also to encourage others to implement their other HTML pre-processor / superset languages that transpile to HTML via Razor-IR?

When I took a stab at my Needle proposal #78 (a less verbose Slim-like or HAML-like front-end language for ASP.NET) back in 2014, it was bit unclear which APIs provide the extension point and I gave up pretty quickly. The main idea was to derive a language from either Slim's or HAML's well-defined spec and replace the Ruby code blocks with C# ones. If you guys have already started such a work, or there is enough interest in developing a light-weight/clean-markup language for ASP.NET, I would be happy to team up. :)

[rynowak]

@am11 - our goals on this as a team haven't changed from the response here https://github.com/aspnet/Razor/issues/78#issuecomment-49671561

As I understand it, what you're talking about would be a different language (ie, not Razor). The right extensibility point for you to plug in would be the MVC view engine, which pretty much gives you full control.

[dazinator]

This sounds spot on. Here are some things I'd like to see:

1. Allow ability to plugin custom directives - easily. (think you mentioned that)
2. Providing an ability to support other design time language services (that compile to js / html) within the razor file would be nice. I'd like for example to write some TypeScript in my CSHTML file rather than some javascript - just to get the nice intellisense and strong typing. The same hooks could be used to support SASS / LESS at design time etc. I know it's usually preferrable to keep js / css in seperate files, but seriously, being able to deserialise the model to javascript directly in a razor page is just so damned convenient that I usually do have a little bit of client side code in my razor pages. I'd still prefer that client side code to be typescript though!
3. Not being able to use Sections within ViewComponents is a joke that must be solved :-) I don't know if this one is a Razor limitation, or just an MVC design issue, or a constraint thats imposed by a combination of both or something else! (think I have covered all the possibilities there) - but I think it needs to be solved because any non trivial VC author soon meets the need of wanting to include js on the page with their VC and then they discover this problem and it reflects poorly on MVC / Razor in my opinion.
4. Slightly related to 3, but being able to inject a script into the pages head, after the razor file has already rendered the head section. Imagine a component that executes when the Razor page is rendered (e.g a ViewComponent) - if that component is executed at the bottom of the page, how can it include a script / css file in the Head section of the page. I think if understand the concept of "multiple passes" correctly then this might be possible with Razor.Evolution? At the moment, most people solve this by collecting the scripts them dumping them at the bottom of the page.

[dazinator]

Oh yeah and one more thing as per @NTaylorMullen

Ya your custom TagHelperTypeResolver will not work for tooling. It is something we're looking to extend in future Razor releases though. For more info check out: aspnet/Razor#836

As long as there is eventually a solution for this one I'll consider that a win

[tuespetre]

@rynowak wasn't sure where was a good place to discuss this, but https://github.com/aspnet/Razor/blob/dev/src/Microsoft.VisualStudio.LanguageServices.Razor/DefaultRazorEngineDirectiveResolver.cs looks 🎆 new and exciting. I'm assuming that is some WIP stuff towards making it easier for local customizations to the razor engine to be picked up by the language service. Is that so?

[rynowak]

Yes, but the ability to use local customization won't be in VS for a while...

[tuespetre]

I figured as much, but it makes me happy to see a glimpse of it!