Feat: Service to programmatically add custom Headers such as Open-Graph on Prerender

[iJungleboy]

This is a continuation of #1967, in which this was already discussed.

The solution for it was #1984 which added a field in the page settings to add meta-headers manually. But that's not exactly what we had in mind.

Our need is that a software developed on Oqtane can add headers to the prerender. Example use case:

1. A blog which adds open-graph headers which are different for each post
2. A public Recipe DB or Real-Estate listing, which needs to add Open-Graph headers
3. A system which can add no-index headers to certain pages which shouldn't be available in any crawler
4. Adding CSP security meta headers to the head

In many such cases there is not a real Oqtane page per visible page, and the module on the page would need to be able to add these headers. I'm specifically concerned about pre-render, but of course a service that would work in all cases would be even better.

Note that a system such as the resources wouldn't work, as the final headers are usually not known at the time the resources are picked up (which is very early, before the module actually knows much about the request, so it can't retrieve page specific details at that time yet).

Your thoughts?

[sbwalker]

If you need to inject information into the page during pre-render, the logic needs to be executed server-side (ie. in _Host.cshtml). However it sounds like you want to be able to include this logic in your module components. The problem is that modules are only executed client-side - they are razor components which are dynamically rendered and depend on state which is generated by the client-side router. So it is not possible to instantiate modules server-side and execute logic. The only reason why the Resources concept works is because it is a stateless static property. More research will be required for this feature.

[iJungleboy]

So I may have the wrong idea here, but this is what I understand:

• The client can run in the browser or in a very isolated thread on the server in something that I'll just call the Client Process for now.
• The initial HTML is generated in the Server Process (or please correct me if this actually happens in the Client Process)
• Pre-Render probably augments this, by letting the Server Process talk to the Client Process and inserting the final HTML into the initial HTML, simulating what the browser would do

So this opens these questions:

1. Is the process I think is happening correct, or does the Server Process also render the client stuff (is it the same process, in which case it would know more and also be able to share services - in which case it would be easy to have a service transfer the necessary headers)
2. If the process is completely isolated, how could the client pass not just the HTML, but additional data to the prerenderer...

[sbwalker]

The process is quite a bit different than you imagine. Pre-rendering occurs on the server so it is only possible when using the .NET Core Hosted approach with Blazor (which Oqtane uses). Essentially this means your app bootstrap page must be a dynamic _Host.cshtml page rather than a static index.html page. Pre-rendering has very limited capabilities as the code is not executing in a browser context ie. you cannot use JS Interop to execute any JavaScript, etc.. During pre-rendering your razor components are executed on the server - which includes executing life cycle events such as OnInitialized and OnParametersSet - but not OnAfterRender. So the components will generate HTML content - which is then sent to the client as part of the page output. Once the client receives it, the component is "activated". On Blazor Server this includes creating the SignalR connection, and on Blazor WebAssembly it means loading and running the code in the browser. It is important to note that in all cases, components are fully re-executed client-side - duplicating the work that was done during pre-rendering (ie. any initial component state which was created during the pre-rendering phase is lost during the transition from pre-rendering to client-side activation).

So the challenge is that during pre-render you want a razor component to insert content into the head of a razor page. Typically this is only possible using JS Interop (as Blazor does not allow DOM manipulation) - but JS Interop only works once the component is activated client-side, so it is not an option here. In addition, since the same component code is executed both server-side and client-side, this logic would need to be aware of its environment ( and there is no OnPreRender life cycle event). So more research will definitely be required.

[iJungleboy]

Im trying to wrap my head around this, and I've tried a bit of code and read some docs such as https://learn.microsoft.com/en-us/aspnet/core/blazor/components/control-head-content?view=aspnetcore-6.0 just to better understand it.

From what I see there are instructions how to do it, but eg placing a <component type="typeof(HeadOutlet)" render-mode="ServerPrerendered" /> in the _hosts.cshtml has no effect. I have a feeling that the cause is as follows:

The demo app has the headers and such in the _Layout.cshtml, so my guess is that the hosts (in the server process) can get the entire HTML (from the client process) which includes headers that the client needs.

In Oqtane I believe that the hosts.cshtml is trying to do more work, by only letting the client provide stuff in the div in the body - so the client can't control the entire body. This I guess makes it harder to give back header + body, because they would be separate units and Blazor only expects to get one unit (the body) from the client.

Does that make sense? This is just guessing...

[sbwalker]

The native HeadOutlet solution provided in .NET 6 has a lot of dependencies and limitations which do not make it suitable for Oqtane. The HeadOutlet was designed for the default Blazor application development approach where you use the default Blazor router, a Layout component, and routable "page" components (where there is a single component rendered on a page at a time) - none of which are applicable in Oqtane.

[sbwalker]

There is more info on HeadOutlet issues at #37293. Overall it appears to be a very clumsy solution.

[iJungleboy]

Thanks. I assume you are referencing https://github.com/dotnet/aspnetcore/issues/37293

I don't seem to understand how the <component on the server process talks to the client process, and how data is returned for pre-rendering. Would you have any link or something I could look at? Then I could wizard up some PoC.

[sbwalker]

There is an open source third party component which manages head elements including support for server-side rendering. It even claims to offers support for more scenarios than the native HeadOutlet feature in Blazor:

https://github.com/jsakamoto/Toolbelt.Blazor.HeadElement

After reviewing the code, I see that it registers a Scoped service during startup called HeadElementHelperStore which provides state management for storing values that can be set within components and accessed during the prerendering phase. It also registers a middleware called HeadElementServerPrerenderingMiddleware to intercept the pipeline and modify the page output. It uses a third party library (AngleSharp) to load the page output into a DOM object on the server so that it can perform manipulations before sending it to the client.

I do have concerns about the performance impact of this implementation as it is intercepting the output for every request and loading it into memory for manipulation (ie. potential for memory leaks). There also appears to be a lot of complicated usage of semaphors and locks which are red flags for scalability. In addition, it uses a third party library and Oqtane tries to avoid third party dependencies.

That being said, I do think that the general approach is valid. The scoped service is the best way to manage state between components and server during prerendering. However rather than using middleware, I would think that an approach similar to the MVC ActionFilter would be simpler and more optimized:

http://www.binaryintellect.net/articles/133da380-b62b-4384-a99f-b1b2e105776e.aspx

However, Blazor uses a Razor Page rather than MVC as its bootstrapper - and Razor Pages do not support ActionFilters - they use PageFilters instead. So the solution would need to determine how to use the PageFilter approach to modify the Response output.

https://www.learnrazorpages.com/razor-pages/filters

My concern is that this may also have the potential to impact scalability due to the loading of the response body into memory and then using string manipulation. So it would need to be intelligent so that it is only executed when required (ie. when a component sets a page header property).

The point of this post is to only explain the research I have done so far related to this feature request.

[iJungleboy]

@sbwalker thanks for the research and update.

I believe this is the way to go. I also believe that adding more third-party modules is not ideal, so if we can basically provide the same functionality with just a few additional classes, it would probably be better that adding third-party componets.

[sbwalker]

I attempted to use a PageFilter to intercept, modify, and inject the response but I cannot seem to get it to work. So perhaps a middleware approach is the only option. The article by Rick Strahl provides a good explanation of the challenges in creating this type of middleware:

https://weblog.west-wind.com/posts/2020/Mar/29/Content-Injection-with-Response-Rewriting-in-ASPNET-Core-3x

[sbwalker]

I think we should hold off on this feature until .NET 8 as it appears that Blazor United could make this much simpler by providing support for much easier server-side rendering using standard components

[thabaum]

Look at the request here and I have been looking at the updated notes from .NET 6/7/8 from

https://learn.microsoft.com/en-us/aspnet/core/blazor/components/control-head-content?view=aspnetcore-6.0

and

https://learn.microsoft.com/en-us/aspnet/core/blazor/components/prerendering-and-integration?view=aspnetcore-6.0&pivots=server

Trying to think of the different approaches.

adding BlazorHead package

@using BlazorHead in the _Imports.razor

In the App.razor file add the HeadElement:

<HeadElement>
    <title>My Title</title>
    <meta name="description" content="This is a my description." />
    <meta name="keywords" content="Oqtane, Blazor, Entity Framework, Open Source" />
    <!-- Add any other metadata elements here -->
    <meta property="og:title" content="My Content Title" />
    <meta property="og:description" content="This is the content description." />
    <meta property="og:image" content="https://anotherexample.com/images/myimage.jpg" />
    <!-- Add any other Open Graph metadata elements here -->
</HeadElement>

The we have a way to have our modules interact with the header content.

@inherits LayoutComponentBase

@{ 
    string pageTitle = "My Page Title";
    string pageDescription = "This is a description of my Oqtane Blazor page.";
    string pageKeywords = "Oqtane, Blazor, Entity Framework, Open Source, ASP.NET, web development";
}

<HeadElement>
    <title>@pageTitle</title>
    <meta name="description" content="@pageDescription" />
    <meta name="keywords" content="@pageKeywords" />
    <!-- Add any other metadata elements here -->
</HeadElement>

<div class="container">
    @Body
</div>

OR

Create a custom component that will be used for prerendering:

@page "/prerender"
@using System.Net.Http
@using Microsoft.AspNetCore.Components.WebAssembly.Services
@inject HttpClient HttpClient
@inject NavigationManager NavigationManager

@code {
    [Parameter] public RenderFragment ChildContent { get; set; }

    protected override async Task OnInitializedAsync()
    {
        var requestUri = NavigationManager.Uri;

        var httpClientHandler = new HttpClientHandler();
        // Add your custom headers here
        httpClientHandler.Headers.Add("MyCustomHeader", "MyCustomValue");

        var client = new HttpClient(httpClientHandler);
        var response = await client.GetAsync(requestUri);
        var content = await response.Content.ReadAsStringAsync();
        SetParametersAsync(ParameterView.FromDictionary(new Dictionary<string, object> { ["Content"] = content }));
    }
}

startup.cs

    AddHttpContextAccessor();
    AddHttpClient();

_Host.cshtml

@page "/"
@addTagHelper *, Microsoft.AspNetCore.Mvc.TagHelpers
@{
    Layout = "_Layout";
}
<prerender>
    <app>
        <component type="typeof(App)" render-mode="WebAssemblyPrerendered" />
    </app>
</prerender>

Which approach is going to help resolve this issue? Are we able to update the Oqtane page header now from within our components or is this something we need to implement?

[sbwalker]

@thabaum is the code above actual working code or just your theoretical thoughts on an approach? I m asking because if you read this thread from the top, there have already been efforts to get HeadOutlet to work in Oqtane but they were unsuccessful because HeadOutlet has a dependency on _Layout which is not compatible with the Oqtane approach.

[thabaum]

Since this is for working with Oqtane I would say they are theoretical, however both examples should work. I have not test modded it for Oqtane. If you have not tried these approaches and one looks like it may work for Oqtane give it a spin.

I am trying to understand the issue see if any code ideas may help. I am very interested in anything SEO for Oqtane.

[thabaum]

@iJungleboy

I see in your example you used the render mode "ServerPrerendered" , does it make any difference to set it to "WebAssemblyPrerendered" in Oqtane? Have you tried that? I maybe twice as confused here.

<component type="typeof(HeadOutlet)" render-mode="ServerPrerendered" />

Try:

<component type="typeof(App)" render-mode="WebAssemblyPrerendered" />

talking specifically about the render-mode="WebAssemblyPrerendered" if that makes a difference.

This should help SEO for internal search engines as well to deliver more accurate and relevant searches within the framework.

Not trying to take it off topic It would also be good to have descriptive Urls generated for our applications SEO. Then I think we can really put a cap on it (SEO). This is something I need in an upcoming project Oqtane would need to deliver to get that spot. Honestly this really is a hold up for me knowing this is an issue.

Back to being able to prerender meta data... I need definitive solution here that I can build upon as a solid foundation. Almost every project in mind I see a use case for this. I am glad this limitation is being discussed as I would of hit this wall...

[sbwalker]

@thabaum the render mode is set dynamically in Oqtane and can be specified independently for each Site:

The default setting is Enabled. So Oqtane does pre-render by default.

It also has the ability to set the page title and metadata for each page in a site:

These values are dynamically injected server-side - so Oqtane is SEO friendly already.

This specific issue is related to the ability to PROGRAMMATICALLY set attributes such as title or metadata at runt-time within module code - which is a scenario that is not currently supported.

[iJungleboy]

@thabaum I don't know anything about WebAssemblyPrerendered, but I assume it pre-runs some of the code so that the client could be faster in building the dom.

But it doesn't sound like that would change anything. The core challenge is that "inner code" should be able to tell the page that certain properties in the rendered HTML should be modified.

As of now Oqtane is missing that ability, as it would require some API the modules could call to inform the wrapping page of the modified values.

[thabaum]

@sbwalker Thanks for the clarity. It has been a while I do now recall Oqtane is ran both server or client hosting models and .NET 8 is suppose to combine the two, kinda mesh them together which I look forward to as the features they are going to give I have created code that handles the same way. I like one less thing to worry about.

@iJungleboy I appreciate the clarity here as well.

An API or service sounds like the best solution for everyone.

Should we:

• Create a service that scrapes a page to auto-generate based on content if left blank or maybe an auto-generate switch on the page when creating it.

• Create APIs that allows developers to update specific pages overriding meta data and url ending with slugs for SEO enhancements?

Things to consider?:

• With two competing vendor components on page which wins? A priority module setting and only one can be allowed set per page?

• Caching for performance, does Oqtane have something under the hood able to cache this already or do we need to add something more here such as the kind of cache used?

So to me after fully reviewing this discussion it sounds to me like we need a meta API and also a slug API?

I would be happy to attempt a working example of these if they are not being worked on currently for some example ways we can make this functionality in Oqtane. As I said I am highly interested in knowing exactly how all this works and dont mind investing into it just so I know it well for future projects. It sounds like it would be a good challenge. I can see a discussion forum or blog home having different meta and page slug content generated as you drill down levels into a catalog as well. Different levels of control to take into consideration which maybe the page that is decided upon that will be at the top dictating which module is selected to rule?

[iJungleboy]

@thabaum thanks for assisting here - I hope you'll get something to fly ;)

I think the solution would be ideal as follows:

1. Oqtane prefills the value as it believes it should be - basically something with a read/write property get;set;
2. Any module can read this and/or modify it
3. Final value is used to create the page

in 95% of all cases, no module makes a change

in the cases where modules do make a change, it's commonly a list/details scenario, which will only have 1 module anyhow.

In the rare case that multiple modules do something, the developer has to figure out what he wants.

[thabaum]

Looking at .NET 8, it is suppose to create a "static file" first with sections that can update from different states. I am trying to understand any issues that may be resolved in .NET 8, and if that functionality already resides in Preview go ahead and break Oqtane to work and play with it.

I am wondering if this is somewhat what @sbwalker is talking about. Regardless I still feel an API is the strongest option and should be used by Maui as well to enhance in-house searchability.

95% of the time no module makes a change, but the site may be based on the one module that does, and it does it a lot...

I plan on creating a personal breaking version of Oqtane with .NET 8 possibly Preview 3 in the future to test out ideas because this is about when I really want to move forward with a larger scale .NET Core Blazor project than that what I have been normally working on. If we can get Oqtane firing on all by then it should be one starter project to beat with fastest concept to production times.

[thabaum]

@iJungleboy Read and update before the page renders meta data that can be injected.

Maybe hard coding parameters for certain types of meta data or open-graph along with twitter allowed from the API so that it can be checked to ensure it has valid information that wont break the app or potentially inject wrong bad stuff. This way this API function can only be used for this one purpose.

I will try to create something acceptable I understand when things are kind of lacking full requirements to be included. It should be that way to keep things at a standard. If anyone can take what I work on further down another path or the same one and beat me with a PR, DO IT!

I can focus on other places as well but this is going to be one I don't expect to complete, but I am going to try as I need this as well. This is pretty straight forward for a request, just making it happen as another Oqtane Framework core feature.

@sbwalker is an API concept an acceptable route to take here if anyone in the community can manage to create something working?

[sbwalker]

@thabaum I already mentioned earlier in this thread that based on my assumptions about Blazor United it may provide the technical mechanism for a module to programmatically set head elements server-side. So my preference is to wait for Blazor United to be available in a preview form (which it's not yet) before exploring this further.

[thabaum]

@sbwalker

In the event it does not meet our needs I will still play with this API option. I am envisioning a control panel dashboard item where site admins can view and manage all slugs/meta data/analytics metrics and such for all modules/pages/sites by type. Maybe a bit overkill let a developer do it for themselves.

Also slug generation for files and such, built into Oqtane was what I was somewhat hoping for, but I guess everyone can create this for themselves as well.

I can see doing this all on a page module as well so I understand where you are coming from.

The model I was going to propose to start as an Oqtane SEO Manager was this one below. Although I know some of page meta data is set in other places this would get overridden if set adding any that contain data:

public class SEOManager
{
    [Key]
    public int MetadataId { get; set; }

    // Basic SEO properties
    public string PageUrl { get; set; }  // URL of the page
    public string Title { get; set; }  // Page title
    public string Description { get; set; }  // Page description
    public string ImageUrl { get; set; }  // URL of the main image used on the page
    public string Keywords { get; set; }  // Page keywords
    public string Author { get; set; }  // Author of the page

    // Canonical URL
    public string CanonicalUrl { get; set; }  // Canonical URL of the page

    // Open Graph properties
    public string OpenGraphTitle { get; set; }  // Title for Open Graph
    public string OpenGraphDescription { get; set; }  // Description for Open Graph
    public string OpenGraphImageUrl { get; set; }  // URL of the image for Open Graph
    public string OpenGraphType { get; set; }  // Type of Open Graph content

    // Twitter Card properties
    public string TwitterTitle { get; set; }  // Title for Twitter Card
    public string TwitterDescription { get; set; }  // Description for Twitter Card
    public string TwitterImageUrl { get; set; }  // URL of the image for Twitter Card
    public string TwitterCardType { get; set; }  // Type of Twitter Card content

    // Schema.org properties
    public string SchemaOrgType { get; set; }  // Type of Schema.org content
    public string SchemaOrgName { get; set; }  // Name of Schema.org content
    public string SchemaOrgDescription { get; set; }  // Description of Schema.org content
    public string SchemaOrgImageUrl { get; set; }  // URL of the image for Schema.org content
    public string SchemaOrgUrl { get; set; }  // URL of the page for Schema.org content

    // URL slug
    public string Slug { get; set; }  // URL slug of the page

    // Module and Page IDs
    public int ModuleId { get; set; }  // ID of the module the page belongs to
    public int PageId { get; set; }  // ID of the page this metadata belongs to

    // Custom metadata properties (if needed)
    public Dictionary<string, string> CustomMetadata { get; set; }  // Dictionary to store custom metadata key-value pairs
}

One last note: There are a number of ways to go about this, but I was thinking the most futuristic proof way that would work with Oqtane Server, Maui and Client. Maybe I am going off topic now? Maybe I missed the objective entirely here in what Oqtane can provide? If anything .NET 8 should just make this approach run even better?

I have had another relating solution come to light to test with using js. I will go into more detail sharing later if it in fact it can update the header server-side with updated programmatically computer assisted data retrieving parameters of information from a dynamic page.

If I can complete a working API using Oqtane maybe I will put that out in another repo for testing and post about it in the Oqtane "show and tell" section. I am not trying to overtake this conversation. If we are waiting for a solution, this thread is done for now.

Thanks again, sorry to be a bug :)

[thabaum]

I have looked deeply into this and took an approach of KISS/DRY/SOLID principles to create something with an API I will thoroughly test here soon after reviewing all the related code.

My plan is to create the controller as discussed since all the pages are client-side and only talk via controllers and update the _Host.cshtml header area to either update the existing metadata or use the existing metadata code. My first approach is to overwrite as I want to do so much more than just what is being discussed in this area. (extending what is there may bring a breaking change so I didnt want to even try it but I will once I get my current code base tested I designed for this)

What I have created is a bit over the top, so I will simplify it extremely simple once I get all the ins and outs figured out with the javascript services repositories and controllers along with the DB Model which looks nothing like what I proposed earlier although it would work. I went with a more "generic" approach so it can be easily extended as needed with custom headers and such including validation of sites and so on...

I am hoping by this weekend I will have some hope or news on my front with this situation for you @iJungleboy i can share. Again not sure how much I will share because I have a complete SEO package I am building for another project so I have to have this feature and was hoping Oqtane could produce it (sooner than later future proofed).

I also did find a few things with the sitemap I can enhance it to help with Oqtane SEO too I may share when I have time to test further.

Just thought I would share an update since this is on hold until .NET 8 I am hoping the solution I create will be bullet proof and serve well for anything extra it can give later.

I am trying not to introduce "breaking changes" or depenedencies. In fact I created an entire chart and statistics system I am going to test on it I may donate as well since ChartJS and other things we can't include as it adds potentially unwanted weight to a developers Oqtane Framework project.

Anyhow I have learned so much about the Framework with this it kind of went against what I trained on in the past year with only CLient-Side pages Services using Server-Side controller and repository design, I see how this is an issue. Wish me luck :)

[sbwalker]

@iJungleboy As a side effect of researching the Blazor United capability for .NET 8, I believe I have finally found a viable solution for programmatically injecting elements into the page head. I will explain more this coming week.

[iJungleboy]

awesome!

[sbwalker]

After a lot of research and dead-ends, .NET 7 has finally made this possible:

1. The ability to have multiple root components in your default document - Traditionally Blazor applications only had a single root component in the page body. In order to allow Blazor components to be used more easily in MVC and Razor Pages, enhancements were introduced to support multiple root components. This also made the head of the page available for dynamic rendering. https://github.com/dotnet/aspnetcore/issues/22638, https://github.com/dotnet/aspnetcore/issues/14434

2. An overhaul of the pre-rendering in Blazor - Prior to .NET 7 the root components were executed in series which created major challenges if there were dependencies between them. https://github.com/dotnet/aspnetcore/pull/40316, https://github.com/dotnet/aspnetcore/pull/40512

2811 provides the ability to inject content into the head or body of page programmatically. It works both in the server-side pre-rendering phase as well as the client-side rendering phase - without requiring JS Interop. This capability will result in a variety of optimizations as workloads which were implemented on both the server and client can be consolidated - improving performance and simplifying the code. More PRs to follow...

[iJungleboy]

Awesome - thanks for continuing on this.

Do you think this would prioritize the switch to .net 7, or still rather wait for .net 8 (which is the thing I currently assume is planned)?

[sbwalker]

@iJungleboy you might want to browse the Discussions as there are a lot of threads about this topic in the past 2 weeks:

https://github.com/oqtane/oqtane.framework/discussions

The dev branch is already upgraded to .NET 7 and there are a variety of enhancements already completed, in anticipation of of the version 4.0.0 release. This includes programmatic support for Meta tags, etc...

[sbwalker]

implemented in 4.0