[Blazor] Serving .gz Compressed Sitemap Files with app.MapStaticAssets()

[JeepNL]

Is there an existing issue for this?

• [X] I have searched the existing issues

Describe the bug

I am encountering an issue with serving .gz compressed sitemap files in a Blazor application using app.MapStaticAssets().

Here are the details:

Environment:

Development OS: Windows 11 Production OS: Ubuntu VPS Web Server: Kestrel (In production behind YARP) .NET Version: .NET 9 preview 6

Description:

My sitemaps are dynamically generated from database records after the production build and are .gz compressed. Each sitemap contains a maximum of 50,000 URLs. It's possible for new sitemaps to be generated while the site is live if more records are added to the database.

Using app.UseStaticFiles(), I can serve and download the .gz files without any issues.

When switching to app.MapStaticAssets(), attempting to download a file like nj_sitemap3.xml.gz via https://localhost:10443/nj_sitemap3.xml.gz results in a browser error:

This page contains the following errors:
error on line 1 at column 1: Encoding error
Below is a rendering of the page up to the first error.
The file does not download correctly, though the status code returned is 200 OK.

Expected Behavior

The .gz file should download correctly without encoding errors, as it does with app.UseStaticFiles().

Actual Behavior:

The browser displays an encoding error and the file does not download correctly, despite a 200 OK status code.

Any guidance or solutions to resolve this issue would be greatly appreciated.

Steps To Reproduce

Generate a .gz compressed sitemap file. Configure app.MapStaticAssets() to serve static files. Attempt to download the sitemap file via a web browser.

Exceptions (if any)

No response

.NET Version

.NET 9 Preview 6

Anything else?

Additional Context:

Screenshot of the browser error:

[javiercn]

@JeepNL thanks for contacting us.

This is explicitly out of support for MapStaticAssets as it collects and generates the information during the build/publish process.

If you have a "dynamic" scenario where you need to generate the file at runtime, we recommend using an explicit endpoint for it instead of trying to wire up the file int o MapStaticAssets.

[JeepNL]

@javiercn thank you for your reply.

I understand that MapStaticAssets collects and generates the information during the build/publish process, but I didn't understand why it generated an error when I was trying to serve the dynamically generated.gz compressed sitemap file.

Your suggestion is much appreciated, and I will look into how to use an explicit endpoint for it.

MapStaticAssets is very new and I understand there's not yet much documentation available, but maybe it's an idea to include a sample, in the docs, on how to do this, because I think more developers will try to serve dynamically generated compressed files (not just sitemaps).

If you want you can close this 'issue'. Again, thank you.

[javiercn]

@JeepNL Thanks for the additional details.

I understand that MapStaticAssets collects and generates the information during the build/publish process, but I didn't understand why it generated an error when I was trying to serve the dynamically generated.gz compressed sitemap file.

I can't exactly picture what you are doing, so it's hard for me to tell what's going wrong. There can be lots of things that can cause issues.

MapStaticAssets is very new and I understand there's not yet much documentation available, but maybe it's an idea to include a sample, in the docs, on how to do this, because I think more developers will try to serve dynamically generated compressed files (not just sitemaps).

You wouldn't do this through MapStaticAssets. You can either just use MapGet("path", => return File(...)) or wire up the StaticAssets middleware, something like this.

var builder = CreateAppBuilder();
builder.Use((ctx,nxt) => { ctx.SetEndpoint(null); return nxt(); });
builder.UseStaticFiles(new StaticFileOptions { FileProvider = ... });
var rd = builder.Build();
app.Map("path/to/file", rd);

The big advantage of MapStaticAssets is that we handle compression, Content negotiation ETags and Caching, but those are things that are independent and that you can wire up in your endpoint by setting the right metadata (ContentEncodingMetadata) for content encoding negotiation, for example.

[JeepNL]

@javiercn thank you for your reply

I can't exactly picture what you are doing, so it's hard for me to tell what's going wrong. There can be lots of things that can cause issues.

It's easy to reproduce this 'error': just put a .gz compressed file, any file (it doesn't need to be a sitemap), in the wwwroot directory after the build, and then point your browser to that file. In other words, try to download it from the wwwroot directory via the browser. Be sure you use MapStaticAssets.

You wouldn't do this through MapStaticAssets. You can either just use MapGet("path", => return File(...)) or wire up the StaticAssets middleware, something like this.

Yes, I understood what you meant. However, as a non-native English speaker, I may have expressed myself unclearly. My suggestion is to add some documentation for MapStaticAssets to inform developers that they should use an explicit endpoint to serve dynamically generated compressed files. Including an example of how to do this would be very helpful.

What I would like to do is use MapStaticAssets (because of all its advantages over UseStaticFiles) and then use app.MapGet to map an explicit endpoint to the compressed sitemap files. I think I can figure out how to do this with the help of ChatGPT and GitHub Copilot :) If you can provide a sample, that would be very much appreciated too, but I don't want to take up any more of your time, as this isn't a bug.

[JeepNL]

Oh, I forgot, My project/website is a Blazor SSR application. (Not Blazor Server/Blazor WASM)

[javiercn]

@JeepNL thanks for the additional details.

Yes, I understood what you meant. However, as a non-native English speaker, I may have expressed myself unclearly.

No worries. I'm not a native English speaker either.

It's easy to reproduce this 'error': just put a .gz compressed file, any file (it doesn't need to be a sitemap), in the wwwroot directory after the build, and then point your browser to that file. In other words, try to download it from the wwwroot directory via the browser.

Now this makes more sense. What's happening here is that we detect the content type based on the extension, but we don't apply a content-encoding header. If you apply a Content-Encoding: gzip header to the response, it'll likely work.

When doing this at build/publish time you can leave the compression to us.

For dynamically serving the file at runtime, the static files option is the best approach, as it'll give you a balance between quality and speed. If you really don't care about speed, you can replicate what we do for Static Assets. It's not really complicated. We compute ETags based on the content of the file, and we set the caching headers to no-cache. We also provide a fingerprinted version with immutable cache headers and serve the compressed version through content-negotiation, but this is something you don't need to do. For example, you can just check the header and if it accepts gzip, serve the .gz version with the right content-encoding header. It's work, but not more work than it was in the past with static files.

It's up to you to decide how much do you care about it (how much do you win in terms of bandwidth/speed), but it's likely you don't get much for a single file to benefit from doing all the work vs compressing the file or using the response compression middleware.

I'll file a separate issue on the docs to track adding a sample on how to do this dynamically, in case there isn't one.

[JeepNL]

@javiercn thank you for your reply

I'm sorry if I didn't make myself clear enough about what's actually happening in my setup. I have a separate C# console program running every 20 minutes while the website is live, which generates compressed sitemaps and places them in the wwwroot directory. These files are already compressed, and they don't need to be compressed again. I just want to serve them to search engines like Bing and Google by adding the URLs in their search consoles on the web.

This worked with app.UseStaticFiles(), but doesn't work with app.MapStaticAssets(). That's the only change I've made to the code.

Your suggestion to use an explicit endpoint makes a lot of sense, and I will try to do that. I have already tried something like that before I created this issue, and it didn't work, but that was probably my fault. I need to spend more time on it.

I understand that you mentioned balancing quality and speed with live compression, but since my files are already compressed, I only need to serve them as-is.

[javiercn]

@JeepNL also, you can mix and match app.UseStaticFiles with app.MapStaticAssets. The latter will take care of things that are known at build/publish time and UseStaticFiles will handle anything that was not present at build time. For that, the main thing is to make sure that UseStaticFiles is called after UseRouting.

It is explicitly designed this way because MapStaticAssets is meant to be a very specialized version of UseStaticFiles with much more knowledge/optimizations performed during the build process.

If it helps, check the implementation of UseBlazorFrameworkFiles here https://github.com/dotnet/aspnetcore/blob/b41e166aac4746253b541ce512eaeff8dbf73112/src/Components/WebAssembly/Server/src/ComponentsWebAssemblyApplicationBuilderExtensions.cs#L34

[JeepNL]

@javiercn Thank you for your reply.

Also, you can mix and match app.UseStaticFiles with app.MapStaticAssets.

Ah, I didn't know that! Thank you. I'll try to make it work, and I will let you know when it does!

[dotnet-policy-service[bot]]

This issue has been resolved and has not had any activity for 1 day. It will be closed for housekeeping purposes.

See our Issue Management Policies for more information.

[JeepNL]

@javiercn FYI

I've found a solution, which works in this my use case, now I don't need to use UseStaticFiles(), but only MapStaticAssets()

{fileName} (the wildcard) can be anything, but not with the extension .gz, (the extension causes the problem) so something like 1, 2 etc. and then the url will be /nj_sitemap1 (etc)

The file(s) nj_sitemap{fileName}.xml.gz need(s) to exist in the wwwroot directory.

app.MapGet("/nj_sitemap{fileName}", async (HttpContext context, string fileName) =>
{
    // Construct the full file path
    string? fullPath = Path.Combine("wwwroot", $"nj_sitemap{fileName}.xml.gz");

    if (File.Exists(fullPath))
    {
        context.Response.Headers[HeaderNames.CacheControl] = "no-cache, no-store, must-revalidate";
        context.Response.Headers[HeaderNames.Pragma] = "no-cache";
        context.Response.Headers[HeaderNames.Expires] = "0";
        context.Response.ContentType = "application/gzip";
        context.Response.Headers.Append("Content-Disposition", $"attachment; filename=nj_sitemap{fileName}.xml.gz");
        await context.Response.SendFileAsync(fullPath);
    }
    else
    {
        context.Response.StatusCode = StatusCodes.Status404NotFound;
    }
});