search

dotnet / aspnetcore

ASP.NET Core is a cross-platform .NET framework for building modern cloud-based web applications on Windows, Mac, or Linux.
https://asp.net
MIT License
35.15k stars 9.92k forks source link

Support OOTB common log format #56334

Open rmannibucau opened 2 months ago

rmannibucau commented 2 months ago

Is there an existing issue for this?

Is your feature request related to a problem? Please describe the problem.

I saw https://github.com/dotnet/aspnetcore/pull/33251 but it seems it does not cover the original issue. My issue is that default output of aspnet server (or using "standard" extensions) does not comply to the http server standard (common log or extended format). Since this format (and its extended flavor) is very well tooled (log grabber etc) it is neat to be able to reuse it for dotnet apps to avoid to create one log parser per app/team/techno.

Describe the solution you'd like

This request is about adding UseCommongLogFormat(/*optionally configuration for the extended pattern which is useful when using logfields or attributes to enrich the output*/).

Additional context

No response

amcasey commented 2 months ago

@wtgodbe Might know the current and intended levels of support. @JamesNK may have thoughts on how this relates to our other observability work.

Either way, I don't realistically expect a change in this area in the 9.0 time frame.

rmannibucau commented 2 months ago

If it helps, I know tomcat copied/took inspiration from httpd (https://httpd.apache.org/docs/2.4/mod/mod_log_config.html) and guess there is no reason aspnet does not get such a configurable+standard middleware for a normalised k8s case. Happy to also help hacking it if it would enable to move forward.

JamesNK commented 2 months ago

I'm not familiar with the HTTP server log standard discussed here.

For collecting log information, we have other existing options: Structured JSON console output for listening to standard console output, and OpenTelemetry+OTLP for exporting and pushing logs to a 3rd party data source.

The criteria to add a new option would be how many people want it, how accepted and stable the standard is, and the amount of work to implement. We often use GitHub issues to measure demand, and a steady stream of people upvoting or commenting on an issue is the indication.

Because logging is very extensible, this feature could easily be added by a 3rd party NuGet package from the community. I don't think this has to be built into .NET.

rmannibucau commented 2 months ago

For collecting log information, we have other existing options: Structured JSON console output for listening to standard console output, and OpenTelemetry+OTLP for exporting and pushing logs to a 3rd party data source.

This is what I tend to setup myself when I have loki but the format inside often stays common log (or more exactly extended common log but since it would be configurable it is close IMHO). So often you do a pipeline where you parse json > parse common log since other servers often just put the common log in a json string or just stays raw (with a label to filter the pipeline to use).

Here is a sample for the other big frontend http server out there (nginx), it contains a promtail sample and some log lines samples if it helps: https://sbcode.net/grafana/nginx-promtail/

The minimal format is defined there https://www.w3.org/Daemon/User/Config/Logging.html#common-logfile-format but it is often extended to enable to access other fields and enrich it with specific needs (often headers+fields/attributes of the request).

Side note: even express.js has morgan which does exactly that kind of middleware (https://expressjs.com/en/resources/middleware/morgan.html). Indeed it is a dependency but I would be happy it to be another NuGet package while built-in in the project (the key point is being provided by this project to be trusted and get guarantees on its stability and feature in time).

The criteria to add a new option would be how many people want it, how accepted and stable the standard is, and the amount of work to implement. We often use GitHub issues to measure demand, and a steady stream of people upvoting or commenting on an issue is the indication.

The way I envision is to enable the configuration (indeed my top wish is reusing tomcat/httpd syntax but if something unrelated it would be good and with a flag to enable json-string encoding it would enable any custom format from json to extended common log while all attributes of the request can be exposed properly - I didn't manage to do it with w3c middleware but maybe it is a lack of knowledge on my side, hope not). Since it is configurable it does cover everyone needs and avoid the need of any other middleware so I guess itis 100% and would enable to deprecate at some point all other options.

It would even enable to remove W3CLoggingMiddleware to replace it by this configurable option in terms of implementation (not API for backward compat, ack) and solve in a single place bugs like the "finally" part missing in case of exception (taken time for ex for w3c). So having a single (impl) logging middleware is a bug plus for aspnet IMHO.

Because logging is very extensible, this feature could easily be added by a 3rd party NuGet package from the community. I don't think this has to be built into .NET.

Obviously, but the value is to be there by default.

Hope it makes sense.