log/slog: structured, leveled logging

[jba]

We propose a new package providing structured logging with levels. Structured logging adds key-value pairs to a human-readable output message to enable fast, accurate processing of large amounts of log data.

See the design doc for details.

[jba]

it seems you are just arguing you don't like the API

@beoran, I actually do like the API on the page. But it leads to a couple of problems, the worst of which is that it seems to require handing users a pool-allocated value, which is dangerous (can result in use-after-free bugs). We considered that, implemented it in fact, and decided against it.

With a carefully written handler, calls to LogAttrs with a small number of Attrs (currently 5, but that's an implementation detail) will not allocate. I haven't benchmarked against zerolog, but I'd guess we'd come within a factor of 2.

[beoran]

@jba Interesting. If possible I would like to seen an example of just how using Zerolog can result in use after free bugs, since I never encountered those in practice. A benchmark would be most welcome. I consider performance the most important aspect of logging since normally, logs are only read by developers in case something goes wrong. When that doesn't happen and all is well, logging wastes electricity and money, so I think we should try to squeeze as much performance as possible.

[jba]

@beoran, I believe

e := logger.Info()
e.Msg("a")
e.Msg("b")

is a use-after-free error. Info allocates an Event from a pool, and Msg frees it.

[beoran]

@jba Yes, that seems correct it would cause problems to do that. Although, we are not supposed to use Zerolog like this, and in practice I never did. Zerolog is safe as long as we use the chaining API correctly. But I can see that for the Go standard library you don't want to allow such risks.

[AndrewHarrisSPU]

Am I correct that zerolog ensures an Event is consumed/no longer referenced when Msg is correctly used? Passing a Record to a Handler doesn’t - this leads to a different but related class of bugs

[beoran]

@AndrewHarrisSPU , yes, a look at the Zerolog source code shows that Msg and related functions write the log and finalize the Event if needed. That is why it is documented that an event should be discarded after calling Msg on it. In practice, this is easy as long as we use Zerolog in a chained API fashion and not store Event in variables.

[jba]

this leads to a different but related class of bugs

@AndrewHarrisSPU, what bugs are those?

[AndrewHarrisSPU]

@jba I was thinking about bugs that would exist only if slog were eagerly encoding bytes into a pooled buffer near the logging call. For example, if NewRecord(...) implied obtaining a sync.Pool buffer, the following scenario:

1. A logging call invokes r := NewRecord(...), and passes r it to handler1
2. handler1 eagerly writes bytes into some Record buffer, and defers freeing buffer until handler2 returns.
3. handler2 is concurrently lazy; it calls go writeRecordRemotely(r), and returns
4. h1 frees the pooled buffer
5. The go routine uses the buffer later, so there's a use-after-free bug fairly distant from step(1).

With the flexibility of Record/Handler, allowing for Handler composition, it's not quite right to eagerly encode the output bytes. It is safe to compose a zerolog-as-a-Handler device in the position of handler2 in this example, but possibly not as log or handler1.

[jba]

There is no pooled allocation in Record. The built-in handlers use a pooled []byte, but not in a way that users can see. There is an eagerly encoded []byte, but it is allocated normally.

[AndrewHarrisSPU]

Yeah, I feel like I miscommunicated - I'm just saying that, where the proposal discusses a tradeoff for more flexibility compared to zerolog, the eager encoding approach used by zerolog is a limitation on the flexibility of a Handler to adjust a Record. (Also I think it's a question that could be asked about micro-benchmarks - in this way the behavior being benchmarked not conceptually equivalent)

[ethanvc]

@jba First of all, it is a fantastic work.

I have a suggestion. Can you please make pc filed in Record struct Accessible? It will be more flexible to user. Reason for doing so is Record is a struct to record all log information, there have chance to make record when not in callstack. and this make more unified like other fileds.

[ethanvc]

I also see some receiver of Record struct is by value, this mix is not recommended by typical golang rul. Furthermore, pointer receiver is a bit fast then value receiver for Record.

[ainar-g]

Here is my experience report after trying to switch a small (<2,000 sloc) project from a simple Printf-based levelled logging package to the current proposed implementation. Items are numbered not by significance, but to simplify discussion.

(Note, these are not necessarily change requests, but rather pain points and random thoughts I had while doing the switch.)

1. It seems like there is no way to either instantiate your own default handler or change the default handler's level. This is confusing, because a lot of projects really just need a way to lower the level to debug when -v or --debug is provided.

2. Related to this, the TextHandler does not have the same format as the default handler. This is once again confusing, because there is no indication that the default handler is different from a TextHandler.

3. Some APIs, including http.Server in the standard library, require a log.Logger. I couldn't find an easy way to turn a slog.Logger into one.

4. The slog.Logger.Error API is a bit weird for those coming from simpler, Printf-based APIs. My main issue is that it requires both a msg and an err. So, if your project already has informative errors with messages like

   updating services: at index 2: shutting down: i/o timeout

   then what do you put into the msg field? If you just leave it empty, it still prints a msg. I would consider just leaving it at

   func (l Logger) Error(err error, args ...any) { /* … */ }

   since any additional message could be wrapped into the error itself with fmt.Errorf.

5. Unlike the methods of stdlib's log.Logger, which have pointer receivers, methods of slog.Logger have value receivers. I'm a bit concerned that that could become an issue with deeply-nested calls, but perhaps those are just premature optimization concerns.

6. The naming of the level constants seems a bit weird, since my first instinct was to look for LevelDebug, LevelInfo, etc. It seems like a lot of Go packages choose to give related constants a common prefix, including stdlib's log with its log.Ldate, log.Ltime, etc. Then again, logrus and zerolog seem to both follow the reverse convention, so this point is a bit moot.

7. Many projects decode and encode debug levels into text, for example when parsing configuration or reporting service state over JSON, so I feel like slog.Level really needs to implement encoding.TextMarshaler and encoding.TextUnmarshaler. To say nothing about future issues if the package maintainers decide to implement them later.

8. A lot of projects will likely introduce a panic level along with a deferrable helper that could be called with something like:

   func f(ctx context.Context) {
           defer logOnPanic(ctx)
   }

   This is probably not something to include into the package right now, but I feel like there will be a lot of requests for a standard way to do this.

9. Like a few people in the discussion, I don't really understand the purpose of slog.Logger.Context and slog.Logger.WithContext API. It seems to be more of a utility.

10. The package really needs way more examples. Especially with regards to changing levels and the output of the handlers.

Once again, those are just thoughts I've had and decided to share. I think that having a standard levelled logging package would be good for the community in any case.

[jba]

@ethanvc:

Can you please make pc filed in Record struct Accessible?

It's something of an implementation detail. You can get the information in a more usable form with Record.SourceLine.

some receiver of Record struct is by value

Passing Records by value means they incur no heap allocation. That improves performance overall, even though they are copied.

[jba]

@ainar-g, thanks for taking the time to use the package and provide helpful feedback.

It seems like there is no way to either instantiate your own default handler or change the default handler's level. This is confusing, because a lot of projects really just need a way to lower the level to debug when -v or --debug is provided. Related to this, the TextHandler does not have the same format as the default handler. This is once again confusing, because there is no indication that the default handler is different from a TextHandler.

The default handler is there just so that the slog and log packages can work together. Programs that use log can do so with the assurance that packages using the top-level slog functions like slog.Info and so on will produce output that looks like log output and goes to the same place (because it actually goes through the log package). We expect that programs that use slog will set their own handler early, so no output will go to the default handler.

Some APIs, including http.Server in the standard library, require a log.Logger. I couldn't find an easy way to turn a slog.Logger into one.

You could use something like the handlerWriter in the implementation. It's not clear to me that it's worth exporting that for those few cases. (There are exactly three uses of log.Logger that I could find in the standard library: as fields in http.Server, cgi.Handler and httputil.ReverseProxy).

My main issue is that [the Error method] requires both a msg and an err.

It's true that if you have a good error message, you may not need a log message. On the other hand, often the log message provides context: "while unzipping somefile.zip" for instance. Since structured logs are intended to be processed by machine, I think it's important to maintain the invariant that every log event has a message, even if it's an empty one. (And at the risk of sounding like a broken record, you can always write a handler that drops empty message fields.)

Unlike the methods of stdlib's log.Logger, which have pointer receivers, methods of slog.Logger have value receivers.

Making Logger and Record value types is an important optimization in the common case. Copying them a lot could slow things down, it's true, though Loggers are quite small. You can always take a pointer to a Logger and pass that around if you want.

Many projects decode and encode debug levels into text, for example when parsing configuration or reporting service state over JSON, so I feel like slog.Level really needs to implement encoding.TextMarshaler and encoding.TextUnmarshaler. To say nothing about future issues if the package maintainers decide to implement them later.

Level has a String method. We could add a ParseLevel function if there is demand, but I'm not worried if there end up being multiple implementations: they should all behave identically, at least for the strings returned by Level.String. (I assume behavior differences are what you had in mind by "future issues.")

The package really needs way more examples. Especially with regards to changing levels and the output of the handlers.

The package documentation is a work in progress. We've worked hard on the individual symbol godoc, but as you say, there are no examples, and there is no package godoc to explain overall usage (although when there is, it might look a lot like the proposal).

For "changing levels," I assume you mean something like the handler I sketched in the discussion. I can certainly make that an example. What did you mean by "the output of the handlers"? Do you want to see some example output?

[seankhliao]

We expect that programs that use slog will set their own handler early, so no output will go to the default handler.

imo this points to the global logger being an unnecessary distraction. We really want to ensure nothing goes through a global logger, tracking down errant callers is not a fun task.

---

Level really should have UnmarshalText, even though it's not hard to write the parsing function, it's not particularly short, annoying to have to do it every time, and makes using Level in a config struct more difficult then necessary.

There's also an asymmetry where it marshals into a string but can only be unmarshaled from an int.

[deefdragon]

Level really should have UnmarshalText, even though it's not hard to write the parsing function, it's not particularly short, annoying to have to do it every time, and makes using Level in a config struct more difficult then necessary.

I think the issue with having slog have its own Marshaler/Unmarshaler is that slog cant really know what the user is including for the other log levels. I plan on having a trace level and "Important" levels for info+ for example.

[seankhliao]

If you want your own mapping, it can be on a type that implements slog.Leveler. That shouldn't prevent the default Level from not being able to roundtrip through different encodings.

[ethanvc]

Pass by pointer does not mean allocate on heap, and profiling shows pass by pointer is more efficiency on my mac pro m1. Mybe you can profiling it. The Code Review comment can help:

If the receiver is a small array or struct that is naturally a value type (for instance, something like the time.Time type), with no mutable fields and no pointers, or is just a simple basic type such as int or string, a value receiver makes sense. A value receiver can reduce the amount of garbage that can be generated; if a value is passed to a value method, an on-stack copy can be used instead of allocating on the heap. (The compiler tries to be smart about avoiding this allocation, but it can't always succeed.) Don't choose a value receiver type for this reason without profiling first.

[ethanvc]

    // A Handler should treat WithGroup as starting a Group of Attrs that ends
    // at the end of the log event. That is,
    //
    //     logger.WithGroup("s").LogAttrs(slog.Int("a", 1), slog.Int("b", 2))
    //
    // should behave like
    //
    //     logger.LogAttrs(slog.Group("s", slog.Int("a", 1), slog.Int("b", 2)))
    WithGroup(name string) Handler

The comment missing some arguments, because LogAttris's prototype is func (l Logger) LogAttrs(level Level, msg string, attrs ...Attr).

[jaloren]

Overall, I think the design is excellent and I think the library as-is would be a great addition to the standard library. I do have some feedback that I think would improve the design on the margin.

Using Text Handler

I believe one of the most common changes a user will make to the default handlers is adjusting the log level. I am not sure if there are doc or api changes that could reduce the friction.

I started with this:

    logger := slog.New(slog.NewTextHandler(os.Stdout))
    logger.Debug("here is a message")

I then realized that the log level must be defaulting to INFO. Fair enough: so i need to enable it. So then I tried this:

    handler := slog.NewTextHandler(os.Stdout)
    handler.Enabled(slog.DebugLevel)
    logger := slog.New(handler)
    logger.Debug("here is a message")

I clearly did not read the go docs on the Enabled method, which clearly and explicitly say it reports on but does not set. My brain thought "Enabled" meant turned on. Though Enabled is past tense of Enable my brain missed the D and the implications. I thought enabled meant "set the log level to X".

Then I realized, it needed to be this:

    handlerOpts := slog.HandlerOptions{Level: slog.DebugLevel}
    handler := handlerOpts.NewTextHandler(os.Stdout)
    logger := slog.New(handler)
    logger.Debug("here is a message")

As a developer whose written in several programming languages, I would expect the logger to have a setLogLevel. I think that would be much clearer and easy to discover from a casual review of the godoc. If that's considered too radical of a change, I would minimally suggest we include examples on how to set the log level in the NewTextHandler/JsonHandler functions.

I also wonder if it would be worth it to add two convenience methods.

NewTextLogger(writer io.Writer, Level level) Logger
NewJsonLogger(writer io.Writer, Level level) Logger

Possible Bug with KV Pairs and Error method

Consider this code:

    logger := slog.New(slog.NewTextHandler(os.Stdout))
    err := errors.New("there is a problem")
    logger.Error("here is a message", err, "name", "john doe")

It prints:

time=2022-11-04T07:19:00.005-05:00 level=ERROR msg="here is a message" name="john doe" err="there is a problem"

Now imagine you have unbalanced key pairs like so:

 logger.Error("here is a message", err, "name", "john doe", "address")

This returns:

time=2022-11-04T07:19:11.713-05:00 level=ERROR msg="here is a message" name="john doe" address="err=there is a problem"

I suspect this is a bug and it was meant to print:

!BADKEY=address

Ergonomics of Using LogAttrs

If I want the type safety and avoid the foot gun of unbalanced KV pairs. I need to change this:

    logger.Warn("here is a message", "name", "john doe")

To this:

    logger.LogAttrs(slog.WarnLevel, "message", slog.String("name", "john doe"))

Now logger.With almost gives me what i want expect it takes any. What do people think about making logger.LogAttrs work like logger.With. Which is to say like this:

logger.LogAttrs(slog.String("name", "john doe"), slog.Int64("zipcode", 60002)).Warn("here is a message")

And yes I understand this almost replicates logger.With and I am 100% fine with simply change logger.With to take variadic list of Attr instead any. I understand the reason why the api has a variadic list of any. It handles performance and ergonomics well but you obviously can't panic or return an error when you log.

That said, I personally, I prioritize type safety, avoiding foot guns, and clearly documented apis. Yes, i know there is a vet check and no I don't want to ever have to review code like this:

logger.Warn("here is a message", "one", "two", "three", "four","six", "seven", "eight", "nine")

And figure out whether its balanced, the types are correct, nor whether the value matches up with the name properly.

I am advocating that we provide a slightly more ergonomic the api around Attrs for the camp of folks that want to avoid the problems that variadic any brings for key value pairs.

[beoran]

I agree this seems quite unergonomic:

logger.LogAttrs(slog.WarnLevel, "message", slog.String("name", "john doe"))

Something like logger.String("name", "john doe").Warn("message") seems more ergonomic.

[jaloren]

@beoran i like that even more but that would increase the API surface quite a bit.

[beoran]

@jaloren Zerolog does something similar, I think in this case, if performance allows it, then good ergonomic will be essential for this library.

In my honest opinion, slog unfortunately seems to be heading towards having less performance and less economics than Zerolog so, if neither improves before this is released, I will stick to Zerolog and ignore slog as much as possible.

Of course, I like chained API, which is subjective, but in the case of logging a chained API can be more compact and readable.

[Davincible]

@jba has there been thought of or decided against a Handler.Level() method, to fetch the current level?

I found that in some cases I need access to the current level of a handler, and currently the only way to to that is to wrap the logger and manually store it, which feels a bit awkward.

Use case: I want to create a sublogger (i.e. logger with the same fields), but with a different type of handler, and/or different level.

Different type of handler: e.g. part of the code RPC logs, part logs with plain text, with shared custom fields

To use a different handler with the same level, I need to know the level of the parent logger (handler), but currently, the only way to know for sure is to manually store it in a wrapped object. For this is would be nice of the Handler also implements the Leveler interface

EDIT: Inheriting fields into new handlers seems impossible at all with the current handler interface. I thought I could do something like this;

    ctx := logger.With(
        slog.String("component", string(component)),
        slog.String("plugin", name),
    ).Context()

    newLogger = slog.New(handler).WithContext(ctx)

But that doesn't work

EDIT 2: I am most likely miss understanding things here, but from reading the proposal on contexts in loggers, my expectation is that setting values on a context translates into custom fields in the logger.

type keyOne struct{}

// How would this even translate into a field? (without reflection)
ctx = context.WithValue(ctx, keyOne{}, "value 1")
ctx = context.WithValue(ctx, "KeyTwo", "values 2")

logger = logger.WithContext(ctx)

But since this is not the behavior I'm observing with the default handlers provided by slog;

1. am I misunderstanding how contexts in loggers work?
2. understanding correctly, but slog provided handlers simply don't implement this?

[ethanvc]

@jaloren Zerolog does something similar, I think in this case, if performance allows it, then good ergonomic will be essential for this library.

In my honest opinion, slog unfortunately seems to be heading towards having less performance and less economics than Zerolog so, if neither improves before this is released, I will stick to Zerolog and ignore slog as much as possible.

Of course, I like chained API, which is subjective, but in the case of logging a chained API can be more compact and readable.

I also believe zerolog's interface is more ergonomic.

[Davincible]

On the topic of sub-logger levels, as discussed here

The question is, is this common enough to warrant built-in support?

After playing with this design for a bit I feel like there is one significant drawback of having to wrap a handler to achieve a different level like this; forking a handler with increased verbosity for parts of your code to debug.

Therefore I'd like to say it would be very nice to add the following method to the Handler interface:

type Handler interface {
    ...

    // WithLevel returns a new handler, as a copy of the parent handler retaining all fields and groups, but with a different level. 
    WithLevel(level Level) Handler
}

Reasoning

If you want to increase the verbosity of your logging in one or more specific components of your code, currently the only way to do that is to set your root handler to a very low log level/verbosity (e.g. trace), and wrap it multiple times, i.e.:

1. Create a root handler with level = -inf / trace
2. Wrap root handler with LevelHandler in desired primary log level, e.g. info
3. Create primary logger with wrapped handler
4. Create secondary handlers by wrapping root handler with LevelHandler again, this time with desired debug level, e.g. level = debug, level = debug+

With the side effect that I need to store and manage the root handler in this scenario, by wrapping the slog.Logger in my own struct type, and passing around mytype.Logger with state management. This feels like a lot of extra work to get basic functionality out of a stdlib pkg.

This has even more side effects if a lib/framework would like to provide this functionality to its users not talked about here yet.

[jba]

The comment missing some arguments ... @ethanvc, thank you, fixed in https://go-review.googlesource.com/c/exp/+/447960.

[jba]

@jaloren, thanks for your comments.

suggest we include examples on how to set the log level

Examples are forthcoming. This will be one.

I suspect this is a bug

Good catch. Fix is in https://go-review.googlesource.com/c/exp/+/447962.

making logger.LogAttrs work like logger.With

I don't see how to make the fluent API you describe there both safe and zero-alloc. See an earlier comment.

[ethanvc]

func (l Level) String() string {
    str := func(base string, val Level) string {
        if val == 0 {
            return base
        }
        return fmt.Sprintf("%s%+d", base, val)
    }

    switch {
    case l < InfoLevel:
        return str("DEBUG", l-DebugLevel)
    case l < WarnLevel:
        return str("INFO", l)
    case l < ErrorLevel:
        return str("WARN", l-WarnLevel)
    default:
        return str("ERROR", l-ErrorLevel)
    }
}

maybe return str("INFO", l) lost sub InfoLevel? @jba

[jaloren]

@jba right sorry for the confusion. I am aware of that and I am not proposing a fluent api. What I am suggesting is that adding attributes in a type safe manner should be as ergonomic as the use of variadic any.

Specifically this is much less ergonomic

logger.LogAttrs(slog.WarnLevel, "message", slog.String("name", "john doe"))

then this

logger.Warn("here is a message", "name", "john doe")

I believe that variadic any is easy to write but hard to get right (that’s why a vet check will be added) and hard to read once there’s more than four arguments. Consequently, I at least would prefer using the attr alternative but that’s currently much harder to write.

So I am wondering if people would be open to modifying the api to that end.

[jba]

@ethanvc, InfoLevel is zero, so the code is correct. But https://go-review.googlesource.com/c/exp/+/444683 adds -InfoLevel for clarity.

[jba]

I at least would prefer using the attr alternative but that’s currently much harder to write.

Not really; you can write

logger.Warn("message", slog.String("name", "john doe"))

[jba]

@Davincible, we don't want a Handler.Level method because we have the more general Handler.Enabled method, which might use criteria in addition to or other than level to decide whether log output should happen.

However, you could certainly write a Handler.Enabled that enforced a minimum level. I added an example of such a LevelHandler in https://go-review.googlesource.com/c/exp/+/447965. (@ainar-g might also be interested in that.) It only overrides the Enabled method and delegates the others, so it handles your case of the same handler but a different level. To get a handler with the same "level" (that is, implementation of Enabled) but different handling logic, define an implementation of Handler that delegates Enabled but overrides Handle.

I don't really understand your "Reasoning" section in your second comment. Why are so many handlers needed? Define a "root" handler that works for the default case (say, Info level). If you need a handler with a different level, wrap your root handler in a LevelHandler.

setting values on a context translates into custom fields in the logger

It's actually much simpler than that. When you write

logger.WithContext(ctx).Info("msg")

all that we do is put ctx in the Record that is passed to logger.Handler().Handle. After that it's up to the handler.

[Davincible]

@jba

which might use criteria in addition to or other than level to decide whether log output should happen

That makes sense, hadn't thought of the more criteria requirement.

I don't really understand your "Reasoning" section in your second comment

You can disregard that. Turns out it does work as expected. I was testing it incorrectly. WithLevel would still be welcome but it's not as critical as I thought initially

[firelizzard18]

I wonder how many people who don't like the variadic any approach have actually used a logging library like that. Personally, it took me about a week to get used to it when the project I'm working on switched and I've made very few mistakes since then. If you're using constant keys and variable values, the pattern is pretty obvious:

logger.Log("message", "key1", value1, "key2", value2)

It's easy to see errors in that pattern, even when the values are complex. The only case I can think where that would not be true is with non-constant keys or constant values. In my experience those cases are extremely rare.

[deefdragon]

I wonder how many people who don't like the variadic any approach have actually used a logging library like that.

We use that kind of logger at my work in many places. I think I have managed to not need to fix my pr's logging only once or twice. (that is, when having more than the message, I almost always have to fix it. There's just something about the format that I have difficulty parsing)

At this point I basically do everything in my power to avoid adding anything but the message to the final log call, which defeats the purpose of having attribute there.

I just see the "simplicity" as too little gain for the loss of safety.

[ancientlore]

First thanks for working on this, I actually like the flexibility of logging approaches, and kudos for considering OpenTelemetry. I haven't experimented with all features yet, but I think I found a little issue in the text handler - see https://go.dev/play/p/-HLZ-pWvJH0 - it appears to clobber an attribute in some circumstances.

My real question is whether an idea I had has broader appeal or is too specific to my use case. We have some teams that log somewhat large structures for our JSON format. Writing LogValue() functions for these structures can become pretty cumbersome except for some fairly simple use cases. I came up with a reflect-based approach to automate this, also inspired by the json package. A quick-n-dirty implementation example is here: https://go.dev/play/p/HyKBt97vVkM

My initial thought is maybe it's a bit too clever, or a bit too slow for general use, but I thought I would share.

[cschneider4711]

Hi, nice design, thanks for the great proposal and the excellent work you all put into this.

I especially like the convenience level functions to be used consistently in many codebases. Just as an idea to provide some feedback for discussion: Do you think it would make sense to include another named level in these convenience functions: "Security"?

From my IT-Security related work I experience many situations, where security logs are helpful, either in proactively detecting certain security-relevant conditions (mostly done by SIEM tools analyzing logs and monitoring events) and sometimes also during forensic analysis (but then of course also more debug-like logs are used as well).

I know that creating such thing individually with this proposed API would be easy (thx also to the gaps in the numbering scheme), but especially the consistency used among many different codebases is where Go really shines. As in the recent releases security also was a major part (thinking about the great addition of fuzzing in the whole toolchain ecosystem), I think that incorporating a convenience logging function "Security" for security-events directly in the top-level API would generate more security-relevant thoughts within development teams of how to also think about certain security-relevant conditions in their software (and how to log these).

Mostly I can imagine situations or events like login failure events, missing permissions for some actions, invalid data provided (especially when not possible by accident via a client, so a potential hacker bypassed a client-side convenience check and then the server-side input validation kicks in), etc.

The downside of this thought would be to tend to "overload" the deliberately short list of convenience functions for log-levels (who knowns what will be next proposed convenience level?). On the upside one could argue that security should be ubiquitous enough in projects to find a good fit in most projects.

One idea of how this "security event log" level could be handled special (like the error level is taking an error arg for example) could be to ease counting and grouping of such events from logs by having a clearer distinction from the other logs as being a potential security-event and possibly by taking a "type" string argument, which can be provided (freely) by the caller, enabling a grouping of certain security event types. Thereby derivations from usual patterns could easier be detected by tools specifically watching for those consistently-logged security-events in the overall logs. And when the language-idiomatic logging framework possibly gets/has a security-event kind of convenience logging level, integrating this output with intrusion detection or SIEM solutions parsing these logs would be easer.

As said, just an idea to create room for further discussion on the pros and cons about this. Keep up the good work, and I really look forward to the new logging proposal being brought to life.

[cschneider4711]

And on another perspective three more ideas related to logging that came into my mind right now, again, just to inspire discussion of the pros and cons:

Correlation-ID for a Request

Especially for distributed backend systems (with many microservices or asynchronous flows) it could possibly be helpful to propagate in the architecture a unique identifier for the request (like a UUID created on an API gateway) and having this appended/prefixed on every log created under that thread. Something like the Java-style Log MDC (Message Diagnostic Context), which was essentially a ThreadLocal kind of stack to add some prefix stuff that is tied to one particular request and taken in front of any log statement thereunder automatically. Using a middleware to create and push/pop this could be easily done in most http routing frameworks, allowing for easier analysis of single requests under heavy load. Possibly the Context part of this proposal can be used for this by the popular http frameworks, but I wonder how this ID might also propagate into developer-direct usages of this logging proposal automatically? (I have to read the spec proposal more on this, I admit)

Ring-Buffer style of Debug-preserving Appending on Error-Conditions

This idea is partly taken from the Java Util Logging, where a memory-based appender-like construct was used to solve one of the most annoying and sometimes pressing problems of not having enough detailed logs to analyze a certain production error condition (often leading into attempts to re-create these conditions on test systems to gather otherwise not available debug logs):

The idea of a ring-buffer based logging would be to have a ring-buffer (of size n) in-memory, containing logs (each log entry is an entry in the ring-buffer) and it has two level-thresholds defined upon creation:

Let's take for example "Info" and "Error" as the two level-thresholds of the ring-buffer:

• When log-events below the lower level-threshold are logged, they are added to the ring-buffer (like "Debug" logs), but not written to any appending output.
• When a log-event higher-or-equal to the lower level-threshold but also lower than the upper threshold gets logged (for example here "Info" or "Warn") then it gets logged to the appending target (file for example) and added to the ring-buffer.
• When a log-event higher-or-equal to the upper threshold gets logged (for example here "Error") then the complete ring-buffer gets sent to the appender (i.e. written to file or similar).

This design allows to always have the last n detail log statements before a critical event (like error) occurred and now these n logs include the debug logs before the error condition occurred, containing exactly these developer-valuable information of what caused that error or led to this. Otherwise debug logs are not sent to the appender. This level-threshold based ring-buffer implementation (as an optional use of course) plays very well with the numeric-based ordering of even custom levels.

This might be helpful in analyzing production issues, but on the downside might impact performance, as the debug log string is always created (but at least not always sent to appenders like a file etc., which happens only on errors or such high-threshold conditions).

Also some overlapping in logs of timestamps need to be addressed, depending on the appending strategy used: If only the output of the ring-buffer (i.e. the elements removed at the end) are further logged (when conforming to the threshold of, say, "Info" or higher), then a hard crash would possibly loose these n not-yet logged elements. To avoid this, the appending strategy of appending higher-or-equal to the defined threshold (say "Info") statements directly upon entry into the ring-buffer can be used. But then, when on an error-condition the complete ring-buffer (including "Debug") gets logged, there might be duplicates of those.

Secret Filter for Logs (PII, Credentials, Tokens, etc.)

To avoid the (unfortunately often seen) problem of accidentally logging sensitive information like Personally Identifiable Information (PII) or credentials of backend systems or tokens or such, which are sometimes logged as part of error messages etc. by accident it would be helpful for a logging API to provide some kind of global hooks to plug-in any secret masking plugins. These could be created and configured by the individual dev teams, possibly taking regulatory requirements into account. Examples could be to redact email addresses, credit card numbers, session/token identifiers, cloud access keys, api-tokens, etc. based on a list of regex or better simpler filter conditions (as performance otherwise might be an issue).

The idea would be to have an easy way of hooking this into the logging API and also some kind of reference implementation to optionally use for the most common use-cases defined above.

As far as I understood the LogValuer somewhat relates to that, but that seems to be more for deliberate handling of sensitive information by the developers (also good and very important). The idea of this comment is more to have something against an "in case something slipped through" scenario, hence based on text matching.

Sorry for the long comment, just my 2 cents about some issues I've had with different logging implementations and an attempt to create a discussion about different ideas to solve them from the perspective of a logging framework implementation.

[tehsphinx]

In my opinion the design of the convenience functions Debug, Info, Warn and Error doesn't make a clear distinction between

• levels
• logging an error type

Have been working / designing multiple company internal structured log packages and have ended up with a design that separates the two:

log.Info().Error(net.ErrClosed, "additional error message", ...)
log.Error().Msg("some error message", ...)

This example output two log messages:

• the first with info level but filling in details from an error type.
• the second with error level but no error type is attached to the log.

EDIT: There is also the issue of changing a Error call to e.g. a Warn or vice versa, messing up the variadic arguments that come after.

[lpar]

First of all, thanks for taking the time to come up with this proposal. The lack of a sufficient standard logging API in Go has been a pain point for way too long.

That said, like many people above I really dislike the convenience functions. Far too much scope for error. They also take up the short namespace, meaning I'll have to write slog.LogAttrs(slog.ErrorLevel, …attributes…) all the time when I'd rather just be writing slog.Error(…attributes…). How about naming the convenience functions slog.Errorf etc?

I personally prefer chaining style log APIs, but one thing I've noticed in using them is that many make it way too easy to do:

log.Error().Str("input", s).Err("error", err)

...and then wonder why I didn't get any output. I think if I were writing (another) logging package now, I'd make the log level be the last part of the chain, so it's (almost) impossible to forget it. So as @beoran suggested:

logger.String("name", "john doe").Warn("message")

Finally, I'd like to amplify this comment from @ainar-g:

It seems like there is no way to either instantiate your own default handler or change the default handler's level. This is confusing, because a lot of projects really just need a way to lower the level to debug when -v or --debug is provided.

The single worst thing about Java's SLF4J/logback combo is that it's really difficult to switch filter level at run time. Whatever the new Go logging package ends up looking like, I'd like switching the level at runtime to be a one-liner.

[deefdragon]

The talk @jba did on slog can be found here for those interested.

[tehsphinx]

First of all a big thank you! Logging and error handling has been part of my daily Go experience for almost 7 years and getting support of structured logging in the standard library is a big step!

In order for this package to become the standard logging package it must have some advantages and at least no major disadvantages over the existing log packages. In its current state I don't see the balance tipping towards this proposal.

Biggest disadvantage so far is the variadic functions making it really hard to read. I feel they are not in line with making code as readable as possible. They put the performance / memory allocations before readability.

Possible advantages

That being said I want to talk about what could be the advantages of this package.

Transport fields in errors

Since this is part of the standard library it could be natively integrated with the errors package. For structured logging one often needs additional fields that are not available where an error is logged. They are available where the error happened, though. An error type could be added to the errors package that can transport fields and when the error is logged the fields are taken from the error and logged, too. There should be a wrapping option to add more fields along the errors return path.

Every other package out there needs to create a new errors package to do this and developers need to buy into using that package.

Stack Traces

Stack traces of errors could be another advantage. They are costly and should not be enabled by default. They could be enabled as part of the logger configuration and would then trigger the error package in the standard library to use an error that contains the stack trace. It could use a new error implementation in order not to bloat the existing error types. The logging package would extract the stack trace, format it (customizable) and log it as an additional field.

In order for existing packages to achieve similar results they first need to offer their own errors package. Secondly all errors from external sources (the standard library, 3rd party libraries) need to be wrapped with the errors package. The stacks then only contain the trace up to the point where the error was wrapped, not beyond into the standard library or the 3rd party package. If this was implemented in Go's standard library natively all of this extra work for developers would be removed + there would be complete stack traces.

TLDR

The biggest advantage I see for having structured logging in the standard library is to integrate it very closely with the errors package. Errors should be able to contain fields and wrapping an error should provide the possibility to add more fields. Errors should also be able to have stack traces to the source where the error happened which would be extracted and logged by the logging package.

[jba]

I found a little issue in the text handler @ancientlore, thanks for finding the bug. Fixed in https://go-review.googlesource.com/c/exp/+/449696.

[jba]

I came up with a reflect-based approach to automate this

@ancientlore, reflecting on structs and looking at field tags seems like a good idea to me; I imagine we'll see several implementations like yours. But I don't believe it's ready to be in this proposal. It could be something we add later, when we're more clear on the desired features.

[jba]

@cschneider4711, good thoughts about security. But I don't think Security should be a level. You said it yourself:

... ease counting and grouping of such events from logs by having a clearer distinction from the other logs as being a potential security-event...

Exactly—you want to easily filter security-relevant logs. For that, you just need a boolean attribute named "security". As for standardizing on the name or having a convenience function, I'll give my standard answer: we just don't know the exact shape this feature would need to take, so it's better to leave it out now and think about adding it later.

[jba]

@cschneider4711:

Correlation-ID for a Request

See this discussion item for motivation. We have Logger.WithContext and slog.Ctx to support this.

Ring-Buffer style of Debug-preserving Appending on Error-Conditions

It sounds like you could implement this as a handler.

Secret Filter for Logs

Redacting secrets and PII is definitely important for logs. We expect most production-strength handlers will provide a way to do this. Our built-in handlers support the ReplaceAttr option, which allows you to provide a function to filter the attributes of a log record.

[jba]

the design of the convenience functions Debug, Info, Warn and Error doesn't make a clear distinction between (a) levels, and (b) logging an error type

@tehsphinx, that's deliberate (and controversial). We think it will be an ergonomic win overall, since the large majority of calls at the Error level have an associated error. Yes, to downgrade an Error to a Warning you have to turn the error value into an attribute; but you didn't have to make it an attribute in the first place, and most Errors don't get downgraded, so the total toil is less.

An error type could be added to the errors package that can transport fields and when the error is logged the fields are taken from the error and logged, too.

This is easy enough to do on your own: just define your own error type with the fields you want, and give it a LogValue method to log those fields. I don't think anything we could add to the errors package would be as good, because it would have to use something like a map[string]any for the fields, which would be slow and type-unsafe.

As for stack traces, you can always write a handler to output one for the log call. Stack traces in errors are out of scope for this proposal.

[jba]

I'd make the log level be the last part of the chain, so it's (almost) impossible to forget it

@lpar, then you couldn't filter on level until you'd done all the work. That's a big disadvantage for systems that have tons of debug logging that they want disabled most of the time.

You've found the great weakness of this style of API for logging: you either give up early filtering or risk forgetting the terminator.

I'd like switching the level at runtime to be a one-liner.

Here is how to switch the level for the entire program at runtime.

Create a LevelVar:

var globalLevel = new(slog.LevelVar) // Info by default

Set up a default logger that uses it:

func main() {
    slog.SetDefault(slog.New(slog.HandlerOptions{Level: globalLevel}.NewJSONHandler(os.Stderr)))
    ...
}

When you want to turn on debug logging, set the LevelVar:

globalLevel.Set(slog.DebugLevel)

That's it.