EncoderConfig for custom Registered Encoders

[SteelPhase]

I'm wondering if there would be any interest in making zapcore.EncoderConfig extendable so that custom Encoders could rely on it for additional configuration they may need. As far as I understand the code base, this shouldn't break anything.

The reason I'm asking is that I've built a custom encoder for zap for use in our development environment, but it has some additional configuration over the standard zapcore.EncoderConfig. There doesn't seem to be a great way to configure this custom encoder outside of what already exists for zapcore.EncoderConfig due to how RegisterEncoder works. Ideally it could be as simple as the following.

AdditionalConfig map[string]interface{} `json:"additionalConfig" yaml:"additionalConfig"`

[rustysys-dev]

I would potentially be interested in this!

That being said could you give an example of what functionality this might enable for others to understand your proposal better?

[SteelPhase]

Sure, I have an example, based off of the idea here. This is just a rough example of how i'd use it. Probably worth handling the !ok moments as actual errors. I have been messing with the console encoder so that results can be more human readable...

Like this

steelphase@macbook:demo$LOG_FORMAT="CONSOLE-PRETTIER" go run main.go demo-action
2020-05-18T09:34:08.870-0400    INFO    ec.er   demo/main.go:64 we did the thing
{
  "example-environment-bool": true,
  "example-cli-arg-bool": true
}

package zapext

import (
    "go.uber.org/zap"
    "go.uber.org/zap/buffer"
    "go.uber.org/zap/zapcore"

    "go.company.com/org/app/v4/pkg/logging/internal/bufferpool"
)

func init() {
    zap.RegisterEncoder("console-prettier", func(cfg zapcore.EncoderConfig) (zapcore.Encoder, error) {
        jsonConfig := cfg
        jsonConfig.TimeKey = ""
        jsonConfig.LevelKey = ""
        jsonConfig.NameKey = ""
        jsonConfig.CallerKey = ""
        jsonConfig.MessageKey = ""
        jsonConfig.StacktraceKey = ""

        consoleConfig := cfg
        consoleConfig.StacktraceKey = ""

        pce := prettyConsoleEncoder{
            EncoderConfig:  &cfg,
            Encoder:        zapcore.NewConsoleEncoder(jsonConfig),
            consoleEncoder: zapcore.NewConsoleEncoder(consoleConfig),
        }

        if c, ok := cfg.AdditionalConfig["console-prettier"]; ok {
            if pcec, ok := c.(PrettyConsoleEncoderConfig); ok {
                pce.PrettyConsoleEncoderConfig = pcec
            }
        }

        return &pce, nil
    })
}

type jsonDecorator func(dst *buffer.Buffer, src *buffer.Buffer)

// A PrettyConsoleEncoderConfig allows users to configure the additional encoders for prettyConsoleEncoder
type PrettyConsoleEncoderConfig struct {
    DecorateJSON jsonDecorator `json:"decorateJSON" yaml:"decorateJSON"`
}

type prettyConsoleEncoder struct {
    *PrettyConsoleEncoderConfig
    *zapcore.EncoderConfig
    zapcore.Encoder
    consoleEncoder zapcore.Encoder
}

func (pce *prettyConsoleEncoder) Clone() zapcore.Encoder {
    return &prettyConsoleEncoder{
        PrettyConsoleEncoderConfig: pce.PrettyConsoleEncoderConfig,
        EncoderConfig:              pce.EncoderConfig,
        Encoder:                    pce.Encoder.Clone(),
        consoleEncoder:             pce.consoleEncoder.Clone(),
    }
}

func (pce *prettyConsoleEncoder) EncodeEntry(ent zapcore.Entry, fields []zapcore.Field) (*buffer.Buffer, error) {
    line, err := pce.consoleEncoder.EncodeEntry(ent, nil)
    if err != nil {
        return line, err
    }

    jsonRawBuf, err := pce.Encoder.EncodeEntry(ent, fields)
    if err != nil {
        return line, err
    }
    defer jsonRawBuf.Free()

    jsonBuf := bufferpool.Get()
    defer jsonBuf.Free()

    pce.DecorateJSON(jsonBuf, jsonRawBuf)

    line.Write(jsonBuf.Bytes())

    // If there's no stacktrace key, honor that; this allows users to force
    // single-line output.
    if ent.Stack != "" && pce.StacktraceKey != "" {
        line.AppendByte('\n')
        line.AppendString(ent.Stack)
    }

    if pce.LineEnding != "" {
        line.AppendString(pce.LineEnding)
    } else {
        line.AppendString(zapcore.DefaultLineEnding)
    }
    return line, err
}

Example of what setting of the zap.Config would look like.

func getConfig() zap.Config {
    jsonDecorator := &jsoncolor.Decorator{}

    config := zap.NewProductionConfig()
    config.Encoding = "console-prettier"
    config.EncoderConfig.TimeKey = "timestamp"
    config.EncoderConfig.EncodeLevel = encoders.EhnancedColorLevelEncoder(true, nil, nil)
    config.EncoderConfig.EncodeTime = encoders.ISO8601ColorTimeEncoder(&decorate.Format{Foreground: colors.BrightCyan})
    config.EncoderConfig.EncodeCaller = encoders.ShortColorCallerEncoder(&decorate.Format{Foreground: colors.Yellow})
    config.EncoderConfig.EncodeName = encoders.FullColorNameEncoder(&decorate.Format{Foreground: colors.Green})

    if config.AdditionalConfig == nil {
        config.AdditionalConfig = map[string]interface{}{}
    }

    config.AdditionalConfig["console-prettier"] = zapext.PrettyConsoleEncoderConfig{
        DecorateJSON : jsonDecorator.Decorate
    }

    return config
}

[prashantv]

I think having a way to pass additional encoder-specific configuration makes sense, since we allow custom encoders to be registered.

EncoderConfig is currently intended to unmarshalled from JSON/YAML, so we should think about how to maintain that experience for most cases.

[SteelPhase]

So about that. I don't even think json marshalling functions today. I'm assuming EncoderConfig being marshallable as JSON is left over from a point when it would actually work. I'm pretty sure json can't marshal golang funcs, so LevelEncoder, TimeEncoder, DurationEncoder, CallerEncoder, and NameEncoder will cause errors like json: unsupported type: zapcore.LevelEncoder

[prashantv]

JSON marshalling doesn't work, but JSON unmarshalling should work -- it's useful to specify configurations in JSON/YAML, and use that to initialize a logger. Marshalling isn't quite as useful (it can be useful for introspection, but not as common as loading a config)

[SteelPhase]

I see that now. Though it seems not well implemented. It can't unmarshal custom encoders for any of those values, it just uses the switch default value instead.

[sc0Vu]

I got the same issue here. Want to formalize the log in console. After trace zap code, I was thinking how to replace the jsonEncoder to another. @SteelPhase did your solution work?

[SteelPhase]

yes, I use my custom json encoder quite a bit. My current solution is modified from what I posted.

[sc0Vu]

Cool! Seems zap only support json and I want the data looks like key=value.

So I forked zap and updated to this:

DEBUG config/logger.go:82 Got response: getblockpeak [139.315401ms] server=us1.prenet.diode.io:41046 

You can found here: https://github.com/diodechain/zap