WIP: Structured and Levelled Logging

[bradleyfalzon]

One thing production GopherCI doesn't have is structured logging.

• Structured logging to make filtering logs simpler, and graphing metrics easier.
• Errors to go to 3rd party service for email notification (including stack traces).
• Levelled logging, so debug logs can be used for development and info+ for production. I don't believe we'd need more than debug, info and error.

My logging experience is mostly just with github.com/sirupsen/logrus, I'll use this because I know it, but there may be other libraries more appropriate (ideally smaller).

For email notifications, the logger will likely be aware of the 3rd party service, this is to ensure stack traces can be generated. Generally, loggers shouldn't be aware of these services, the application should just output logs to stdout, and different tools can watch these logs and behave as desired. Having the logger aware of these services means we don't need to log the full stacktrace, and can just send it directly. This is my current thinking, I'm not certain about this just yet.

• Don't write the output of tools, this is no longer necessary to store as we have it in DB now thanks to #109.
• Create a internal/logging/ package which contains shared logging interfaces and constructors, such as NewLog.
• Create at least one interface to support adding fields to logs - this is where things get hard, most logging libraries that do provide interfaces have concrete types (such as structs/maps) in them, which would tie any client package to that implementation. We'd likely need to provide these types.
• This is only for internal use, coupling logging to a particular implementation wouldn't have any major drawbacks (one commit could swap the logger).

[bradleyfalzon]

These are rough thoughts.

https://github.com/bketelsen/logr/blob/master/logr.go

I like this because context logging returns itself and accepts basic types WithField(name string, value interface{}) Logger, there's no WithFields, which is usually where other interfaces require custom structs.

https://godoc.org/github.com/sirupsen/logrus#FieldLogger

This is larger than I need, and contains defined type Fields (which is just map[string]interface{}), but returns the *Entry type.

https://godoc.org/github.com/go-kit/kit/log

go-kit's logger is probably the better so far, but the consumers are tied to it if they want levelled logging, but we could abstract that away I think.

[bradleyfalzon]

OK I'm leaning towards the following, provided by ./internal/logger and a func New(...) Logger which provides a logger wrapping logrus and sentry.

// Logger is a service to write structured, levelled logs with context.
type Logger interface {
    // Debug level for developer concerned debugging, not visible in production.
    Debug(args ...interface{})
    Debugf(format string, args ...interface{})

    // Info logs general events.
    Info(args ...interface{})
    Infof(format string, args ...interface{})

    // Error logs, errors. An error should only be logged once.
    Error(args ...interface{})
    Errorf(format string, args ...interface{})

    // Fatal logs an error and then immediately terminates execution.
    Fatal(args ...interface{})
    Fatalf(format string, args ...interface{})

    // With adds context to a logger.
    With(name string, value interface{}) Logger
}

[bradleyfalzon]

Closed via #122.