W3C extended log file format fromatter

[syncplify]

With very little effort I was able to implement a custom Formatter to make the ConsoleWriter write log lines in a W3C Extended Log File Format compliant way (see: https://www.w3.org/TR/WD-logfile.html)

Although JSON structured logging is all the rage today (and, in fact, I personally prefer JSON big time) there's a lot of log analyzers out there that are unable to read JSON, but are perfectly capable of importing W3C log files.

The problem is that there is no Formatter in the FileWriter writer, which is exactly where it would need to be, if we are to be able to save log files in W3C format.

The way I see it, this could be achieved 2 ways:

• add a Formatter to the existing FileWriter interface
• create a whole new W3CLogFileWriter interface (probably not a good idea)

Also, W3C log files require a "header" to be written at the beginning of every file, so ideally each time the file is rotated this header should be automatically written to the file, before we can actually add log lines to it.

Here's a W3C log file header example (taken from Microsoft IIS):

#Software: Internet Information Services 8.0 
#Version: 1.0 
#Date: 2021-05-02 17:42:15 
#Fields: time c-ip cs-method cs-uri-stem sc-status cs-version

This could probably be achieved with some custom wizardry in the Cleaner function of the FileWriter interface, but while we're at it, why not make everyone life's easier, and provide a string property for it, and have the Writer automagically add it at the beginning of each new file, right when the log its rotated?

JSON logging is still far far better, I think we can all agree on that. But W3C log file format is not dead, and is required by more users than I originally thought possible. Any feedback on this would be greatly appreciated. Thank you in advance.

[phuslu]

what if https://play.golang.org/p/t3SBYkF4YRl or https://play.golang.org/p/eNtrQBmUrSn ?

[syncplify]

That's a good approach. I'll toy a little bit with it on Monday, to see what I can do.

The "log header" part each time a file is rotated remains an open issue. I could add lines by logging them of course, but if lines come in at the same time from other groutines, the header may end up intertwined with those lines. Headers should be "atomically written".

But thank you for pointing me the right direction regarding writer-chaining to address the W3C log format. Much appreciated.

[phuslu]

For the "log header", this is a problem. When I implementing csv logger, I also encountered this problem.

Currently I have no good ideas to solve this problem in a pretty way.

[phuslu]

A straightforward way is, add a LogHeader callback to FileWriter

type FileWriter struct {
    // LogHeader specifies an optional log header after rotation
    LogHeader func(fileinfo os.FileInfo) []byte
}

But this a little ugly, because it tied logger and writer. Any thoughts/comments ?

[syncplify]

I agree, this is an "easy" solution but comes with a "cost" (logger and writer being tied as you mentioned). On the other hand the weight of this tie wouldn't be heavier than the Rotate callback in my humble opinion. Leaving the callback function pointer to nil would ensure the Writer works just like it does now, but would provide an option for those who need it to specify a custom behavior that only applies to this writer.

If, on the other hand, we think that a custom atomic header would be useful for other writers as well, then maybe we can come up with a dedicated additional interface{} that all Writers implement (on top of io.Writer and io.WriteCloser) although this could end up being slightly more cumbersome to implement.

One more possibility could be (consequences to be evaluated) to "derive" a descendant of FileWriter... something like:

type FileWriterWithHeaderCallback struct {
    FileWriter
    /...
}

(but this may or may not be a rabbit hole, just put it out there as food for thoughts)

[phuslu]

I create a PR based on "Header" callback idea, https://github.com/phuslu/log/pull/42 please take a look and comment, thanks!

[syncplify]

Checked your PR #42 and it looks perfect to me. Easy, elegant, no frills, just plain effective.

[phuslu]

thanks, I merged and tagged v1.0.73