rs / zerolog

Zero Allocation JSON Logger
MIT License
10.33k stars 564 forks source link

Add an option to limit max log's size after escaping #660

Open joeyave opened 5 months ago

joeyave commented 5 months ago

I use Zerolog for logging some JSON strings with RawJSON function. Sometimes that JSON can be very large. The problem is that logging system has certain limitation for length of the log's line and I can't control that. Max length is about 16KB. I really need to log at least part of that JSON (with some other metadata) so I've tried to trim it to 16KB and use Bytes function.

var LogSizeLimit = 15000

len(jsonBytes) > LogSizeLimit {
    jsonBytesCopy := make([]byte, len(jsonBytes))
    copy(jsonBytesCopy, jsonBytes)
    event = event.Bytes("body", jsonBytesCopy[:util.LogSizeLimit])
}

The problem here is that Bytes function escapes some special characters (double quotes etc). As result that string becomes bigger and exceeds my limit of 16KB.

I've tried to lower LogSizeLimit to let's say 15KB. It helps but not for all cases. There should be more elegant solution.

Also I thought about escaping that trimmed JSON string myself and counting how may new symbols added. Than I can trim that string even more. But that solution sounds somehow complicated and bug-prone.

len(jsonBytes) > LogSizeLimit {
    jsonBytesCopy := make([]byte, len(jsonBytes))
    copy(jsonBytesCopy, jsonBytes)

    unescaped := jsonBytesCopy[:util.LogSizeLimit]
    diff := len(util.EscapeForZerolog(unescaped)) - util.LogSizeLimit

    event = event.Bytes("body", unescaped[:util.LogSizeLimit-diff])
}

Maybe I can disable escaping in Zerolog? Or should I strictly limit my log's line size to half of the logging system's limit: 16KB / 2 = 7.5KB? Maybe I could limit log's size on Zerolog's side?