System-wide logging improvements

[kralicky]

The zap logger setup we have has been working well so far, but as the project expands we should consider: a) potential improvements, refactors, or redesigns of how we use loggers across places in the code b) newer/better/standardized libraries to switch to

I've tried a few other logging libraries, and so far none seem to be as good as zap for our use cases, but more research is required.

slog (upcoming standard library structured logging)

• It has a design similar to zap, and a good feature set and API
• It is still very much in development, and lacks a lot of features that I like from zap (e.g. custom log output handlers)

zerolog

• Inspired by/derived from zap, but with fewer options for customization. For this reason it probably isn't a good fit.
• Performance improvements are very interesting though - I've noticed that zap is pretty heavy on memory allocations (although there are a lot of optimizations we can make in our own code).

---

Some issues we are running into currently:

• Plugin logs are weird. I'm considering making a fork of go-plugin and swapping out hclog because the interoperability with zap is complicated
• Sampling lacks important features such as logging the number of dropped logs. I've implemented this in my fork of zap which we use, but it's annoying to maintain.

[jan-law]

As discussed earlier, noting some additional improvement suggestions

• fetch agent plugin logs remotely instead of manually
• send test log output to a file
• I've kept the sampling behaviour the same as before. However, slog-sampling lets you configure the % rate of sampled logs for each log level (eg log 10% Info, 100% error and above). Suggestions welcome