search

containerd / runwasi

Facilitates running Wasm / WASI workloads managed by containerd
Apache License 2.0
1.01k stars 82 forks source link

Enable Binary logging support #630

Open jsturtevant opened 6 days ago

jsturtevant commented 6 days ago

Containerd has pluggable logging which includes binary logging: https://github.com/containerd/containerd/blob/c7f7fa5390f761b83c3a69cd5709a8d04d912969/core/runtime/v2/README.md#logging

Currently we support fifo/file/npipe

Mossaka commented 6 days ago

What's the main benefits of supporting Binary logging?

squillace commented 6 days ago

the main benefit of binary logging is that you can do highly structured snapshots and various other customized things to not only compress but treat the log as a kind of database. So, for example, systemd and mysql (in fact, almost every db or db service) uses binary logging. Size is not the main concern, but "structured responsiveness".

outside of that, you can just compress the text log if size is your only concern (though ongoing size is typically smaller with binary).

In this case, we have a Windows customer who needs this to make runwasi a "drop-in" replacement for the current shim.

jsturtevant commented 4 days ago

From https://github.com/containerd/containerd/issues/6639:

Example policies that a custom log plugin might enforce include: Log data is pre-processed to remove potentially sensitive information (e.g. via https://github.com/stripe/unilog) Pre-processed log data is not allowed to be written to persistent disk. Log lines may be persisted to remote storage at different sampling levels depending on content.

learned a little bit more about how binary loggers should work: they are expected to be called, and piping I/O in to the custom logger. An example the logger is in https://github.com/containerd/containerd/pull/3085