Closed jsturtevant closed 6 months ago
We identified a perf impact when writing logs.
Would be curious to learn more about costs of logging to pipe/fifo.
Would be curious to learn more about costs of logging to pipe/fifo.
@radu-matei @lann could you share some of your findings?
I suspect that the performance issues have more to do with mutex contention in FifoLogger
, but I haven't done any analysis to confirm.
mutex contention
I wonder if going this route would be more viable instead of introducing a custom env var? parking_lot's mutex or maintaining a message queue.
I think it should be both. There is an expectation from users that since they are using env
crate they can adjust the log level. In the current set up, it is impossible to get trace
logs from the shim or other components since we only go to the debug
level.
@mxpv I've logged #249 for further investigation on the logging issue.
Of the feedback in the PR above above which changes would you like to see before merging this?
Let's have this one in. We can follow up with further improvements.
We identified a perf impact when writing logs. In this case it would be helpful to only run with logs set at a "warning" or "error" level. We attempted to turn most logs off via containerd with
containerd -l fatal
but it still dumped logs.After some investigation it appears containerd only sets a flag not a level (https://github.com/containerd/containerd/blob/7467d81987e1a54dd2a89e052ae9429a4c921e9e/core/runtime/v2/binary.go#L66-L68) and our code will default to info https://github.com/containerd/rust-extensions/blob/cc445f54c0fa8e9b35376550874422c932ad1566/crates/shim/src/logger.rs#L135-L139
changes
This reads the environment variable
RUST_LOG
and will use value from there if set. It can be set to any values in the log crate. Otherwise the behavior is the same. If the-debug
flag is passed it will choose the highest of the two values.This also introduces ability to configure the default setting through the shim
Config
object:If all the configuration fails it will fallback to logging to to
info
level.