search

networkservicemesh / deployments-k8s

Apache License 2.0
42 stars 34 forks source link

Change log level on the fly without restart #12296

Open szvincze opened 1 week ago

szvincze commented 1 week ago

Overview

Currently the log level can be configured via environment variable NSM_LOG_LEVEL which means the process will use the configured value for its whole lifetime. It is fine in most of the cases, but when one would like to change the log level because closer monitoring is needed, it would be expected to be able to do the change without restarting the process that would be monitored. In the vast majority of the cases the restart makes impossible the immediate monitoring action, because after the configuration change we should initiate restart and then just wait for the issue to happen again.

For temporary setting it would be good if the log level could be changed via sending IPC signals to the process. For example, if we would like to monitor forwarder-vpp then we send SIGUSR1 to the process and it changes its log level to TRACE, when we captured the needed logs the original log level can be restored by sending signal SIGUSR2.

arp-est commented 1 week ago

Hi, We also ha a reference implementation, if this signalling approach is okay with you guys

https://github.com/networkservicemesh/cmd-forwarder-vpp/pull/1171

Ex4amp1e commented 1 week ago

Possible solution:

  1. Add term "NSM config"
  2. Move LOG_LEVEL configuring from ENV config to NSM config
  3. [Optional] In future - move other ENVs

Algorithm for LOG_LEVEL:

  1. Every cmd repo will contain structure, that we will read data from file
  2. On every request there will be check if the file has been modified, in positive case - read new value and update LOG_LEVEL in structure
  3. In scope k8s deployment - we will store data in configmap and mount it into each pod as file
denis-tingaikin commented 1 week ago

cc @edwarnicke