sonic-net / sonic-buildimage

Scripts which perform an installable binary image build for SONiC
Other
730 stars 1.4k forks source link

memory_checker log spam when feature is disabled #14899

Open bluecmd opened 1 year ago

bluecmd commented 1 year ago

Steps to reproduce the issue:

  1. sudo config feature state telemetry disabled
  2. sudo config feature state snmp disabled

Describe the results you received:

The log contains the following every minute:

May  1 06:10:59.311151 svc-kg-sw1 INFO memory_checker: [memory_checker] Exits without checking memory usage since container 'telemetry' is not running!
May  1 06:11:59.270778 svc-kg-sw1 INFO memory_checker: [memory_checker] Exits without checking memory usage since container 'telemetry' is not running!
May  1 06:12:59.345380 svc-kg-sw1 INFO memory_checker: [memory_checker] Exits without checking memory usage since container 'telemetry' is not running!
May  1 06:13:59.349895 svc-kg-sw1 INFO memory_checker: [memory_checker] Exits without checking memory usage since container 'telemetry' is not running!
May  1 06:14:59.440201 svc-kg-sw1 INFO memory_checker: [memory_checker] Exits without checking memory usage since container 'telemetry' is not running!
May  1 06:15:59.595638 svc-kg-sw1 INFO memory_checker: [memory_checker] Exits without checking memory usage since container 'telemetry' is not running!
May  1 06:16:59.494024 svc-kg-sw1 INFO memory_checker: [memory_checker] Exits without checking memory usage since container 'telemetry' is not running!
May  1 06:17:59.549450 svc-kg-sw1 INFO memory_checker: [memory_checker] Exits without checking memory usage since container 'telemetry' is not running!
May  1 06:18:59.608446 svc-kg-sw1 INFO memory_checker: [memory_checker] Exits without checking memory usage since container 'telemetry' is not running!
May  1 06:19:59.637789 svc-kg-sw1 INFO memory_checker: [memory_checker] Exits without checking memory usage since container 'telemetry' is not running!
May  1 06:20:59.940657 svc-kg-sw1 INFO memory_checker: [memory_checker] Exits without checking memory usage since container 'telemetry' is not running!
May  1 06:21:59.718121 svc-kg-sw1 INFO memory_checker: [memory_checker] Exits without checking memory usage since container 'telemetry' is not running!
May  1 06:22:59.805145 svc-kg-sw1 INFO memory_checker: [memory_checker] Exits without checking memory usage since container 'telemetry' is not running!
May  1 06:23:59.808988 svc-kg-sw1 INFO memory_checker: [memory_checker] Exits without checking memory usage since container 'telemetry' is not running!

Describe the results you expected:

if the feature is disabled, memory_checker should not spam the log about it not running.

Output of show version:


SONiC Software Version: SONiC.202211-kamel.232-a3b178d10
SONiC OS Version: 11
Distribution: Debian 11.6
Kernel: 5.10.0-18-2-amd64
Build commit: a3b178d10
Build date: Sun Apr 30 23:18:50 UTC 2023
Built by: ubuntu@kamel-builder-2023-04-07-1245

Additional information you deem important (e.g. issue happens only occasionally):

Right now we can mitigate this by removing /etc/monit/conf.d/monit_snmp in our Ansible playbooks, but having something like the following in the monit file might be a good solution:

###############################################################################
## Monit configuration for snmp container
###############################################################################
check program container_memory_snmp with path "/usr/bin/run-if-feature-enabled snmp /usr/bin/memory_checker snmp 4294967296"
    if status == 3 for 10 times within 20 cycles then exec "/usr/bin/docker exec snmp supervisorctl restart snmp-subagent"

where run-if-feature-enabled could be:

#!/bin/bash

if [[ "$(sonic-cfggen -d -v 'FEATURE["${1}"]["state"]')" == "disabled" ]]; then
  exit 0
fi
shift
exec "${@}"
gechiang commented 1 year ago

@bluecmd looks like you have a fix in place for this. c an you please go ahead raise a PR for this and we can review and merge it. Thanks!