[Dual-ToR] Continues warning messages for non-mux ports

[volodymyrsamotiy]

Description

Steps to reproduce the issue:

1. Bring up Dual ToR community topology (described in "sonic-mgmt" repo)
2. On "active" and/or "standby" ToR verify that MUX port is up and "healthy":

   root@sonic-dualtor-active:/home/admin# show muxcable status Ethernet2
   PORT        STATUS    SERVER_STATUS    HEALTH    HWSTATUS    LAST_SWITCHOVER_TIME
   ----------  --------  ---------------  --------  ----------  ---------------------------
   Ethernet2   active    active           healthy   consistent  2022-Dec-22 12:10:10.120741

3. On "active" and/or "standby" ToR verify that warning messages are printed to syslog for non-mux ports every 1 minute:

   root@sonic:/home/admin# tail -F /var/log/syslog | grep ycabled
   Dec 28 12:46:47.864491 sonic WARNING pmon#ycabled: Could not retreive fieldvalue pairs for Ethernet70, inside config_db table MUX_CABLE
   Dec 28 12:46:47.914971 sonic WARNING pmon#ycabled: Could not retreive fieldvalue pairs for Ethernet106, inside config_db table MUX_CABLE
   Dec 28 12:46:47.916068 sonic WARNING pmon#ycabled: Could not retreive fieldvalue pairs for Ethernet112, inside config_db table MUX_CABLE
   Dec 28 12:46:47.917248 sonic WARNING pmon#ycabled: Could not retreive fieldvalue pairs for Ethernet60, inside config_db table MUX_CABLE
   Dec 28 12:46:48.013374 sonic WARNING pmon#ycabled: Could not retreive fieldvalue pairs for Ethernet108, inside config_db table MUX_CABLE
   Dec 28 12:46:48.036657 sonic WARNING pmon#ycabled: Could not retreive fieldvalue pairs for Ethernet104, inside config_db table MUX_CABLE
   Dec 28 12:46:48.037545 sonic WARNING pmon#ycabled: Could not retreive fieldvalue pairs for Ethernet120, inside config_db table MUX_CABLE
   Dec 28 12:46:48.038952 sonic WARNING pmon#ycabled: Could not retreive fieldvalue pairs for Ethernet114, inside config_db table MUX_CABLE
   Dec 28 12:46:48.067108 sonic WARNING pmon#ycabled: Could not retreive fieldvalue pairs for Ethernet0, inside config_db table MUX_CABLE
   Dec 28 12:46:48.069149 sonic WARNING pmon#ycabled: Could not retreive fieldvalue pairs for Ethernet74, inside config_db table MUX_CABLE
   ...

Describe the results you received:

On "active" and "standby" ToR continues warning messages are printed to syslog for non-mux ports every 1 minute:

Dec 28 12:46:48.069149 sonic WARNING pmon#ycabled: Could not retreive fieldvalue pairs for Ethernet74, inside config_db table MUX_CABLE

Describe the results you expected:

No continues warning messages for non-mux ports

Output of show version:

SONiC Software Version: SONiC.202205_1_rc.10-cf489f92a_Internal
Distribution: Debian 11.6
Kernel: 5.10.0-18-2-amd64
Build commit: cf489f92a
Build date: Tue Dec 27 12:49:12 UTC 2022
Built by: sw-r2d2-bot@r-build-sonic-ci02-244

Platform: x86_64-mlnx_msn2700-r0
HwSKU: Mellanox-SN2700-D48C8
ASIC: mellanox
ASIC Count: 1
Serial Number: MT1805K20439
Model Number: MSN2700-CS2F
Hardware Revision: A2
Uptime: 12:59:06 up  6:01,  1 user,  load average: 0.48, 0.86, 1.07
Date: Wed 28 Dec 2022 12:59:06

Docker images:
REPOSITORY                                         TAG                                 IMAGE ID       SIZE
docker-syncd-mlnx                                  202205_1_rc.10-cf489f92a_Internal   3dd5cfe14079   870MB
docker-syncd-mlnx                                  latest                              3dd5cfe14079   870MB
docker-platform-monitor                            202205_1_rc.10-cf489f92a_Internal   04ba8b63408b   874MB
docker-platform-monitor                            latest                              04ba8b63408b   874MB
urm.nvidia.com/sw-nbu-sws-sonic-docker/sonic-wjh   1.3.1                               410975934531   310MB
docker-orchagent                                   202205_1_rc.10-cf489f92a_Internal   d5a77d2c1034   479MB
docker-orchagent                                   latest                              d5a77d2c1034   479MB
docker-fpm-frr                                     202205_1_rc.10-cf489f92a_Internal   15579cbb07f1   490MB
docker-fpm-frr                                     latest                              15579cbb07f1   490MB
docker-teamd                                       202205_1_rc.10-cf489f92a_Internal   d5bde31d1c69   460MB
docker-teamd                                       latest                              d5bde31d1c69   460MB
docker-macsec                                      202205_1_rc.8-223871506_Internal    6c6c7eab485a   462MB
docker-snmp                                        202205_1_rc.10-cf489f92a_Internal   386918d216da   489MB
docker-snmp                                        latest                              386918d216da   489MB
docker-dhcp-relay                                  202205_1_rc.8-223871506_Internal    40d7a5e36c44   454MB
docker-sonic-telemetry                             202205_1_rc.10-cf489f92a_Internal   db85dfa47571   525MB
docker-sonic-telemetry                             latest                              db85dfa47571   525MB
docker-lldp                                        202205_1_rc.10-cf489f92a_Internal   c61acc1f4ec4   487MB
docker-lldp                                        latest                              c61acc1f4ec4   487MB
docker-mux                                         202205_1_rc.10-cf489f92a_Internal   f8ef97f10772   493MB
docker-mux                                         latest                              f8ef97f10772   493MB
docker-database                                    202205_1_rc.10-cf489f92a_Internal   07c7169f3dc3   444MB
docker-database                                    latest                              07c7169f3dc3   444MB
docker-router-advertiser                           202205_1_rc.10-cf489f92a_Internal   ac73c735984e   444MB
docker-router-advertiser                           latest                              ac73c735984e   444MB
docker-nat                                         202205_1_rc.10-cf489f92a_Internal   2261eda769df   430MB
docker-nat                                         latest                              2261eda769df   430MB
docker-sflow                                       202205_1_rc.10-cf489f92a_Internal   b2ec2833ce86   428MB
docker-sflow                                       latest                              b2ec2833ce86   428MB
docker-sonic-mgmt-framework                        202205_1_rc.10-cf489f92a_Internal   8563d382fb45   557MB
docker-sonic-mgmt-framework                        latest                              8563d382fb45   557MB

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):

[yxieca]

@vdahiya12 will work on this issue.

[liat-grozovik]

@vdahiya12 could you please update on progress and if this is expected to be fixed on both 202205 and 202305? IMO flooded log is something critical