[TD3] crm returns incorrect counter for used FDB entry

[bingwang-ms]

Description crm show resources fdb gives incorrect FDB entry count. It returns 1 although there is no FDB. The polling interval has been set to 1 second.

root@str2-7050cx3-acs-01:~# crm show resources fdb

Resource Name      Used Count    Available Count
---------------  ------------  -----------------
fdb_entry                   1              32767

root@str2-7050cx3-acs-01:~# fdbshow 
No.    Vlan    MacAddress    Port    Type
-----  ------  ------------  ------  ------
Total number of entries 0

root@str2-7050cx3-acs-01:~# show mac
No.    Vlan    MacAddress    Port    Type
-----  ------  ------------  ------  ------
Total number of entries 0

Steps to reproduce the issue: The issue is not consistently repro.

1. Run test_crm_fdb_entry.

Describe the results you received: crm show resources fdb should return 0 if no FDB entry exists.

Describe the results you expected: crm show resources fdb should return 1.

Additional information you deem important (e.g. issue happens only occasionally):

**Output of `show version`:**

SONiC Software Version: SONiC.HEAD.169-c146eeaa
Distribution: Debian 10.7
Kernel: 4.19.0-9-2-amd64
Build commit: c146eeaa
Build date: Tue Dec 22 14:07:33 UTC 2020
Built by: johnar@jenkins-worker-22

Platform: x86_64-arista_7050cx3_32s
HwSKU: Arista-7050CX3-32S-C32
ASIC: broadcom
ASIC Count: 1
Serial Number: JPE20033081
Uptime: 09:45:22 up  6:20,  2 users,  load average: 0.43, 1.83, 2.18

Docker images:
REPOSITORY                    TAG                 IMAGE ID            SIZE
docker-sonic-telemetry        HEAD.169-c146eeaa   d6f316245613        555MB
docker-sonic-telemetry        latest              d6f316245613        555MB
docker-syncd-brcm             HEAD.169-c146eeaa   6cf635032824        726MB
docker-syncd-brcm             latest              6cf635032824        726MB
docker-snmp                   HEAD.169-c146eeaa   e76565656569        525MB
docker-snmp                   latest              e76565656569        525MB
docker-teamd                  HEAD.169-c146eeaa   da0d0d5b3a44        524MB
docker-teamd                  latest              da0d0d5b3a44        524MB
docker-sonic-mgmt-framework   HEAD.169-c146eeaa   55718ad9e66b        641MB
docker-sonic-mgmt-framework   latest              55718ad9e66b        641MB
docker-router-advertiser      HEAD.169-c146eeaa   2558e1230a8d        480MB
docker-router-advertiser      latest              2558e1230a8d        480MB
docker-platform-monitor       HEAD.169-c146eeaa   ddd90e632b2f        605MB
docker-platform-monitor       latest              ddd90e632b2f        605MB
docker-lldp                   HEAD.169-c146eeaa   9d9b758fff1f        520MB
docker-lldp                   latest              9d9b758fff1f        520MB
docker-dhcp-relay             HEAD.169-c146eeaa   ee7e2f914bfa        487MB
docker-dhcp-relay             latest              ee7e2f914bfa        487MB
docker-database               HEAD.169-c146eeaa   cf96413b03e7        480MB
docker-database               latest              cf96413b03e7        480MB
docker-orchagent              HEAD.169-c146eeaa   425d0ad7b0da        554MB
docker-orchagent              latest              425d0ad7b0da        554MB
docker-nat                    HEAD.169-c146eeaa   1ef25954b9f9        526MB
docker-nat                    latest              1ef25954b9f9        526MB
docker-fpm-frr                HEAD.169-c146eeaa   2f134a3a81ce        539MB
docker-fpm-frr                latest              2f134a3a81ce        539MB
docker-sflow                  HEAD.169-c146eeaa   cc3ad45aa34c        523MB
docker-sflow                  latest              cc3ad45aa34c        523MB

**Attach debug file `sudo generate_dump`:**

```
(paste your output here)
```

[prsunny]

I tested this manually and could observe counters being updated correctly. May be is it related to polling interval? I did the learn/flush after setting polling interval.

admin@str2-acs-02:~$ show mac
No.    Vlan    MacAddress    Port    Type
-----  ------  ------------  ------  ------
Total number of entries 0

admin@str2-acs-02:~$ crm show resources fdb

Resource Name      Used Count    Available Count
---------------  ------------  -----------------
fdb_entry                   0              32767

admin@str2-acs-02:~$ show mac
  No.    Vlan  MacAddress         Port       Type
-----  ------  -----------------  ---------  -------
    1    1000  98:03:9B:03:22:01  Ethernet4  Dynamic
Total number of entries 1
admin@str2-acs-02:~$ crm show resources fdb

Resource Name      Used Count    Available Count
---------------  ------------  -----------------
fdb_entry                   1              32766
admin@str2-acs-02:~$ fdbclear 
FDB entries are cleared.
admin@str2-acs-02:~$ show mac
No.    Vlan    MacAddress    Port    Type
-----  ------  ------------  ------  ------
Total number of entries 0
admin@str2-acs-02:~$ crm show resources fdb

Resource Name      Used Count    Available Count
---------------  ------------  -----------------
fdb_entry                   0              32767

[bingwang-ms]

The issue shouldn't be caused by polling interval because I have set it to 1 second. The issue is not consistently repro. I only encountered twice when debugging test_crm_fdb.

[bingwang-ms]

I just repro. Can you try to run test_crm_fdb_entry continuously and see if it repro?

admin@str2-7050cx3-acs-01:~$ crm show summary 

Polling Interval: 1 second(s)

admin@str2-7050cx3-acs-01:~$ fdbshow 
No.    Vlan    MacAddress    Port    Type
-----  ------  ------------  ------  ------
Total number of entries 0
admin@str2-7050cx3-acs-01:~$ crm show resources fdb

Resource Name      Used Count    Available Count
---------------  ------------  -----------------
fdb_entry                   1              32767

admin@str2-7050cx3-acs-01:~$ fdbshow 
No.    Vlan    MacAddress    Port    Type
-----  ------  ------------  ------  ------
Total number of entries 0

[george-deng88]

I think it is not a issue, because TD3 internal used 1 fdb counter:

drivshell>l2 show l2 show mac=00:e0:ec:cc:1b:a2 vlan=1 GPORT=0x0 modid=0 port=0/cpu0 Static CPU

it is not show in sonic show mac, but it can show in crm.