kamelnetworks / sonic-buildimage

Kamel Networks' pending patches to SONiC
https://dev.azure.com/kamelnetworks/sonic/_build?definitionId=5
Other
3 stars 1 forks source link

queuestat: Empty on 202211 #13

Open bluecmd opened 6 months ago

bluecmd commented 6 months ago

Seems to have worked on 202111 but on 202211 it is empty:

(vrf:mgmt)bluecmd@ixp-lab-sw1:~$ queuestat
  Port    TxQ    Counter/pkts    Counter/bytes    Drop/pkts    Drop/bytes
------  -----  --------------  ---------------  -----------  ------------

  Port    TxQ    Counter/pkts    Counter/bytes    Drop/pkts    Drop/bytes
------  -----  --------------  ---------------  -----------  ------------

  Port    TxQ    Counter/pkts    Counter/bytes    Drop/pkts    Drop/bytes
------  -----  --------------  ---------------  -----------  ------------
[..]
bluecmd commented 6 months ago

Seems the COUNTERS_QUEUE_NAME_MAP table is empty in COUNTERS_DB:

127.0.0.1:6379[2]> hgetall COUNTERS_QUEUE_NAME_MAP
(empty array)

In 202111 it is populated.

bluecmd commented 6 months ago

Seems to be handled by "Flex counter" in SONiC. It seems enabled:

(vrf:mgmt)bluecmd@ixp-lab-sw1:~$ sudo counterpoll show
Type                        Interval (in ms)    Status
--------------------------  ------------------  --------
QUEUE_STAT                  default (10000)     enable
PORT_STAT                   default (1000)      enable
PORT_BUFFER_DROP            default (60000)     enable
RIF_STAT                    default (1000)      enable
QUEUE_WATERMARK_STAT        default (60000)     enable
PG_WATERMARK_STAT           default (60000)     enable
PG_DROP_STAT                default (10000)     enable
BUFFER_POOL_WATERMARK_STAT  default (60000)     enable
ACL                         10000               enable

I do see the following message in the log:

Feb 28 13:45:07.272066 ixp-lab-sw1 WARNING python3: :- blockable: Key '{COUNTERS_PORT_NAME_MAP}' unavailable in database '{COUNTERS_DB}'
Feb 28 13:45:07.278638 ixp-lab-sw1 WARNING python3: :- blockable: Key '{COUNTERS_PORT_NAME_MAP}' unavailable in database '{COUNTERS_DB}'
bluecmd commented 6 months ago

Might be related to that we do not have a BUFFER_QUEUE table in the config? Maybe related to this feature: https://github.com/sonic-net/SONiC/blob/417a096734f24b6d6ee8e852c74a164410568fe1/doc/SONiC_202205_Release_Notes.md#queue-statistics-based-on-queue-configurations-and-not-max

Although this shouldn't matter, as if the interface is not specified it should default to max number of buffer counters.

Ethernet60 seems to read a sensible default anyway:

Mar  1 15:35:03.837925 ixp-lab-sw1 INFO swss#orchagent: :- initializeQueues: Get 20 queues for port Ethernet60
Mar  1 15:35:03.879777 ixp-lab-sw1 INFO swss#orchagent: :- initializeQueues: Get queues for port Ethernet60

getQueueTypeAndIndex seems to be called 3 times, but I can find 4 call sites, so maybe the missing one is ours?

bluecmd commented 6 months ago

Loading the following configuration made queue stats appear for Ethernet68:

{
  "BUFFER_POOL": {
    "ingress_lossless_pool": {
      "size": "32744448",
      "type": "ingress",
      "mode": "static"
    },
    "egress_lossy_pool": {
      "size": "32744448",
      "type": "egress",
      "mode": "dynamic"
    }
  },
  "BUFFER_PROFILE": {
    "ingress_lossy_profile": {
      "pool": "ingress_lossless_pool",
      "size": "0",
      "static_th": "32744448"
    },
    "egress_lossy_profile": {
      "pool": "egress_lossy_pool",
      "size": "0",
      "dynamic_th": "3"
    }
  },
  "BUFFER_PG": {
    "Ethernet68|0": {
      "profile": "ingress_lossy_profile"
    }
  },
  "BUFFER_QUEUE": {
    "Ethernet68|0-6": {
      "profile": "egress_lossy_profile"
    }
  }
}
bluecmd commented 6 months ago

FWIW, this is the queue stats on one of the more active ports on 202111:

       Port    TxQ    Counter/pkts    Counter/bytes    Drop/pkts    Drop/bytes
-----------  -----  --------------  ---------------  -----------  ------------
Ethernet124    UC0    137424676816   80666063346143        10191      14384228
Ethernet124    UC1               0                0            0             0
Ethernet124    UC2               0                0            0             0
Ethernet124    UC3               0                0            0             0
Ethernet124    UC4               0                0            0             0
Ethernet124    UC5               0                0            0             0
Ethernet124    UC6               0                0            0             0
Ethernet124    UC7         6927623        634638331            0             0
Ethernet124    UC8               0                0            0             0
Ethernet124    UC9               0                0            0             0
Ethernet124   MC10           14848         17854334            0             0
Ethernet124   MC11               0                0            0             0
Ethernet124   MC12               0                0            0             0
Ethernet124   MC13               0                0            0             0
Ethernet124   MC14               0                0            0             0
Ethernet124   MC15               0                0            0             0
Ethernet124   MC16               0                0            0             0
Ethernet124   MC17         1613345        150992262            0             0
Ethernet124   MC18               0                0            0             0
Ethernet124   MC19               0                0            0             0