search

sonic-net / sonic-buildimage

Scripts which perform an installable binary image build for SONiC
Other
717 stars 1.38k forks source link

[DNX]- sonic-clear macsec, does not clear the macsec counters since rekey causes anomaly in the counters. #19311

Open amitpawar12 opened 2 months ago

amitpawar12 commented 2 months ago

Description

Steps to reproduce the issue:

  1. Sent some packets over macsec.
  2. Check the macsec statistics - show macsec
  3. Issue - sonic-clear macsec command.
  4. Wait for sometime, and check the macsec statistics again with - show macsec.

Describe the results you received:

  1. Sent some packets over macsec.

  2. Check the macsec statistics:

MACsec Egress SC (18cXXXXe34b20001)
-----------  -
encoding_an  1
-----------  -
    MACsec Egress SA (1)
    -------------------------------------  ----------------------------------------------------------------

-------------- curtailed output ------------- ssci 1 SAI_MACSEC_SA_ATTR_CURRENT_XPN 879928291 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED 854011660550696 <<<<<<<<<< SAI_MACSEC_SA_STAT_OCTETS_PROTECTED 0 SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED 879928290 SAI_MACSEC_SA_STAT_OUT_PKTS_PROTECTED 0


3. Clear the macsec statistics:

sonic-clear counters; sonic-clear pfccounters; sonic-clear macsec; sudo ip netns exec asic0 sonic-clear queuecounters


4. Check the macsec stats again:

  IFACE    STATE    RX_OK      RX_BPS    RX_UTIL    RX_ERR    RX_DRP    RX_OVR    TX_OK      TX_BPS    TX_UTIL    TX_ERR    TX_DRP    TX_OVR

Ethernet32 U 64 197.76 B/s 0.00% 0 24 0 40 139.23 B/s 0.00% 0 0 0 Ethernet40 U 63 170.31 B/s 0.00% 0 24 0 39 110.57 B/s 0.00% 0 0 0

MACsec Egress SC (18cXXXXe34b20001)
-----------  -
encoding_an  1
-----------  -
    MACsec Egress SA (0)
    -------------------------------------  ----------------------------------------------------------------

-------------- curtailed output ------------- ssci 1 SAI_MACSEC_SA_ATTR_CURRENT_XPN 2 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED 488 SAI_MACSEC_SA_STAT_OCTETS_PROTECTED 0 SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED 2 SAI_MACSEC_SA_STAT_OUT_PKTS_PROTECTED 0

  IFACE    STATE    RX_OK      RX_BPS    RX_UTIL    RX_ERR    RX_DRP    RX_OVR    TX_OK      TX_BPS    TX_UTIL    TX_ERR    TX_DRP    TX_OVR

Ethernet32 U 92 125.24 B/s 0.00% 0 35 0 58 100.01 B/s 0.00% 0 0 0 Ethernet40 U 92 125.25 B/s 0.00% 0 35 0 58 99.93 B/s 0.00% 0 0 0

MACsec Egress SC (18cXXXXe34b20001)
-----------  -
encoding_an  1
-----------  -
    MACsec Egress SA (1)
    -------------------------------------  ----------------------------------------------------------------

-------------- curtailed output ------------- ssci 1 SAI_MACSEC_SA_ATTR_CURRENT_XPN 2 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED 854011660554600 <<<<<<<< SAI_MACSEC_SA_STAT_OCTETS_PROTECTED 0 SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED 1 SAI_MACSEC_SA_STAT_OUT_PKTS_PROTECTED 0


#### Describe the results you expected:

'sonic-clear macsec' should have cleared the octets encrypted. The same issue is also for ingress SA as well. Hence it becomes difficult to compare the number of packets encrypted during the test, as the counter just piles up pretty quickly.

#### Output of `show version`:

(paste your output here)

#### Output of `show techsupport`:

(paste your output here or download and attach the file here )



#### Additional information you deem important (e.g. issue happens only occasionally):

<!--
     Also attach debug file produced by `sudo generate_dump`
-->
judyjoseph commented 2 months ago

@amitpawar12 I checked on our lab testbed -- I see it cleans correctly.

admin@svcstr-xxxx-lc1-1:~$ show macsec 
MACsec port(Ethernet0)
---------------------  ---------------
cipher_suite           GCM-AES-XPN-256
enable                 false
enable_encrypt         true
enable_protect         true
enable_replay_protect  false
profile                macsec_profile
replay_window          0
send_sci               true
---------------------  ---------------
        MACsec Egress SC (185b00506bfe0001)
        -----------  -
        encoding_an  0
        -----------  -
MACsec port(Ethernet8)
---------------------  ---------------
cipher_suite           GCM-AES-XPN-256
enable                 true
enable_encrypt         true
enable_protect         true
enable_replay_protect  false
profile                macsec_profile
replay_window          0
send_sci               true
---------------------  ---------------
        MACsec Egress SC (185b00506bfe0001)
        -----------  -
        encoding_an  0
        -----------  -
                MACsec Egress SA (0)
                -------------------------------------  ----------------------------------------------------------------
                auth_key                               xx
                next_pn                                1
                sak                                    xx
                salt                                   xx
                ssci                                   2
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         6894
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    878831
                SAI_MACSEC_SA_STAT_OCTETS_PROTECTED    0
                SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  6893
                SAI_MACSEC_SA_STAT_OUT_PKTS_PROTECTED  0
                -------------------------------------  ----------------------------------------------------------------
        MACsec Ingress SC (ba7422dfc4370002)
                MACsec Ingress SA (0)
                ---------------------------------------  ----------------------------------------------------------------
                active                                   true
                auth_key                                 xxx
                lowest_acceptable_pn                     1
                sak                                      xxx
                salt                                     xxx
                ssci                                     1
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           7336
                SAI_MACSEC_SA_STAT_IN_PKTS_DELAYED       0
                SAI_MACSEC_SA_STAT_IN_PKTS_INVALID       0
                SAI_MACSEC_SA_STAT_IN_PKTS_LATE          0
                SAI_MACSEC_SA_STAT_IN_PKTS_NOT_USING_SA  0
                SAI_MACSEC_SA_STAT_IN_PKTS_NOT_VALID     0
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            7020
                SAI_MACSEC_SA_STAT_IN_PKTS_UNCHECKED     0
                SAI_MACSEC_SA_STAT_IN_PKTS_UNUSED_SA     0
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      611088
                SAI_MACSEC_SA_STAT_OCTETS_PROTECTED      0
                ---------------------------------------  ----------------------------------------------------------------

admin@svcstr-xxxx-lc1-1:~$ sonic-clear macsec 
Clear MACsec counters
admin@svcstr-xxxx-lc1-1:~$ show macsec 
Last cached time was 2024-06-18 15:52:56.439575
MACsec port(Ethernet0)
---------------------  ---------------
cipher_suite           GCM-AES-XPN-256
enable                 false
enable_encrypt         true
enable_protect         true
enable_replay_protect  false
profile                macsec_profile
replay_window          0
send_sci               true
---------------------  ---------------
        MACsec Egress SC (185b00506bfe0001)
        -----------  -
        encoding_an  0
        -----------  -
MACsec port(Ethernet8)
---------------------  ---------------
cipher_suite           GCM-AES-XPN-256
enable                 true
enable_encrypt         true
enable_protect         true
enable_replay_protect  false
profile                macsec_profile
replay_window          0
send_sci               true
---------------------  ---------------
        MACsec Egress SC (185b00506bfe0001)
        -----------  -
        encoding_an  0
        -----------  -
                MACsec Egress SA (0)
                -------------------------------------  ----------------------------------------------------------------
                auth_key                               xxx
                next_pn                                1
                sak                                    xxx
                salt                                   xxx
                ssci                                   2
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         0
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    0
                SAI_MACSEC_SA_STAT_OCTETS_PROTECTED    0
                SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  0
                SAI_MACSEC_SA_STAT_OUT_PKTS_PROTECTED  0
                -------------------------------------  ----------------------------------------------------------------
        MACsec Ingress SC (ba7422dfc4370002)
                MACsec Ingress SA (0)
                ---------------------------------------  ----------------------------------------------------------------
                active                                   true
                auth_key                                 xxx
                lowest_acceptable_pn                     1
                sak                                      xxx
                salt                                     xxx
                ssci                                     1
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           0
                SAI_MACSEC_SA_STAT_IN_PKTS_DELAYED       0
                SAI_MACSEC_SA_STAT_IN_PKTS_INVALID       0
                SAI_MACSEC_SA_STAT_IN_PKTS_LATE          0
                SAI_MACSEC_SA_STAT_IN_PKTS_NOT_USING_SA  0
                SAI_MACSEC_SA_STAT_IN_PKTS_NOT_VALID     0
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            0
                SAI_MACSEC_SA_STAT_IN_PKTS_UNCHECKED     0
                SAI_MACSEC_SA_STAT_IN_PKTS_UNUSED_SA     0
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      0
                SAI_MACSEC_SA_STAT_OCTETS_PROTECTED      0
                ---------------------------------------  ----------------------------------------------------------------
judyjoseph commented 2 months ago

Can you share the exact sequence you tried ? Also you have the traffic stopped when you clear ?

Another option with sonic-clear macsec --clean-cache true, which will clear the cache if there.

amitpawar12 commented 2 months ago

Sure @judyjoseph. Let me also try with '--clean-cache true' and get back to you.

Thanks, -A

judyjoseph commented 2 months ago

@amitpawar12 I found an issue with "sonic-clear macsec" when the rekey is enabled. I will raise a PR to fix this. Meanwhile as a work around disable rekey by setting rekey_interval = 0 in th emacsec profile, it should work fine.

judyjoseph commented 2 months ago

Working with Brcm, via CSP CS00012356026

judyjoseph commented 1 month ago

I checked on this again. So according to the current implementation, when we do a rekey -- the key (combination of "PORT:SA_ID:AN_bit" changes ) and the various counters v.z IN_PKTS_OK, OCTETS_ENCRYPTED etc gets reset.

jujoseph@STG01-0101-0200-01T2-lc01:/usr/local/lib/python3.9/d   jujoseph@STG01-0101-0200-01T2-lc01:/usr/local/lib/python3.9/d
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         164917 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         763872
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           9137 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           3987
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         644661 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         245380
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           4502 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           1205
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         191301 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         185796
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           1213 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           1020
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         835082 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         274623
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           4577 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           1313
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         62     |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         22
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           61   |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           23
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         62     |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         22
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           61   |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           23
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         63     |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         24
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           63   |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           25
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         23516  |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         44101
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           9056 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           2010
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         53245  |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         118070
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           3054 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           6712
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         72903  |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         150805
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           7795 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           1619
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         67696  |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         151932
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           3083 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN  0
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         108594 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           6757
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           5622 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         237686
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         910367 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           2062
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           4836 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         221143
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         681157 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           2115
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           4184 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         231087
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         706208 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           2078
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           5447 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         266794
                                                              >                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           2328
jujoseph@STG01-0101-0200-01T2-lc01:/usr/local/lib/python3.9/d   jujoseph@STG01-0101-0200-01T2-lc01:/usr/local/lib/python3.9/d
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            9187 |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            3955
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            4542 |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            1203
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            1272 |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            1017
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            4617 |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            1311
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            62   |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            21
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            62   |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            21
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            64   |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            23
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            9095 |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            2009
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            3065 |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            6709
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            7821 |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            1618
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            3097 |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            6754
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            5705 |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            1538
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            4919 |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            2112
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            4255 |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            2075
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            5538 |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            2326
jujoseph@netjb1-westus2:~$ diff -y /tmp/a /tmp/b | less
jujoseph@STG01-0101-0200-01T2-lc01:/usr/local/lib/python3.9/d   jujoseph@STG01-0101-0200-01T2-lc01:/usr/local/lib/python3.9/d
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         164917 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         763872
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           9137 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           3987
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         644661 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         245380
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           4502 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           1205
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         191301 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         185796
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           1213 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           1020
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         835082 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         274623
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           4577 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           1313
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         62     |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         22
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           61   |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           23
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         62     |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         22
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           61   |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           23
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         63     |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         24
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           63   |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           25
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         23516  |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         44101
jujoseph@STG01-0101-0200-01T2-lc01:/usr/local/lib/python3.9/d   jujoseph@STG01-0101-0200-01T2-lc01:/usr/local/lib/python3.9/d
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         164917 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         763872
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           9137 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           3987
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         644661 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         245380
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           4502 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           1205
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         191301 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         185796
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           1213 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           1020
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         835082 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         274623
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           4577 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           1313
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         62     |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         22
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           61   |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           23
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         62     |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         22
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           61   |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           23
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         63     |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         24
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           63   |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           25
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         23516  |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         44101
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           9056 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           2010
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         53245  |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         118070
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           3054 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           6712
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         72903  |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         150805
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           7795 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           1619
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         67696  |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         151932
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           3083 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN  0
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         108594 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           6757
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           5622 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         237686
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         910367 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           2062
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           4836 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         221143
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         681157 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           2115
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           4184 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         231087
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         706208 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           2078
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           5447 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         266794
                                                              >                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           2328
jujoseph@STG01-0101-0200-01T2-lc01:/usr/local/lib/python3.9/d   jujoseph@STG01-0101-0200-01T2-lc01:/usr/local/lib/python3.9/d
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            9187 |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            3955
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            4542 |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            1203
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            1272 |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            1017
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            4617 |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            1311
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            62   |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            21
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            62   |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            21
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            64   |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            23
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            9095 |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            2009
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            3065 |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            6709
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            7821 |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            1618
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            3097 |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            6754
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            5705 |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            1538
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            4919 |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            2112
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            4255 |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            2075
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            5538 |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            2326
jujoseph@STG01-0101-0200-01T2-lc01:/usr/local/lib/python3.9/d   jujoseph@STG01-0101-0200-01T2-lc01:/usr/local/lib/python3.9/d
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    475375 |                 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    475960
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      2333 |                 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      2335
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    538597 |                 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    539161
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      2953 |                 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      2955
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    537715 |                 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    538327
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      2711 |                 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      2713
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    571909 |                 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    572610
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      2577 |                 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      2580
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    222900 |                 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    222900
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      7817 |                 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      7817
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    228299 |                 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    228299
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      7173 |                 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      7173
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    217302 |                 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    217302
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      6411 |                 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      6411
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    475926 |                 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    502955
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      2323 |                 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      2451
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    121159 |                 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    127715
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      9177 |                 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      9704
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    186399 |                 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    196994
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      2083 |                 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      2201
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    154223 |                 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    505675
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      9151 |                 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      1533
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    504957 |                 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    521389
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      1530 |                 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      1569
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    520766 |                 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    516602
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      1566 |                 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      1557
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    516026 |                 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    520054
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      1554 |                 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      1559
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    519365 <
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      1555 <
jujoseph@STG01-0101-0200-01T2-lc01:/usr/local/lib/python3.9/d   jujoseph@STG01-0101-0200-01T2-lc01:/usr/local/lib/python3.9/d
                SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  168127 |                 SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  779025
                SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  671066 |                 SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  246498
                SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  232895 |                 SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  186872
                SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  878004 |                 SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  275811
                SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  63     |                 SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  21
                SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  63     |                 SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  22
                SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  64     |                 SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  24
                SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  23861  |                 SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  12
                SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  54061  |                 SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  0
                SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  73681  |                 SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  242
                SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  68659  |                 SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  99
                SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  113031 |                 SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  353214
                SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  947821 |                 SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  222271
                SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  718088 |                 SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  232494
                SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  741377 |                 SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  267858

Let me know your observations.

amitpawar12 commented 3 weeks ago

Hi @judyjoseph , @vmittal-msft - I still continue to see this issue. The reset happens for a while. But after 2-5 mins, the counters pop up again.

judyjoseph commented 1 week ago

@amitpawar12 can you check if the macsec session is getting rekeyed. This happens on rekey either from sonic/ixia end.

amitpawar12 commented 1 week ago

@judyjoseph - You are right. I configured the rekey interval to an hour to test this.

Log output:

1. Counters incremented:
admin@ixre-egl-board73:~$ show macsec
Last cached time was 2024-08-27 15:04:16.175319
MACsec port(Ethernet144)
---------------------  ---------------
cipher_suite           GCM-AES-XPN-256
enable                 true
enable_encrypt         true
enable_protect         true
enable_replay_protect  false
profile                256_XPN_SCI
replay_window          0
send_sci               true
---------------------  ---------------
    MACsec Egress SC (xxxxxx)
    -----------  -
    encoding_an  0
    -----------  -
        MACsec Egress SA (0)
        -------------------------------------  ----------------------------------------------------------------
        auth_key                               xxxx
        next_pn                                1
        sak                                    xxxxx
        salt                                   xxx
        ssci                                   xxx
        SAI_MACSEC_SA_ATTR_CURRENT_XPN         12614100869
        SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    12311362440103
        SAI_MACSEC_SA_STAT_OCTETS_PROTECTED    0
        SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  12614100869
        SAI_MACSEC_SA_STAT_OUT_PKTS_PROTECTED  0
        -------------------------------------  ----------------------------------------------------------------

2. Cleared the counter:
admin@ixre-egl-board73:~$ sonic-clear macsec
Clear MACsec counters

3. Waited for some time. Did not send any traffic and checked if the counters are getting back to old values. 

admin@ixre-egl-board73:~$ show macsec
Last cached time was 2024-08-27 15:10:43.673238
MACsec port(Ethernet144)
---------------------  ---------------
cipher_suite           GCM-AES-XPN-256
enable                 true
enable_encrypt         true
enable_protect         true
enable_replay_protect  false
profile                256_XPN_SCI
replay_window          0
send_sci               true
---------------------  ---------------
    MACsec Egress SC (xxxxxx)
    -----------  -
    encoding_an  0
    -----------  -
        MACsec Egress SA (0)
        -------------------------------------  ----------------------------------------------------------------
        auth_key                               xxxxxxx
        next_pn                                1
        sak                                    xxxxxxx
        salt                                   xxxxxxxxx
        ssci                                   xx
        SAI_MACSEC_SA_ATTR_CURRENT_XPN         23
        SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    5635
        SAI_MACSEC_SA_STAT_OCTETS_PROTECTED    0
        SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  23
        SAI_MACSEC_SA_STAT_OUT_PKTS_PROTECTED  0
        -------------------------------------  ----------------------------------------------------------------

4. The counters are not getting incremented as was observed earlier.  Rekeying is the cause which might be triggering the old stats to come back or increment along with existing values.
abdosi commented 1 week ago

Will not be fix for 202205. FIx wil targetted for 202405.

amitpawar12 commented 1 week ago

@judyjoseph - as a data point, we ran a controlled test with fixed packets.

What we observed is that before the rekey, the values are correctly updated but on rekey, they just go to some junk value.

Snapshot:

Before rekeying:
    MACsec Ingress SC (XXXX0001)
        MACsec Ingress SA (1)
        ---------------------------------------  ----------------------------------------------------------------
        active                                   true
        auth_key                                 XXXX
        lowest_acceptable_pn                     1
        sak                                      XXXX
        salt                                     XXXX
        ssci                                     2
        SAI_MACSEC_SA_ATTR_CURRENT_XPN           44800004
        SAI_MACSEC_SA_STAT_IN_PKTS_DELAYED       0
        SAI_MACSEC_SA_STAT_IN_PKTS_INVALID       0
        SAI_MACSEC_SA_STAT_IN_PKTS_LATE          0
        SAI_MACSEC_SA_STAT_IN_PKTS_NOT_USING_SA  0
        SAI_MACSEC_SA_STAT_IN_PKTS_NOT_VALID     0
        SAI_MACSEC_SA_STAT_IN_PKTS_OK            44800006
        SAI_MACSEC_SA_STAT_IN_PKTS_UNCHECKED     0
        SAI_MACSEC_SA_STAT_IN_PKTS_UNUSED_SA     0
        SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      45158401068
        SAI_MACSEC_SA_STAT_OCTETS_PROTECTED      0

After rekeying:
    MACsec Ingress SC (XXXX0001)
        MACsec Ingress SA (0)
        ---------------------------------------  ----------------------------------------------------------------
        active                                   true
        auth_key                                 XXXX
        lowest_acceptable_pn                     1
        sak                                      XXXX
        salt                                     XXXX
        ssci                                     2
        SAI_MACSEC_SA_ATTR_CURRENT_XPN           3
        SAI_MACSEC_SA_STAT_IN_PKTS_DELAYED       0
        SAI_MACSEC_SA_STAT_IN_PKTS_INVALID       0
        SAI_MACSEC_SA_STAT_IN_PKTS_LATE          0
        SAI_MACSEC_SA_STAT_IN_PKTS_NOT_USING_SA  0
        SAI_MACSEC_SA_STAT_IN_PKTS_NOT_VALID     0
        SAI_MACSEC_SA_STAT_IN_PKTS_OK            2
        SAI_MACSEC_SA_STAT_IN_PKTS_UNCHECKED     0
        SAI_MACSEC_SA_STAT_IN_PKTS_UNUSED_SA     0
        SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      95693495115754
        SAI_MACSEC_SA_STAT_OCTETS_PROTECTED      0

Every time, after rekeying, we see different counter although there is no traffic flowing through the system:

    MACsec Ingress SC (XXXX0001)
        MACsec Ingress SA (1)
        ---------------------------------------  ----------------------------------------------------------------
        active                                   true
        auth_key                                 XXXX
        lowest_acceptable_pn                     1
        sak                                      XXXX
        salt                                     XXXX
        ssci                                     2
        SAI_MACSEC_SA_ATTR_CURRENT_XPN           3
        SAI_MACSEC_SA_STAT_IN_PKTS_DELAYED       0
        SAI_MACSEC_SA_STAT_IN_PKTS_INVALID       0
        SAI_MACSEC_SA_STAT_IN_PKTS_LATE          0
        SAI_MACSEC_SA_STAT_IN_PKTS_NOT_USING_SA  0
        SAI_MACSEC_SA_STAT_IN_PKTS_NOT_VALID     0
        SAI_MACSEC_SA_STAT_IN_PKTS_OK            2
        SAI_MACSEC_SA_STAT_IN_PKTS_UNCHECKED     0
        SAI_MACSEC_SA_STAT_IN_PKTS_UNUSED_SA     0
        SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      95693495118958
        SAI_MACSEC_SA_STAT_OCTETS_PROTECTED      0
        ---------------------------------------  ----------------------------------------------------------------