search

sonic-net / sonic-buildimage

Scripts which perform an installable binary image build for SONiC
Other
743 stars 1.44k forks source link

[Nokia-7215] swss container crash after removing CUSTOM-TYPE ACL table #20738

Open lizhijianrd opened 3 weeks ago

lizhijianrd commented 3 weeks ago

Description

On Nokia-7215 DUT, I created 3 dataplane ACL table of custom-type, then I remove the ACL tables and swss container crash. This issue can consistently repro on latest 202405 marvell-armhf image. This issue seems related to IP_PROTOCOL in IPv6 custom ACL table type. By following below repro steps, if I remove IP_PROTOCOL from BMCDATAV6 definition, then I can't repro same issue.

Steps to reproduce the issue:

I'm using the general Mx configuration setup. Below is some config snapshot before test:

admin@bjw2-can-7215-2:~$ show vlan br
+-----------+-----------------+------------+----------------+-------------+-----------------------+
|   VLAN ID | IP Address      | Ports      | Port Tagging   | Proxy ARP   | DHCP Helper Address   |
+===========+=================+============+================+=============+=======================+
|      1000 | 192.168.0.1/24  | Ethernet0  | untagged       | disabled    | 192.0.0.1             |
|           | fc02:1000::1/64 | Ethernet1  | untagged       |             | 192.0.0.2             |
|           |                 | Ethernet2  | untagged       |             | 192.0.0.3             |
|           |                 | Ethernet3  | untagged       |             | 192.0.0.4             |
|           |                 | Ethernet4  | untagged       |             | 192.0.0.5             |
|           |                 | Ethernet5  | untagged       |             | 192.0.0.6             |
|           |                 | Ethernet6  | untagged       |             | 192.0.0.7             |
|           |                 | Ethernet7  | untagged       |             | 192.0.0.8             |
|           |                 | Ethernet8  | untagged       |             | 192.0.0.9             |
|           |                 | Ethernet9  | untagged       |             | 192.0.0.10            |
|           |                 | Ethernet10 | untagged       |             | 192.0.0.11            |
|           |                 | Ethernet11 | untagged       |             | 192.0.0.12            |
|           |                 | Ethernet12 | untagged       |             | 192.0.0.13            |
|           |                 | Ethernet13 | untagged       |             | 192.0.0.14            |
|           |                 | Ethernet14 | untagged       |             | 192.0.0.15            |
|           |                 | Ethernet15 | untagged       |             | 192.0.0.16            |
|           |                 | Ethernet16 | untagged       |             | 192.0.0.17            |
|           |                 | Ethernet17 | untagged       |             | 192.0.0.18            |
|           |                 | Ethernet18 | untagged       |             | 192.0.0.19            |
|           |                 | Ethernet19 | untagged       |             | 192.0.0.20            |
|           |                 | Ethernet20 | untagged       |             | 192.0.0.21            |
|           |                 | Ethernet21 | untagged       |             | 192.0.0.22            |
|           |                 | Ethernet22 | untagged       |             | 192.0.0.23            |
|           |                 | Ethernet23 | untagged       |             | 192.0.0.24            |
|           |                 | Ethernet24 | untagged       |             | 192.0.0.25            |
|           |                 | Ethernet25 | untagged       |             | 192.0.0.26            |
|           |                 | Ethernet26 | untagged       |             | 192.0.0.27            |
|           |                 | Ethernet27 | untagged       |             | 192.0.0.28            |
|           |                 | Ethernet28 | untagged       |             | 192.0.0.29            |
|           |                 | Ethernet29 | untagged       |             | 192.0.0.30            |
|           |                 | Ethernet30 | untagged       |             | 192.0.0.31            |
|           |                 | Ethernet31 | untagged       |             | 192.0.0.32            |
|           |                 | Ethernet32 | untagged       |             | 192.0.0.33            |
|           |                 | Ethernet33 | untagged       |             | 192.0.0.34            |
|           |                 | Ethernet34 | untagged       |             | 192.0.0.35            |
|           |                 | Ethernet35 | untagged       |             | 192.0.0.36            |
|           |                 | Ethernet36 | untagged       |             | 192.0.0.37            |
|           |                 | Ethernet37 | untagged       |             | 192.0.0.38            |
|           |                 | Ethernet38 | untagged       |             | 192.0.0.39            |
|           |                 | Ethernet39 | untagged       |             | 192.0.0.40            |
|           |                 | Ethernet40 | untagged       |             | 192.0.0.41            |
|           |                 | Ethernet41 | untagged       |             | 192.0.0.42            |
|           |                 | Ethernet42 | untagged       |             | 192.0.0.43            |
|           |                 | Ethernet43 | untagged       |             | 192.0.0.44            |
|           |                 | Ethernet44 | untagged       |             | 192.0.0.45            |
|           |                 | Ethernet45 | untagged       |             | 192.0.0.46            |
|           |                 |            |                |             | 192.0.0.47            |
|           |                 |            |                |             | 192.0.0.48            |
+-----------+-----------------+------------+----------------+-------------+-----------------------+
admin@bjw2-can-7215-2:~$ show acl table
Name        Type       Binding     Description    Stage    Status
----------  ---------  ----------  -------------  -------  --------
DATAACL     L3         Ethernet46  DATAACL        ingress  Active
                       Ethernet47
EVERFLOW    MIRROR     Ethernet0   EVERFLOW       ingress  Active
                       Ethernet1
                       Ethernet2
                       Ethernet3
                       Ethernet4
                       Ethernet5
                       Ethernet6
                       Ethernet7
                       Ethernet8
                       Ethernet9
                       Ethernet10
                       Ethernet11
                       Ethernet12
                       Ethernet13
                       Ethernet14
                       Ethernet15
                       Ethernet16
                       Ethernet17
                       Ethernet18
                       Ethernet19
                       Ethernet20
                       Ethernet21
                       Ethernet22
                       Ethernet23
                       Ethernet24
                       Ethernet25
                       Ethernet26
                       Ethernet27
                       Ethernet28
                       Ethernet29
                       Ethernet30
                       Ethernet31
                       Ethernet32
                       Ethernet33
                       Ethernet34
                       Ethernet35
                       Ethernet36
                       Ethernet37
                       Ethernet38
                       Ethernet39
                       Ethernet40
                       Ethernet41
                       Ethernet42
                       Ethernet43
                       Ethernet44
                       Ethernet45
                       Ethernet46
                       Ethernet47
EVERFLOWV6  MIRRORV6   Ethernet0   EVERFLOWV6     ingress  Active
                       Ethernet1
                       Ethernet2
                       Ethernet3
                       Ethernet4
                       Ethernet5
                       Ethernet6
                       Ethernet7
                       Ethernet8
                       Ethernet9
                       Ethernet10
                       Ethernet11
                       Ethernet12
                       Ethernet13
                       Ethernet14
                       Ethernet15
                       Ethernet16
                       Ethernet17
                       Ethernet18
                       Ethernet19
                       Ethernet20
                       Ethernet21
                       Ethernet22
                       Ethernet23
                       Ethernet24
                       Ethernet25
                       Ethernet26
                       Ethernet27
                       Ethernet28
                       Ethernet29
                       Ethernet30
                       Ethernet31
                       Ethernet32
                       Ethernet33
                       Ethernet34
                       Ethernet35
                       Ethernet36
                       Ethernet37
                       Ethernet38
                       Ethernet39
                       Ethernet40
                       Ethernet41
                       Ethernet42
                       Ethernet43
                       Ethernet44
                       Ethernet45
                       Ethernet46
                       Ethernet47
NTP_ACL     CTRLPLANE  NTP         NTP_ACL        ingress  Active
SNMP_ACL    CTRLPLANE  SNMP        SNMP_ACL       ingress  Active
SSH_ONLY    CTRLPLANE  SSH         SSH_ONLY       ingress  Active
  1. Setup custom ACL table type in running-config:
    1. Write below content to ~/acl_table_type.json: 
      {
"ACL_TABLE_TYPE": {
    "BMCDATA": {
        "MATCHES": ["SRC_IP", "DST_IP", "ETHER_TYPE", "IP_TYPE", "IP_PROTOCOL", "IN_PORTS", "L4_SRC_PORT", "L4_DST_PORT", "L4_SRC_PORT_RANGE", "L4_DST_PORT_RANGE"],
        "ACTIONS": ["PACKET_ACTION", "COUNTER"],
        "BIND_POINTS": ["PORT"]
    },
    "BMCDATAV6": {
        "MATCHES": ["SRC_IPV6", "DST_IPV6", "ETHER_TYPE", "IP_TYPE", "IP_PROTOCOL", "IN_PORTS", "L4_SRC_PORT", "L4_DST_PORT", "L4_SRC_PORT_RANGE", "L4_DST_PORT_RANGE", "ICMPV6_TYPE", "ICMPV6_CODE", "TCP_FLAGS"],
        "ACTIONS": ["PACKET_ACTION", "COUNTER"],
        "BIND_POINTS": ["PORT"]
    }
}
}
    2. Write the custom ACL table type to running-config: sonic-cfggen -j acl_table_type.json -w
  2. Issue below command to add 3 ACL tables on DUT: 
    admin@sonic:~$ sudo config acl add table -p Vlan1000 -s ingress BMC_ACL_NORTHBOUND BMCDATA
admin@sonic:~$ sudo config acl add table -p Vlan1000 -s ingress BMC_ACL_NORTHBOUND_V6 BMCDATAV6
admin@sonic:~$ sudo config acl add table -p Ethernet46,Ethernet47 -s ingress BMC_ACL_SOUTHBOUND_V6 BMCDATAV6
  3. Issue below command to remove 3 ACL tables from DUT: 
    admin@sonic:~$ sudo config acl remove table BMC_ACL_NORTHBOUND && sudo config acl remove table BMC_ACL_NORTHBOUND_V6 && sudo config acl remove table BMC_ACL_SOUTHBOUND_V6
  4. Check the docker status and see swss container exited.
admin@sonic:~$ docker ps -a
CONTAINER ID   IMAGE                                COMMAND                  CREATED             STATUS                     PORTS     NAMES
b5d772f4e32d   docker-snmp:latest                   "/usr/local/bin/supe…"   About an hour ago   Up 56 seconds                        snmp
b8408d7e5721   docker-platform-monitor:latest       "/usr/bin/docker_ini…"   About an hour ago   Up About a minute                    pmon
4293b795241a   docker-sonic-mgmt-framework:latest   "/usr/local/bin/supe…"   About an hour ago   Up About a minute                    mgmt-framework
b77d0e27815d   docker-lldp:latest                   "/usr/bin/docker-lld…"   About an hour ago   Up About a minute                    lldp
4352981b228f   docker-sonic-gnmi:latest             "/usr/local/bin/supe…"   About an hour ago   Up 2 minutes                         gnmi
d19f2cb57702   66324d682340                         "/usr/bin/docker_ini…"   2 hours ago         Up 3 minutes                         dhcp_relay
b4aad98de111   docker-router-advertiser:latest      "/usr/bin/docker-ini…"   2 hours ago         Up 3 minutes                         radv
4c3b1f0e47d6   docker-fpm-frr:latest                "/usr/bin/docker_ini…"   2 hours ago         Up 3 minutes                         bgp
2ff571668fa9   docker-syncd-mrvl:latest             "/usr/local/bin/supe…"   2 hours ago         Up 4 minutes                         syncd
db4e827c80e1   docker-teamd:latest                  "/usr/local/bin/supe…"   2 hours ago         Up 4 minutes                         teamd
767d70121070   docker-orchagent:latest              "/usr/bin/docker-ini…"   2 hours ago         Exited (0) 3 seconds ago             swss
4fb56fa6bc67   docker-eventd:latest                 "/usr/local/bin/supe…"   2 hours ago         Up 4 minutes                         eventd
d72a15f7ab18   docker-database:latest               "/usr/local/bin/dock…"   2 hours ago         Up 50 minutes                        database

Describe the results you received:

I see swss container crash.

Describe the results you expected:

Expect swss running stable.

Output of show version:

SONiC Software Version: SONiC.202405.689073-cf8484700
SONiC OS Version: 12
Distribution: Debian 12.6
Kernel: 6.1.0-22-2-armmp
Build commit: cf8484700
Build date: Thu Nov  7 11:18:51 UTC 2024
Built by: azureuser@3b2e9d8cc000000

Platform: armhf-nokia_ixs7215_52x-r0
HwSKU: Nokia-7215
ASIC: marvell
ASIC Count: 1

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):

prgeor commented 1 week ago

@lizhijianrd looks to be issue with Marvell SAI, can you check with Pavan from Nokia

mlok-nokia commented 1 week ago

@Pavan-Nokia Hi Pavan, please take a look this issue

Pavan-Nokia commented 1 week ago

This issue has been fixed by below 2 Pr's (SAI + EZB file update)

https://github.com/sonic-net/sonic-buildimage/pull/20770 -- Already Merged https://github.com/sonic-net/sonic-buildimage/pull/20838 -- Needs merge and backport to 202405