search

sonic-net / sonic-buildimage

Scripts which perform an installable binary image build for SONiC
Other
730 stars 1.4k forks source link

orchagent crashed after applying "VLAN_SUB_INTERFACE" config #6305

Open bingwang-ms opened 3 years ago

bingwang-ms commented 3 years ago

Description I did some investigation on the failure of sub_port_interfaces.test_sub_port_interfaces and found that applying following config file will cause orchagent crash.

{
    "VLAN_SUB_INTERFACE": {

        "Ethernet2.20": {
            "admin_status" : "up"
        },
        "Ethernet2.20|172.16.0.5/30": {},

        "Ethernet2.10": {
            "admin_status" : "up"
        },
        "Ethernet2.10|172.16.0.1/30": {}
    }
}

The error logs are

Dec 28 05:50:23.784217 str-7260cx3-acs-2 NOTICE swss#portsyncd: :- onMsg: nlmsg type:16 key:Ethernet2 admin:1 oper:1 addr:74:83:ef:09:33:1c ifindex:1623 master:1582
Dec 28 05:50:23.785358 str-7260cx3-acs-2 INFO systemd-udevd[24683]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Dec 28 05:50:23.785705 str-7260cx3-acs-2 INFO systemd-udevd[24683]: Using default interface naming scheme 'v240'.
Dec 28 05:50:23.786247 str-7260cx3-acs-2 NOTICE swss#portsyncd: :- onMsg: nlmsg type:16 key:Ethernet2.10 admin:0 oper:0 addr:74:83:ef:09:33:1c ifindex:1708 master:0 type:vlan
Dec 28 05:50:23.786559 str-7260cx3-acs-2 NOTICE swss#portsyncd: :- onMsg: Cannot find Ethernet2.10 in port table
Dec 28 05:50:23.792103 str-7260cx3-acs-2 NOTICE swss#portsyncd: :- onMsg: nlmsg type:16 key:Ethernet2.10 admin:1 oper:1 addr:74:83:ef:09:33:1c ifindex:1708 master:0 type:vlan
Dec 28 05:50:23.792103 str-7260cx3-acs-2 NOTICE swss#portsyncd: :- onMsg: Cannot find Ethernet2.10 in port table
Dec 28 05:50:23.802278 str-7260cx3-acs-2 NOTICE swss#orchagent: :- addSubPort: Sub interface Ethernet2.10 inherits mtu size 9100 from parent port Ethernet2
Dec 28 05:50:23.803205 str-7260cx3-acs-2 ERR syncd#syncd: [0] SAI_API_ROUTER_INTERFACE:brcm_sai_create_router_interface:319 Error processing rtr intf attribute failed with error -196604.
Dec 28 05:50:23.803205 str-7260cx3-acs-2 WARNING syncd#syncd: :- sai_serialize_enum: enum value -196604 not found in enum sai_status_t
Dec 28 05:50:23.803205 str-7260cx3-acs-2 ERR syncd#syncd: :- sendApiResponse: api SAI_COMMON_API_CREATE failed in syncd mode: -196604
Dec 28 05:50:23.803205 str-7260cx3-acs-2 WARNING syncd#syncd: :- sai_serialize_enum: enum value -196604 not found in enum sai_status_t
Dec 28 05:50:23.803614 str-7260cx3-acs-2 ERR syncd#syncd: :- processQuadEvent: attr: SAI_ROUTER_INTERFACE_ATTR_VIRTUAL_ROUTER_ID: oid:0x300000000007c
Dec 28 05:50:23.803614 str-7260cx3-acs-2 ERR syncd#syncd: :- processQuadEvent: attr: SAI_ROUTER_INTERFACE_ATTR_SRC_MAC_ADDRESS: 74:83:EF:09:33:1C
Dec 28 05:50:23.803614 str-7260cx3-acs-2 ERR syncd#syncd: :- processQuadEvent: attr: SAI_ROUTER_INTERFACE_ATTR_TYPE: SAI_ROUTER_INTERFACE_TYPE_SUB_PORT
Dec 28 05:50:23.803614 str-7260cx3-acs-2 ERR syncd#syncd: :- processQuadEvent: attr: SAI_ROUTER_INTERFACE_ATTR_PORT_ID: oid:0x1000000000021
Dec 28 05:50:23.803614 str-7260cx3-acs-2 ERR syncd#syncd: :- processQuadEvent: attr: SAI_ROUTER_INTERFACE_ATTR_OUTER_VLAN_ID: 10
Dec 28 05:50:23.803614 str-7260cx3-acs-2 ERR syncd#syncd: :- processQuadEvent: attr: SAI_ROUTER_INTERFACE_ATTR_ADMIN_V4_STATE: true
Dec 28 05:50:23.803614 str-7260cx3-acs-2 ERR syncd#syncd: :- processQuadEvent: attr: SAI_ROUTER_INTERFACE_ATTR_ADMIN_V6_STATE: true
Dec 28 05:50:23.803614 str-7260cx3-acs-2 ERR syncd#syncd: :- processQuadEvent: attr: SAI_ROUTER_INTERFACE_ATTR_MTU: 9100
Dec 28 05:50:23.803614 str-7260cx3-acs-2 ERR syncd#syncd: :- processQuadEvent: attr: SAI_ROUTER_INTERFACE_ATTR_NAT_ZONE_ID: 0
Dec 28 05:50:23.804460 str-7260cx3-acs-2 WARNING swss#orchagent: :- sai_deserialize_enum: enum -196604 not found in enum sai_status_t
Dec 28 05:50:23.804460 str-7260cx3-acs-2 WARNING swss#orchagent: :- sai_serialize_enum: enum value -196604 not found in enum sai_status_t
Dec 28 05:50:23.804460 str-7260cx3-acs-2 ERR swss#orchagent: :- create: create status: -196604
Dec 28 05:50:23.804460 str-7260cx3-acs-2 ERR swss#orchagent: :- addRouterIntfs: Failed to create router interface Ethernet2.10, rv:-196604
Dec 28 05:50:23.805373 str-7260cx3-acs-2 INFO swss#/supervisord: orchagent terminate called after throwing an instance of 'std::runtime_error'
Dec 28 05:50:23.805373 str-7260cx3-acs-2 INFO swss#/supervisord: orchagent   what():  Failed to create router interface.

syslog.tar.gz

Steps to reproduce the issue:

  1. Run sub_port_interfaces.test_sub_port_interfaces

Describe the results you received:

Describe the results you expected:

Additional information you deem important (e.g. issue happens only occasionally):

**Output of `show version`:**

SONiC Software Version: SONiC.HEAD.175-df13245b
Distribution: Debian 10.7
Kernel: 4.19.0-9-2-amd64
Build commit: df13245b
Build date: Thu Dec 24 02:01:34 UTC 2020
Built by: johnar@jenkins-worker-22

Platform: x86_64-arista_7260cx3_64
HwSKU: Arista-7260CX3-D108C8
ASIC: broadcom
ASIC Count: 1
Serial Number: SSJ17432414
Uptime: 06:03:57 up  5:53,  2 users,  load average: 1.44, 1.48, 2.06

Docker images:
REPOSITORY                    TAG                 IMAGE ID            SIZE
docker-syncd-brcm             HEAD.175-df13245b   105614c2a0ad        725MB
docker-syncd-brcm             latest              105614c2a0ad        725MB
docker-snmp                   HEAD.175-df13245b   0d41b39eca93        525MB
docker-snmp                   latest              0d41b39eca93        525MB
docker-teamd                  HEAD.175-df13245b   736b66a0ba03        523MB
docker-teamd                  latest              736b66a0ba03        523MB
docker-nat                    HEAD.175-df13245b   8c658d90c903        525MB
docker-nat                    latest              8c658d90c903        525MB
docker-router-advertiser      HEAD.175-df13245b   fb202bd05216        479MB
docker-router-advertiser      latest              fb202bd05216        479MB
docker-platform-monitor       HEAD.175-df13245b   c99888eb05b9        604MB
docker-platform-monitor       latest              c99888eb05b9        604MB
docker-lldp                   HEAD.175-df13245b   8226b60e5c45        519MB
docker-lldp                   latest              8226b60e5c45        519MB
docker-dhcp-relay             HEAD.175-df13245b   72693024f19a        486MB
docker-dhcp-relay             latest              72693024f19a        486MB
docker-database               HEAD.175-df13245b   79ba588de175        479MB
docker-database               latest              79ba588de175        479MB
docker-orchagent              HEAD.175-df13245b   b254776a8e87        553MB
docker-orchagent              latest              b254776a8e87        553MB
docker-sonic-telemetry        HEAD.175-df13245b   3b336806b7c4        553MB
docker-sonic-telemetry        latest              3b336806b7c4        553MB
docker-sonic-mgmt-framework   HEAD.175-df13245b   44bdcc06871d        680MB
docker-sonic-mgmt-framework   latest              44bdcc06871d        680MB
docker-fpm-frr                HEAD.175-df13245b   4d77e5f907de        539MB
docker-fpm-frr                latest              4d77e5f907de        539MB
docker-sflow                  HEAD.175-df13245b   d99fda4d2916        523MB
docker-sflow                  latest              d99fda4d2916        523MB
**Attach debug file `sudo generate_dump`:**

```
(paste your output here)
```
JafarSeyedi commented 3 years ago

sub-interface still not working in BRCM SAI 5.0.0.6

Aug 3 11:56:37.966317 sonic NOTICE swss#orchagent: :- addSubPort: Sub interface Ethernet0.2 inherits mtu size 9100 from parent port Ethernet0 Aug 3 11:56:37.973345 sonic NOTICE swss#orchagent: :- setHostIntfsStripTag: Set SAI_HOSTIF_VLAN_TAG_KEEP to host interface: Ethernet0 Aug 3 11:56:38.056975 sonic INFO syncd#syncd: [none] SAI_API_ROUTER_INTERFACE:_brcm_sai_sub_router_intf_l2_config:5437 Creating vlan Aug 3 11:56:38.057406 sonic ERR syncd#syncd: [none] SAI_API_VLAN:_brcm_sai_vlan_create_internal_vfi:4083 MC-GRP create failed with error Feature unavailable (0xfffffff0). Aug 3 11:56:38.057651 sonic ERR syncd#syncd: [none] SAI_API_ROUTER_INTERFACE:_brcm_sai_sub_router_intf_l2_config:5475 internal vfi create failed with error -2. Aug 3 11:56:38.057651 sonic ERR syncd#syncd: [none] SAI_API_ROUTER_INTERFACE:brcm_sai_create_router_interface:587 sub port rif l2 config failed with error -2. Aug 3 11:56:38.057877 sonic ERR syncd#syncd: :- sendApiResponse: api SAI_COMMON_API_CREATE failed in syncd mode: SAI_STATUS_NOT_SUPPORTED Aug 3 11:56:38.058468 sonic ERR syncd#syncd: :- processQuadEvent: attr: SAI_ROUTER_INTERFACE_ATTR_VIRTUAL_ROUTER_ID: oid:0x3000000000022 Aug 3 11:56:38.058688 sonic ERR syncd#syncd: :- processQuadEvent: attr: SAI_ROUTER_INTERFACE_ATTR_SRC_MAC_ADDRESS: 6C:EC:5A:0A:52:CF Aug 3 11:56:38.059184 sonic ERR syncd#syncd: :- processQuadEvent: attr: SAI_ROUTER_INTERFACE_ATTR_TYPE: SAI_ROUTER_INTERFACE_TYPE_SUB_PORT Aug 3 11:56:38.059383 sonic ERR swss#orchagent: :- create: create status: SAI_STATUS_NOT_SUPPORTED Aug 3 11:56:38.059508 sonic ERR swss#orchagent: :- addRouterIntfs: Failed to create router interface Ethernet0.2, rv:-2 Aug 3 11:56:38.059637 sonic ERR swss#orchagent: :- handleSaiCreateStatus: Encountered failure in create operation, exiting orchagent, SAI API: SAI_API_ROUTER_INTERFACE, status: SAI_STATUS_NOT_SUPPORTED Aug 3 11:56:38.060022 sonic ERR syncd#syncd: :- processQuadEvent: attr: SAI_ROUTER_INTERFACE_ATTR_PORT_ID: oid:0x1000000000002 Aug 3 11:56:38.060281 sonic ERR syncd#syncd: :- processQuadEvent: attr: SAI_ROUTER_INTERFACE_ATTR_OUTER_VLAN_ID: 2

stenstad commented 1 year ago

Still happens on Tomahawk with:

BRCM SAI ver: [7.1.111.1], OCP SAI ver: [1.11.0], SDK ver: [sdk-6.5.24]

rlebedys commented 7 months ago

This is still the case in the latest builds. Added more info in https://github.com/sonic-net/sonic-buildimage/issues/6167#issuecomment-1952525022