search

sonic-net / sonic-buildimage

Scripts which perform an installable binary image build for SONiC
Other
730 stars 1.4k forks source link

Orchagent crash observed when we use "config qos reload" with a non-qos configuration #16186

Open rishabhhastu opened 1 year ago

rishabhhastu commented 1 year ago

Description

On a non-qos configuration, when the command "config qos reload" is entered, a one-time orchagent crash is seen.

Steps to reproduce the issue:

  1. Load a configuration that has no QoS specific configs.
  2. Enter the command "config qos reload" to generate QoS related configs
  3. The orchagent crash should be seen

Describe the results you received:

Jun 13 17:09:09.401095 nfc456-5 ERR swss#orchagent: :- doTask: Failed to process invalid buffer task
Jun 13 17:09:09.401121 nfc456-5 ERR swss#orchagent: :- processQueue: malformed key:nfc456-5:Ethernet8:5-6. Must contain 4 tokens
Jun 13 17:09:09.401121 nfc456-5 ERR swss#orchagent: :- doTask: Failed to process invalid buffer task
Jun 13 17:09:09.401121 nfc456-5 ERR swss#orchagent: :- processQueue: malformed key:nfc456-5:Ethernet92:0-2. Must contain 4 tokens
Jun 13 17:09:09.401147 nfc456-5 ERR swss#orchagent: :- doTask: Failed to process invalid buffer task
Jun 13 17:09:09.401147 nfc456-5 ERR swss#orchagent: :- processQueue: malformed key:nfc456-5:Ethernet92:3-4. Must contain 4 tokens
Jun 13 17:09:09.401155 nfc456-5 ERR swss#orchagent: :- doTask: Failed to process invalid buffer task
Jun 13 17:09:09.401173 nfc456-5 ERR swss#orchagent: :- processQueue: malformed key:nfc456-5:Ethernet92:5-6. Must contain 4 tokens
Jun 13 17:09:09.401173 nfc456-5 ERR swss#orchagent: :- doTask: Failed to process invalid buffer task
Jun 13 17:09:09.401182 nfc456-5 ERR swss#orchagent: :- processQueue: malformed key:nfc456-5:Ethernet96:0-2. Must contain 4 tokens
Jun 13 17:09:09.401201 nfc456-5 ERR swss#orchagent: :- doTask: Failed to process invalid buffer task
Jun 13 17:09:09.401201 nfc456-5 ERR swss#orchagent: :- processQueue: malformed key:nfc456-5:Ethernet96:3-4. Must contain 4 tokens
Jun 13 17:09:09.401210 nfc456-5 ERR swss#orchagent: :- doTask: Failed to process invalid buffer task
Jun 13 17:09:09.401230 nfc456-5 ERR swss#orchagent: :- processQueue: malformed key:nfc456-5:Ethernet96:5-6. Must contain 4 tokens
Jun 13 17:09:09.401230 nfc456-5 ERR swss#orchagent: :- doTask: Failed to process invalid buffer task
Jun 13 17:09:09.401239 nfc456-5 NOTICE swss#orchagent: :- processQueue: Set buffer queue nfc456-6:ASIC0:Ethernet0:0-2 to BUFFER_PROFILE_TABLE:egress_lossy_profile
Jun 13 17:09:09.425844 nfc456-5 INFO swss#supervisord: message repeated 5 times: [ intfmgrd RTNETLINK answers: File exists]
Jun 13 17:09:09.425844 nfc456-5 INFO swss#supervisord: orchagent terminate called after throwing an instance of 'std::invalid_argument'
Jun 13 17:09:09.425844 nfc456-5 INFO swss#supervisord: orchagent   what():  stoul
Jun 13 17:09:09.425999 nfc456-5 INFO swss#supervisord: orchagent
Jun 13 17:09:09.865948 nfc456-5 INFO swss#supervisord 2023-06-13 17:09:09,865 INFO exited: orchagent (terminated by SIGABRT (core dumped); not expected)
Jun 13 17:09:10.876542 nfc456-5 INFO swss#supervisor-proc-exit-listener: Process 'orchagent' exited unexpectedly. Terminating supervisor 'swss'
Jun 13 17:09:10.878809 nfc456-5 INFO swss#supervisord 2023-06-13 17:09:10,876 WARN received SIGTERM indicating exit request
Jun 13 17:09:10.878809 nfc456-5 INFO swss#supervisord 2023-06-13 17:09:10,877 INFO waiting for supervisor-proc-exit-listener, rsyslogd, portsyncd, coppmgrd, neighsyncd, vlanmgrd, intfmgrd, portmgrd, buffermgrd, vrfmgrd, nbrmgrd, vxlanmgrd, fdbsyncd, tunnelmgrd to die
Jun 13 17:09:10.878809 nfc456-5 INFO swss#supervisord 2023-06-13 17:09:10,877 INFO stopped: tunnelmgrd (terminated by SIGTERM)
Jun 13 17:09:10.878809 nfc456-5 INFO swss#supervisord 2023-06-13 17:09:10,878 INFO stopped: fdbsyncd (terminated by SIGTERM)
Jun 13 17:09:10.879990 nfc456-5 INFO swss#supervisord 2023-06-13 17:09:10,879 INFO stopped: vxlanmgrd (terminated by SIGTERM)
Jun 13 17:09:10.880067 nfc456-5 INFO swss#supervisord 2023-06-13 17:09:10,879 INFO stopped: nbrmgrd (terminated by SIGTERM)
Jun 13 17:09:10.880623 nfc456-5 INFO swss#supervisord 2023-06-13 17:09:10,880 INFO stopped: vrfmgrd (terminated by SIGTERM)
Jun 13 17:09:11.882803 nfc456-5 INFO swss#supervisord 2023-06-13 17:09:11,882 INFO stopped: buffermgrd (terminated by SIGTERM)
Jun 13 17:09:11.884290 nfc456-5 INFO swss#supervisord 2023-06-13 17:09:11,883 INFO stopped: portmgrd (terminated by SIGTERM)

Describe the results you expected:

Output of show version:

SONiC Software Version: SONiC.branch.202205-ars.7673690e-buildimage.origin.202205-nightly-2023.06.12.04.23
SONiC OS Version: 11
Distribution: Debian 11.7
Kernel: 5.10.0-18-2-amd64
Build commit: 59d6ab827
Build date: Mon Jun 12 07:29:30 UTC 2023
Built by: jenkins@jenkins-arsonic-k8s-7-wx97z

Platform: x86_64-arista_7800r3_48cqm2_lc
HwSKU: Arista-7800R3-48CQM2-C48
ASIC: broadcom
ASIC Count: 1
Serial Number: SSN20370089
Model Number: 7800R3-48CQM2-LC
Hardware Revision: 2a.00
Uptime: 17:31:05 up 22:29,  3 users,  load average: 1.85, 1.84, 1.82
Date: Tue 13 Jun 2023 17:31:05

Docker images:
REPOSITORY                 TAG                                                                            IMAGE ID       SIZE
docker-macsec              latest                                                                         328e005bb739   332MB
docker-dhcp-relay          latest                                                                         5f8fc3225827   323MB
docker-gbsyncd-broncos     branch.202205-ars.7673690e-buildimage.origin.202205-nightly-2023.06.12.04.23   67f3ac540c81   360MB
docker-gbsyncd-broncos     latest                                                                         67f3ac540c81   360MB
docker-gbsyncd-credo       branch.202205-ars.7673690e-buildimage.origin.202205-nightly-2023.06.12.04.23   e688a77b60ef   327MB
docker-gbsyncd-credo       latest                                                                         e688a77b60ef   327MB
docker-syncd-brcm-dnx      branch.202205-ars.7673690e-buildimage.origin.202205-nightly-2023.06.12.04.23   b2ba3fa3e5b4   659MB
docker-syncd-brcm-dnx      latest                                                                         b2ba3fa3e5b4   659MB
docker-sonic-telemetry     branch.202205-ars.7673690e-buildimage.origin.202205-nightly-2023.06.12.04.23   933464b39a05   394MB
docker-sonic-telemetry     latest                                                                         933464b39a05   394MB
docker-teamd               branch.202205-ars.7673690e-buildimage.origin.202205-nightly-2023.06.12.04.23   d121ed13e725   329MB
docker-teamd               latest                                                                         d121ed13e725   329MB
docker-snmp                branch.202205-ars.7673690e-buildimage.origin.202205-nightly-2023.06.12.04.23   d814363b393d   363MB
docker-snmp                latest                                                                         d814363b393d   363MB
docker-router-advertiser   branch.202205-ars.7673690e-buildimage.origin.202205-nightly-2023.06.12.04.23   5f08e689627d   313MB
docker-router-advertiser   latest                                                                         5f08e689627d   313MB
docker-platform-monitor    branch.202205-ars.7673690e-buildimage.origin.202205-nightly-2023.06.12.04.23   d476e3d1036e   438MB
docker-platform-monitor    latest                                                                         d476e3d1036e   438MB
docker-orchagent           branch.202205-ars.7673690e-buildimage.origin.202205-nightly-2023.06.12.04.23   c17ea2dce219   347MB
docker-orchagent           latest                                                                         c17ea2dce219   347MB
docker-mux                 branch.202205-ars.7673690e-buildimage.origin.202205-nightly-2023.06.12.04.23   2035a45c1060   362MB
docker-mux                 latest                                                                         2035a45c1060   362MB
docker-lldp                branch.202205-ars.7673690e-buildimage.origin.202205-nightly-2023.06.12.04.23   c35bdcbaa800   356MB
docker-lldp                latest                                                                         c35bdcbaa800   356MB
docker-fpm-frr             branch.202205-ars.7673690e-buildimage.origin.202205-nightly-2023.06.12.04.23   98f3b8c75030   359MB
docker-fpm-frr             latest                                                                         98f3b8c75030   359MB
docker-database            branch.202205-ars.7673690e-buildimage.origin.202205-nightly-2023.06.12.04.23   92797e40bb46   313MB
docker-database            latest                                                                         92797e40bb46   313MB
docker-sflow               branch.202205-ars.7673690e-buildimage.origin.202205-nightly-2023.06.12.04.23   86f5112b48e8   304MB
docker-sflow               latest                                                                         86f5112b48e8   304MB
docker-nat                 branch.202205-ars.7673690e-buildimage.origin.202205-nightly-2023.06.12.04.23   98a716da402f   306MB
docker-nat                 latest

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):

kenneth-arista commented 1 year ago

@arlakshm