search

sonic-net / sonic-buildimage

Scripts which perform an installable binary image build for SONiC
Other
734 stars 1.41k forks source link

When device/default_sku is set to l2, lldpmgrd will exit abnormally. #17700

Open philo-micas opened 9 months ago

philo-micas commented 9 months ago

Description

When we change the default_sku configuration in the Device product from t1 to l2, for example：

ssp@admin:~/sonic-buildimage/device/micas/x86_64-micas_m2-w6510-48v8c-r0$ cat default_sku
M2-W6510-48V8C l2

When installing the Sonic system under ONIE, lldpmgrd will exit.

admin@sonic:~$ sudo show system-health detail
System status summary

  System status LED  green
  Services:
    Status: Not OK
    Not Running: lldp:lldpmgrd
  Hardware:
    Status: OK

System services and devices monitor list

Name                   Status    Type
---------------------  --------  ----------
lldp:lldpmgrd          Not OK    Process
sonic                  OK        System
rsyslog                OK        Process
root-overlay           OK        Filesystem
var-log                OK        Filesystem
routeCheck             OK        Program
dualtorNeighborCheck   OK        Program
diskCheck              OK        Program
container_checker      OK        Program
vnetRouteCheck         OK        Program
memory_check           OK        Program
container_memory_snmp  OK        Program
container_memory_gnmi  OK        Program
container_eventd       OK        Program
eventd:eventd          OK        Process
database:redis         OK        Process
syncd:syncd            OK        Process
teamd:teammgrd         OK        Process
teamd:teamsyncd        OK        Process
teamd:tlm_teamd        OK        Process
bgp:zebra              OK        Process
bgp:staticd            OK        Process
bgp:bgpd               OK        Process
bgp:fpmsyncd           OK        Process
bgp:bgpcfgd            OK        Process
swss:orchagent         OK        Process
swss:portsyncd         OK        Process
swss:neighsyncd        OK        Process
swss:fdbsyncd          OK        Process
swss:vlanmgrd          OK        Process
swss:intfmgrd          OK        Process
swss:portmgrd          OK        Process
swss:buffermgrd        OK        Process
swss:vrfmgrd           OK        Process
swss:nbrmgrd           OK        Process
swss:vxlanmgrd         OK        Process
swss:coppmgrd          OK        Process
swss:tunnelmgrd        OK        Process
snmp:snmpd             OK        Process
snmp:snmp-subagent     OK        Process
lldp:lldpd             OK        Process
lldp:lldp-syncd        OK        Process
gnmi:gnmi-native       OK        Process
ASIC_TEMP              OK        ASIC
Fantray1_1             OK        Fan
Fantray1_2             OK        Fan
Fantray2_1             OK        Fan
Fantray2_2             OK        Fan
Fantray3_1             OK        Fan
Fantray3_2             OK        Fan
Fantray4_1             OK        Fan
Fantray4_2             OK        Fan
PSU1_FAN1              OK        Fan
PSU2_FAN1              OK        Fan
PSU 1                  OK        PSU
PSU 2                  OK        PSU

System services and devices ignore list

Name    Status    Type
------  --------  ------
admin@sonic:~$

The relevant syslog is as follows:

root@sonic:/var/log# cat syslog | grep lldpmgrd
Jan  2 08:57:03.266901 sonic INFO lldp#supervisord 2024-01-02 08:57:03,266 INFO spawned: 'lldpmgrd' with pid 35
Jan  2 08:57:03.398679 sonic INFO lldp#lldpmgrd[35]: Starting up...
Jan  2 08:57:03.401389 sonic INFO lldp#lldpmgrd[35]: Device Config Opcode: SET Dict {'buffer_model': 'traditional', 'default_bgp_status': 'up', 'de
fault_pfcwd_status': 'disable', 'hwsku': 'M2-W6510-48V8C', 'mac': '1a:2b:3c:4d:12:31', 'platform': 'x86_64-micas_m2-w6510-48v8c-r0', 'timezone': 'U
TC'} Key localhost
Jan  2 08:57:03.401389 sonic INFO lldp#lldpmgrd[35]: Hostname changed old None, new None
Jan  2 08:57:03.402069 sonic INFO lldp#supervisord: lldpmgrd Traceback (most recent call last):
Jan  2 08:57:03.402069 sonic INFO lldp#supervisord: lldpmgrd   File "/usr/bin/lldpmgrd", line 369, in <module>
Jan  2 08:57:03.402290 sonic INFO lldp#supervisord: lldpmgrd     main()
Jan  2 08:57:03.402306 sonic INFO lldp#supervisord: lldpmgrd   File "/usr/bin/lldpmgrd", line 351, in main
Jan  2 08:57:03.402306 sonic INFO lldp#supervisord: lldpmgrd     lldpmgr.run()
Jan  2 08:57:03.402315 sonic INFO lldp#supervisord: lldpmgrd   File "/usr/bin/lldpmgrd", line 319, in run
Jan  2 08:57:03.402465 sonic INFO lldp#supervisord: lldpmgrd     self.lldp_process_device_table_event(op, dict(fvp), key)
Jan  2 08:57:03.402465 sonic INFO lldp#supervisord: lldpmgrd   File "/usr/bin/lldpmgrd", line 253, in lldp_process_device_table_event
Jan  2 08:57:03.402618 sonic INFO lldp#supervisord: lldpmgrd     self.update_hostname(hostname)
Jan  2 08:57:03.402641 sonic INFO lldp#supervisord: lldpmgrd   File "/usr/bin/lldpmgrd", line 88, in update_hostname
Jan  2 08:57:03.402641 sonic INFO lldp#supervisord: lldpmgrd     proc = subprocess.Popen(cmd,stdout=subprocess.PIPE, stderr=subprocess.PIPE)
Jan  2 08:57:03.402641 sonic INFO lldp#supervisord: lldpmgrd   File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
Jan  2 08:57:03.402914 sonic INFO lldp#supervisord: lldpmgrd     self._execute_child(args, executable, preexec_fn, close_fds,
Jan  2 08:57:03.402914 sonic INFO lldp#supervisord: lldpmgrd   File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
Jan  2 08:57:03.403439 sonic INFO lldp#supervisord: lldpmgrd     self.pid = _posixsubprocess.fork_exec(
Jan  2 08:57:03.403439 sonic INFO lldp#supervisord: lldpmgrd TypeError: expected str, bytes or os.PathLike object, not NoneType
Jan  2 08:57:03.414885 sonic INFO lldp#supervisord 2024-01-02 08:57:03,414 INFO exited: lldpmgrd (exit status 1; not expected)
Jan  2 08:57:04.418009 sonic INFO lldp#supervisord 2024-01-02 08:57:04,417 INFO spawned: 'lldpmgrd' with pid 36

Steps to reproduce the issue:

  1. Change device/micas/x86_64-micas_m2-w6510-48v8c-r0/default_sku to l2 mode.
  2. Upgrade the Sonic system using the ONIE method.

Describe the results you received:

The status of lldpmgrd is Not OK.

Describe the results you expected:

The status of all features is OK.

Output of show version:

2024-01-05-_09:08:21_admin@sonic:~$ show version 
2024-01-05-_09:08:22_
2024-01-05-_09:08:22_SONiC Software Version: SONiC.master.0-dirty-20231231.012057
2024-01-05-_09:08:22_SONiC OS Version: 12
2024-01-05-_09:08:22_Distribution: Debian 12.4
2024-01-05-_09:08:22_Kernel: 6.1.0-11-2-amd64
2024-01-05-_09:08:22_Build commit: 6d3cef1a9
2024-01-05-_09:08:22_Build date: Sun Dec 31 09:28:18 UTC 2023
2024-01-05-_09:08:22_Built by: micasrd@micasrd-PowerEdge-R730
2024-01-05-_09:08:22_
2024-01-05-_09:08:22_Platform: x86_64-micas_m2-w6510-48v8c-r0
2024-01-05-_09:08:22_HwSKU: M2-W6510-48V8C
2024-01-05-_09:08:22_ASIC: broadcom
2024-01-05-_09:08:22_ASIC Count: 1
2024-01-05-_09:08:22_Serial Number: AAA1234567891
2024-01-05-_09:08:22_Model Number: 01018905
2024-01-05-_09:08:22_Hardware Revision: 101
2024-01-05-_09:08:22_Uptime: 08:55:30 up 0 min,  1 user,  load average: 0.82, 0.21, 0.07
2024-01-05-_09:08:22_Date: Tue 02 Jan 2024 08:55:30
2024-01-05-_09:08:22_
2024-01-05-_09:08:22_Docker images:
2024-01-05-_09:08:22_REPOSITORY                    TAG                              IMAGE ID       SIZE
2024-01-05-_09:08:22_docker-dhcp-relay             latest                           d0b94c17212a   312MB
2024-01-05-_09:08:22_docker-macsec                 latest                           91464c654a10   331MB
2024-01-05-_09:08:22_docker-eventd                 latest                           8c834b9212c2   302MB
2024-01-05-_09:08:22_docker-eventd                 master.0-dirty-20231231.012057   8c834b9212c2   302MB
2024-01-05-_09:08:22_docker-orchagent              latest                           e4c0f74aa7ba   341MB
2024-01-05-_09:08:22_docker-orchagent              master.0-dirty-20231231.012057   e4c0f74aa7ba   341MB
2024-01-05-_09:08:22_docker-fpm-frr                latest                           1fb8f97a5701   360MB
2024-01-05-_09:08:22_docker-fpm-frr                master.0-dirty-20231231.012057   1fb8f97a5701   360MB
2024-01-05-_09:08:22_docker-nat                    latest                           a6d4008c3e25   332MB
2024-01-05-_09:08:22_docker-nat                    master.0-dirty-20231231.012057   a6d4008c3e25   332MB
2024-01-05-_09:08:22_docker-sflow                  latest                           76e24af2aaed   330MB
2024-01-05-_09:08:22_docker-sflow                  master.0-dirty-20231231.012057   76e24af2aaed   330MB
2024-01-05-_09:08:22_docker-teamd                  latest                           bb3fc3f89243   329MB
2024-01-05-_09:08:22_docker-teamd                  master.0-dirty-20231231.012057   bb3fc3f89243   329MB
2024-01-05-_09:08:22_docker-snmp                   latest                           7a812b54845e   341MB
2024-01-05-_09:08:22_docker-snmp                   master.0-dirty-20231231.012057   7a812b54845e   341MB
2024-01-05-_09:08:22_docker-syncd-brcm             latest                           9e59eb4a1b23   716MB
2024-01-05-_09:08:22_docker-syncd-brcm             master.0-dirty-20231231.012057   9e59eb4a1b23   716MB
2024-01-05-_09:08:22_docker-sonic-gnmi             latest                           b213fcfcdc8c   390MB
2024-01-05-_09:08:22_docker-sonic-gnmi             master.0-dirty-20231231.012057   b213fcfcdc8c   390MB
2024-01-05-_09:08:22_docker-mux                    latest                           fbc470323dd0   351MB
2024-01-05-_09:08:22_docker-mux                    master.0-dirty-20231231.012057   fbc470323dd0   351MB
2024-01-05-_09:08:22_docker-platform-monitor       latest                           89d17fc7e38b   424MB
2024-01-05-_09:08:22_docker-platform-monitor       master.0-dirty-20231231.012057   89d17fc7e38b   424MB
2024-01-05-_09:08:22_docker-lldp                   latest                           df30d069a79b   345MB
2024-01-05-_09:08:22_docker-lldp                   master.0-dirty-20231231.012057   df30d069a79b   345MB
2024-01-05-_09:08:22_docker-gbsyncd-credo          latest                           392899f52364   325MB
2024-01-05-_09:08:22_docker-gbsyncd-credo          master.0-dirty-20231231.012057   392899f52364   325MB
2024-01-05-_09:08:22_docker-gbsyncd-broncos        latest                           798a80ae07a7   353MB
2024-01-05-_09:08:22_docker-gbsyncd-broncos        master.0-dirty-20231231.012057   798a80ae07a7   353MB
2024-01-05-_09:08:22_docker-router-advertiser      latest                           30f4448b9588   302MB
2024-01-05-_09:08:22_docker-router-advertiser      master.0-dirty-20231231.012057   30f4448b9588   302MB
2024-01-05-_09:08:22_docker-database               latest                           f75c57a1b966   302MB
2024-01-05-_09:08:22_docker-database               master.0-dirty-20231231.012057   f75c57a1b966   302MB
2024-01-05-_09:08:22_docker-sonic-mgmt-framework   latest                           5a87cf1b8f09   383MB
2024-01-05-_09:08:22_docker-sonic-mgmt-framework   master.0-dirty-20231231.012057   5a87cf1b8f09   383MB
2024-01-05-_09:08:22_
2024-01-05-_09:08:26_admin@sonic:~$

Additional information you deem important (e.g. issue happens only occasionally):

volodymyrsamotiy commented 8 months ago

Looks like there is a fix and once PR is merged issue will be closed automatically.