search

sonic-net / sonic-buildimage

Scripts which perform an installable binary image build for SONiC
Other
723 stars 1.38k forks source link

show system-health detail will not complete #9530

Closed mbze430 closed 3 months ago

mbze430 commented 2 years ago

Description

sudo show system-health detail doesn't complete and shows the following:

Traceback (most recent call last):
  File "/usr/local/bin/show", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/show/system_health.py", line 110, in detail
    chassis.initizalize_system_led()
AttributeError: 'Chassis' object has no attribute 'initizalize_system_led'

Steps to reproduce the issue:

  1. sudo show system-health detail

Describe the results you expected:

complete with full information

Output of show version:

SONiC Software Version: SONiC.202106.57766-e2b8e2d1d
Distribution: Debian 10.11
Kernel: 4.19.0-12-2-amd64
Build commit: e2b8e2d1d
Build date: Wed Dec  8 17:11:08 UTC 2021
Built by: AzDevOps@sonic-build-workers-000YSN

Platform: x86_64-cel_seastone-r0
HwSKU: Seastone-DX010
ASIC: broadcom
ASIC Count: 1
Serial Number: DX010B2F108423LK100045
Model Number: R0872-F0010-01
Hardware Revision: N/A
Uptime: 23:15:17 up 2 days, 12:18,  1 user,  load average: 1.01, 0.94, 1.03

Docker images:
REPOSITORY                    TAG                      IMAGE ID            SIZE
docker-dhcp-relay             latest                   96fe9f328ca6        428MB
docker-platform-monitor       202106.57766-e2b8e2d1d   46b2c81ae323        636MB
docker-platform-monitor       latest                   46b2c81ae323        636MB
docker-snmp                   202106.57766-e2b8e2d1d   1c6c64d06bb5        462MB
docker-snmp                   latest                   1c6c64d06bb5        462MB
docker-orchagent              202106.57766-e2b8e2d1d   647b930886d5        451MB
docker-orchagent              latest                   647b930886d5        451MB
docker-fpm-frr                202106.57766-e2b8e2d1d   4eaf5f04fe20        451MB
docker-fpm-frr                latest                   4eaf5f04fe20        451MB
docker-nat                    202106.57766-e2b8e2d1d   4d7952ffc394        435MB
docker-nat                    latest                   4d7952ffc394        435MB
docker-sflow                  202106.57766-e2b8e2d1d   4adc78f7d1d9        433MB
docker-sflow                  latest                   4adc78f7d1d9        433MB
docker-macsec                 202106.57766-e2b8e2d1d   a8caf2d88d49        436MB
docker-macsec                 latest                   a8caf2d88d49        436MB
docker-syncd-brcm             202106.57766-e2b8e2d1d   40ffb870205e        717MB
docker-syncd-brcm             latest                   40ffb870205e        717MB
docker-sonic-mgmt-framework   202106.57766-e2b8e2d1d   4615dec8f36c        576MB
docker-sonic-mgmt-framework   latest                   4615dec8f36c        576MB
docker-teamd                  202106.57766-e2b8e2d1d   1544a4170ea9        432MB
docker-teamd                  latest                   1544a4170ea9        432MB
docker-router-advertiser      202106.57766-e2b8e2d1d   8c0c464e5db5        421MB
docker-router-advertiser      latest                   8c0c464e5db5        421MB
docker-lldp                   202106.57766-e2b8e2d1d   499f4a39da20        461MB
docker-lldp                   latest                   499f4a39da20        461MB
docker-sonic-telemetry        202106.57766-e2b8e2d1d   1bc7246fe59f        509MB
docker-sonic-telemetry        latest                   1bc7246fe59f        509MB
docker-database               202106.57766-e2b8e2d1d   e986c6f35b4c        421MB
docker-database               latest                   e986c6f35b4c        421MB

Output of show techsupport:

sonic_dump_dx010_20211213_231617.tar.gz

MaratGubaiev commented 2 years ago

Results of my testing on a mellanox device: 1) branch 202106

"sudo show system-health detail" ``` admin@sonic:~$ sudo show system-health detail System is currently booting... admin@sonic:~$ sudo show system-health detail System is currently booting... admin@sonic:~$ sudo show system-health detail System status summary System status LED green Services: Status: OK Hardware: Status: OK System services and devices monitor list Name Status Type -------------------------- -------- ---------- sonic OK System rsyslog OK Process root-overlay OK Filesystem var-log OK Filesystem routeCheck OK Program diskCheck OK Program container_checker OK Program vnetRouteCheck OK Program container_memory_telemetry OK Program ASIC OK ASIC fan1 OK Fan fan2 OK Fan fan3 OK Fan fan4 OK Fan fan5 OK Fan fan6 OK Fan fan7 OK Fan fan8 OK Fan psu1_fan1 OK Fan psu2_fan1 OK Fan PSU 1 OK PSU PSU 2 OK PSU System services and devices ignore list Name Status Type ----------- -------- ------ psu.voltage Ignored Device ```
"show version" ``` admin@sonic:~$ show version SONiC Software Version: SONiC.202106.54351-1ebe52847 Distribution: Debian 10.11 Kernel: 4.19.0-12-2-amd64 Build commit: 1ebe52847 Build date: Thu Nov 25 14:39:20 UTC 2021 Built by: AzDevOps@sonic-build-workers-000XGC Platform: x86_64-mlnx_msn2700-r0 HwSKU: Mellanox-SN2700 ASIC: mellanox ASIC Count: 1 Serial Number: MT2028X35388 Model Number: MSN2700-CS2FO Hardware Revision: A2 Uptime: 09:16:54 up 22 min, 1 user, load average: 0.44, 0.50, 0.62 Docker images: REPOSITORY TAG IMAGE ID SIZE docker-syncd-mlnx 202106.54351-1ebe52847 792b958f678f 1e+03MB docker-syncd-mlnx latest 792b958f678f 1e+03MB docker-platform-monitor 202106.54351-1ebe52847 32fa4bd153ef 749MB docker-platform-monitor latest 32fa4bd153ef 749MB docker-dhcp-relay latest 916d9cf1d010 428MB docker-snmp 202106.54351-1ebe52847 7fa440dd7434 462MB docker-snmp latest 7fa440dd7434 462MB docker-fpm-frr 202106.54351-1ebe52847 7e1861e01a63 451MB docker-fpm-frr latest 7e1861e01a63 451MB docker-sflow 202106.54351-1ebe52847 b288881d7fd2 433MB docker-sflow latest b288881d7fd2 433MB docker-nat 202106.54351-1ebe52847 e14ea32231df 435MB docker-nat latest e14ea32231df 435MB docker-teamd 202106.54351-1ebe52847 bb64454ecf61 432MB docker-teamd latest bb64454ecf61 432MB docker-router-advertiser 202106.54351-1ebe52847 a90b56daa4b5 421MB docker-router-advertiser latest a90b56daa4b5 421MB docker-lldp 202106.54351-1ebe52847 e2ad97ee3fb3 461MB docker-lldp latest e2ad97ee3fb3 461MB docker-orchagent 202106.54351-1ebe52847 d16e94d969fa 451MB docker-orchagent latest d16e94d969fa 451MB docker-database 202106.54351-1ebe52847 587271cc3354 421MB docker-database latest 587271cc3354 421MB docker-macsec 202106.54351-1ebe52847 f6b76e42e341 436MB docker-macsec latest f6b76e42e341 436MB docker-sonic-telemetry 202106.54351-1ebe52847 61bf0d0569fa 509MB docker-sonic-telemetry latest 61bf0d0569fa 509MB docker-sonic-mgmt-framework 202106.54351-1ebe52847 a7c115e11ab7 576MB docker-sonic-mgmt-framework latest a7c115e11ab7 576MB ```

2) branch master

"sudo show system-health detail" ``` admin@sonic:~$ sudo show system-health detail System status summary System status LED green Services: Status: OK Hardware: Status: OK System services and devices monitor list Name Status Type -------------------------- -------- ---------- sonic OK System rsyslog OK Process root-overlay OK Filesystem var-log OK Filesystem routeCheck OK Program diskCheck OK Program container_checker OK Program vnetRouteCheck OK Program container_memory_telemetry OK Program snmp:snmpd OK Process snmp:snmp-subagent OK Process lldp:lldpd OK Process lldp:lldp-syncd OK Process lldp:lldpmgrd OK Process syncd:syncd OK Process teamd:teammgrd OK Process teamd:teamsyncd OK Process teamd:tlm_teamd OK Process swss:orchagent OK Process swss:portsyncd OK Process swss:neighsyncd OK Process swss:fdbsyncd OK Process swss:vlanmgrd OK Process swss:intfmgrd OK Process swss:portmgrd OK Process swss:buffermgrd OK Process swss:vrfmgrd OK Process swss:nbrmgrd OK Process swss:vxlanmgrd OK Process swss:coppmgrd OK Process swss:tunnelmgrd OK Process bgp:zebra OK Process bgp:staticd OK Process bgp:bgpd OK Process bgp:fpmsyncd OK Process bgp:bgpcfgd OK Process database:redis OK Process telemetry:telemetry OK Process telemetry:dialout OK Process ASIC OK ASIC fan1 OK Fan fan2 OK Fan fan3 OK Fan fan4 OK Fan fan5 OK Fan fan6 OK Fan fan7 OK Fan fan8 OK Fan psu1_fan1 OK Fan psu2_fan1 OK Fan PSU 1 OK PSU PSU 2 OK PSU System services and devices ignore list Name Status Type ----------- -------- ------ psu.voltage Ignored Device ```
"show version" ``` admin@sonic:~$ "show version" SONiC Software Version: SONiC.generic_updater2.0-dirty-20211210.170524 Distribution: Debian 11.1 Kernel: 5.10.0-8-2-amd64 Build commit: afd40984f Build date: Sat Dec 11 02:03:49 UTC 2021 Built by: cord@cord Platform: x86_64-mlnx_msn2700-r0 HwSKU: Mellanox-SN2700 ASIC: mellanox ASIC Count: 1 Serial Number: MT2028X35388 Model Number: MSN2700-CS2FO Hardware Revision: A2 Uptime: 08:48:19 up 23:10, 1 user, load average: 0.63, 0.86, 0.73 Docker images: REPOSITORY TAG IMAGE ID SIZE docker-dhcp-relay latest 5240b754f86d 436MB docker-teamd generic_updater2.0-dirty-20211210.170524 a69713d9ec20 435MB docker-teamd latest a69713d9ec20 435MB docker-syncd-mlnx generic_updater2.0-dirty-20211210.170524 d46f37ca1596 1.01GB docker-syncd-mlnx latest d46f37ca1596 1.01GB docker-snmp generic_updater2.0-dirty-20211210.170524 309779b9e2d3 465MB docker-snmp latest 309779b9e2d3 465MB docker-sonic-mgmt-framework generic_updater2.0-dirty-20211210.170524 093f0ad0a25b 578MB docker-sonic-mgmt-framework latest 093f0ad0a25b 578MB docker-platform-monitor generic_updater2.0-dirty-20211210.170524 4d8ee6d5e79f 806MB docker-platform-monitor latest 4d8ee6d5e79f 806MB docker-sonic-telemetry generic_updater2.0-dirty-20211210.170524 6770440af8d2 511MB docker-sonic-telemetry latest 6770440af8d2 511MB docker-sflow generic_updater2.0-dirty-20211210.170524 188ec4950a4e 436MB docker-sflow latest 188ec4950a4e 436MB docker-orchagent generic_updater2.0-dirty-20211210.170524 052b5f09f3b9 454MB docker-orchagent latest 052b5f09f3b9 454MB docker-router-advertiser generic_updater2.0-dirty-20211210.170524 6202829f8948 423MB docker-router-advertiser latest 6202829f8948 423MB docker-nat generic_updater2.0-dirty-20211210.170524 c3859b7adab6 438MB docker-nat latest c3859b7adab6 438MB docker-mux generic_updater2.0-dirty-20211210.170524 4fb64fc6e87f 475MB docker-mux latest 4fb64fc6e87f 475MB docker-macsec generic_updater2.0-dirty-20211210.170524 30b5a7eca51c 439MB docker-macsec latest 30b5a7eca51c 439MB docker-fpm-frr generic_updater2.0-dirty-20211210.170524 30f291874c63 454MB docker-fpm-frr latest 30f291874c63 454MB docker-lldp generic_updater2.0-dirty-20211210.170524 69a0e7d0abce 463MB docker-lldp latest 69a0e7d0abce 463MB docker-database generic_updater2.0-dirty-20211210.170524 fd09177e9f7a 423MB docker-database latest fd09177e9f7a 423MB ```

So I cannot reproduce the issue on the mellanox platform.

mbze430 commented 2 years ago

could be a specific platform issue. Are there anyone that works on the Celestica?

qnos commented 1 year ago

This is the same issue as issue #11322 , check the comment in that thread.

FunForNOS commented 1 year ago

Can confirm this bug is fixed. Tested with build id 207726.