sonic-net / sonic-buildimage

Scripts which perform an installable binary image build for SONiC
Other
711 stars 1.36k forks source link

sfpshow presence/eeprom can't show correctly, error message is :TypeError: 'NoneType' object is not iterable #9181

Closed AntheaJiang closed 2 years ago

AntheaJiang commented 2 years ago

Description

Did anyone met this error these days in 201911 branch? When run platform api command: sfpshow presence/eeprom, there are errors,the image built in nov 1 is ok,there is no error when run sfpshow presence/eeprom, but error occurs in the image built since nov 3,I don't know how to fix it, is there anyone can help about this?

Steps to reproduce the issue:

1.root@sonic:/# sfpshow presence Traceback (most recent call last): File "/usr/bin/sfpshow", line 416, in cli() File "/usr/lib/python2.7/dist-packages/click/core.py", line 722, in call return self.main(args, kwargs) File "/usr/lib/python2.7/dist-packages/click/core.py", line 697, in main rv = self.invoke(ctx) File "/usr/lib/python2.7/dist-packages/click/core.py", line 1066, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/lib/python2.7/dist-packages/click/core.py", line 895, in invoke return ctx.invoke(self.callback, ctx.params) File "/usr/lib/python2.7/dist-packages/click/core.py", line 535, in invoke return callback(args, *kwargs) File "/usr/bin/sfpshow", line 412, in presence sfp.get_presence() File "/usr/lib/python2.7/dist-packages/utilities_common/multi_asic.py", line 133, in wrapped_run_on_all_asics func(self, args, **kwargs) File "/usr/bin/sfpshow", line 355, in get_presence for i in port_table_keys: TypeError: 'NoneType' object is not iterable

2.root@sonic:/# show interfaces status Interface Lanes Speed MTU FEC Alias Vlan Oper Admin Type Asym PFC


root@sonic:/#

Describe the results you expected:

root@sonic:/home/admin# sfputil show eeprom Ethernet0: SFP EEPROM detected Connector: No separable connector Encoding: 64B66B Extended Identifier: Unknown Extended RateSelect Compliance: QSFP+ Rate Select Version 1 Identifier: QSFP28 or later Length Cable Assembly(m): 0 Length OM1(m): 0 Length OM2(m): 0 Length OM3(2m): 0 Length(km): 0 Nominal Bit Rate(100Mbs): 255 Specification compliance: 10/40G Ethernet Compliance Code: 40GBASE-CR4 Vendor Date Code(YYYY-MM-DD Lot): 2018-10-04 00 Vendor Name: LEONI Vendor OUI: a8-b0-ae Vendor PN: L45593-D900-M115 Vendor Rev: 03 Vendor SN: LEO1840G1DR

Output of show version:

root@sonic:/# show version

SONiC Software Version: SONiC.201911_SX_MX_SecureBoot.0-dirty-20211105.023734
Distribution: Debian 9.13
Kernel: 4.9.0-14-2-amd64
Build commit: 80116dede
Build date: Fri Nov  5 02:43:12 UTC 2021
Built by: jersey@cthgd047

Platform: x86_64-cel_midstone-100x-r0
HwSKU: Midstone-100x
ASIC: innovium
Serial Number: 0C:48:C6:97:FC:B2
Uptime: 15:42:09 up  2:13,  1 user,  load average: 0.47, 0.54, 0.64

Docker images:
REPOSITORY                    TAG                                               IMAGE ID            SIZE
docker-syncd-invm             201911_SX_MX_SecureBoot.0-dirty-20211105.023734   54b1c320531e        406MB
docker-syncd-invm             latest                                            54b1c320531e        406MB
docker-sonic-telemetry        201911_SX_MX_SecureBoot.0-dirty-20211105.023734   45929bce76c0        354MB
docker-sonic-telemetry        latest                                            45929bce76c0        354MB
docker-router-advertiser      201911_SX_MX_SecureBoot.0-dirty-20211105.023734   3460eaf04920        290MB
docker-router-advertiser      latest                                            3460eaf04920        290MB
docker-sonic-mgmt-framework   201911_SX_MX_SecureBoot.0-dirty-20211105.023734   88d93412ad0c        429MB
docker-sonic-mgmt-framework   latest                                            88d93412ad0c        429MB
docker-platform-monitor       201911_SX_MX_SecureBoot.0-dirty-20211105.023734   877e43f702e6        358MB
docker-platform-monitor       latest                                            877e43f702e6        358MB
docker-fpm-frr                201911_SX_MX_SecureBoot.0-dirty-20211105.023734   39ea664270cc        336MB
docker-fpm-frr                latest                                            39ea664270cc        336MB
docker-lldp-sv2               201911_SX_MX_SecureBoot.0-dirty-20211105.023734   33436e425d50        312MB
docker-lldp-sv2               latest                                            33436e425d50        312MB
docker-dhcp-relay             201911_SX_MX_SecureBoot.0-dirty-20211105.023734   088f2f211bb2        300MB
docker-dhcp-relay             latest                                            088f2f211bb2        300MB
docker-database               201911_SX_MX_SecureBoot.0-dirty-20211105.023734   757b08795f47        290MB
docker-database               latest                                            757b08795f47        290MB
docker-teamd                  201911_SX_MX_SecureBoot.0-dirty-20211105.023734   a7666ca17637        315MB
docker-teamd                  latest                                            a7666ca17637        315MB
docker-snmp-sv2               201911_SX_MX_SecureBoot.0-dirty-20211105.023734   ad997e11fead        353MB
docker-snmp-sv2               latest                                            ad997e11fead        353MB
docker-orchagent              201911_SX_MX_SecureBoot.0-dirty-20211105.023734   56c1662ecf25        334MB
docker-orchagent              latest                                            56c1662ecf25        334MB
docker-sflow                  201911_SX_MX_SecureBoot.0-dirty-20211105.023734   92778074c802        316MB
docker-sflow                  latest                                            92778074c802        316MB
docker-nat                    201911_SX_MX_SecureBoot.0-dirty-20211105.023734   1bbf5ddeef14        317MB
docker-nat                    latest  

Additional information you deem important (e.g. issue happens only occasionally):

I met this error in September ,and solved it by update codes from Azure/201911,but this time it doesn't work,error still exist by updating code.

AntheaJiang commented 2 years ago

docker ps result: root@sonic:/# docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 53d8eae0f8d6 docker-sonic-telemetry:latest "/usr/bin/supervisord" 3 hours ago Up 2 hours telemetry 46303d851b69 docker-sonic-mgmt-framework:latest "/usr/bin/supervisord" 3 hours ago Up 6 minutes mgmt-framework 7d31495aabd9 docker-snmp-sv2:latest "/usr/bin/supervisord" 3 hours ago Up 5 minutes snmp 2da2c0359f8c docker-lldp-sv2:latest "/usr/bin/docker-lld…" 3 hours ago Up 2 hours lldp 4740870420b3 docker-router-advertiser:latest "/usr/bin/docker-ini…" 3 hours ago Up 5 minutes radv 434412ed04b9 docker-dhcp-relay:latest "/usr/bin/docker_ini…" 3 hours ago Up 5 minutes dhcp_relay bd3e3fba9a8d docker-syncd-invm:latest "/usr/bin/supervisord" 3 hours ago Up 5 minutes syncd 8777af003ea7 docker-teamd:latest "/usr/bin/supervisord" 3 hours ago Up 5 minutes teamd b8e1847bc0b6 docker-orchagent:latest "/usr/bin/docker-ini…" 3 hours ago Up 6 minutes swss 88c9aaf53fd3 docker-fpm-frr:latest "/usr/bin/supervisord" 3 hours ago Up 2 hours bgp 0b518868142e docker-platform-monitor:latest "/usr/bin/docker_ini…" 3 hours ago Up 2 hours pmon d2fd18fd62b8 docker-database:latest "/usr/local/bin/dock…" 3 hours ago Up 2 hours database

AntheaJiang commented 2 years ago

@bbinxie Hi, sorry to bother you,but did you see this failure like this before?I have no idea how to solve it at all.

lguohan commented 2 years ago

@Hanly-Cameo, can you take a look, this is related to innovium platform.

nicwu-cel commented 2 years ago

It has been approved that it's caused by the mismatch of ipd.ko and linux kernel. When this problem happens, perform "lsmod | grep ipd" will have no output.

AntheaJiang commented 2 years ago

@Hanly-Cameo Hi,as nicwy-cel's comment,it is that ipd module make sfpshow can't run normally,and the issue still exists by now ,could you please help to update ipd.deb ?

AntheaJiang commented 2 years ago

There is another failure phenomenon :key is _PORT_TABLE instead of PORT_TABLE in APPL_DB Annotation 2021-07-27 113443 and there are no PORT_TABLE in state_db: 11

AntheaJiang commented 2 years ago

find the rootcause:it‘s because switch chip model changed,and sdk/sai add switch failed the issue can be closed now.

AspenBen commented 2 years ago

@AntheaJiang How did u resolve the bug ( 1 Dec 2021) about "_PORT_TABLE" in APPL_DB

AntheaJiang commented 2 years ago

@AntheaJiang How did u resolve the bug ( 1 Dec 2021) about "_PORT_TABLE" in APPL_DB

@AspenBen It‘s said as above :there are some errors about switch chip configure, and You'd better give the logs to your sdk owner and let them check