sonic-net / sonic-buildimage

Scripts which perform an installable binary image build for SONiC
Other
741 stars 1.43k forks source link

[chassis]ValueError exception in interface counter #19192

Open sdszhang opened 5 months ago

sdszhang commented 5 months ago

Description

exception in show interface count or portstats

$ sudo ip netns exec asic2 portstat | grep -E 'IFACE|Ethernet256|Ethernet264'

Traceback (most recent call last):
  File "/usr/local/bin/portstat", line 675, in <module>
    main()
  File "/usr/local/bin/portstat", line 635, in main
    cnstat_dict, ratestat_dict = portstat.get_cnstat_dict()
  File "/usr/local/bin/portstat", line 151, in get_cnstat_dict
    self.collect_stat()
  File "/usr/local/lib/python3.9/dist-packages/utilities_common/multi_asic.py", line 157, in wrapped_run_on_all_asics
    func(self,  *args, **kwargs)
  File "/usr/local/bin/portstat", line 161, in collect_stat
    cnstat_dict, ratestat_dict = self.get_cnstat()
  File "/usr/local/bin/portstat", line 215, in get_cnstat
    cnstat_dict[port] = get_counters(port)
  File "/usr/local/bin/portstat", line 182, in get_counters
    fields[pos] = str(int(fields[pos]) + int(fvs[counter_name]))
ValueError: invalid literal for int() with base 10: '1.8342735403386e+14'

Steps to reproduce the issue:

  1. Sending large volume of traffic using traffic generator.
  2. run "portstat" or "show interface count" or "sonic-clear counter"

Describe the results you received:

Describe the results you expected:

Output of show version:

admin@svcstr2-8800-lc4-1:~$ show ver

SONiC Software Version: SONiC.20230532.14
SONiC OS Version: 11
Distribution: Debian 11.8
Kernel: 5.10.0-23-2-amd64
Build commit: 511a884d5f
Build date: Tue Apr 23 17:33:45 UTC 2024
Built by: cloudtest@badcc15fc000001

Platform: x86_64-88_lc0_36fh_m-r0
HwSKU: Cisco-88-LC0-36FH-M-O36
ASIC: cisco-8000
ASIC Count: 3
Serial Number: FOC2548N36J
Model Number: 88-LC0-36FH-M
Hardware Revision: 1.0
Uptime: 03:41:06 up 12 days,  1:52,  2 users,  load average: 6.58, 4.86, 4.43
Date: Tue 04 Jun 2024 03:41:06

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):

prabhataravind commented 5 months ago

@sdszhang is this seen only on chassis or on pizza box devices as well? Could you please confirm if you are working on this?

sdszhang commented 5 months ago

@sdszhang is this seen only on chassis or on pizza box devices as well? Could you please confirm if you are working on this? @prabhataravind I'm suspecting it's on chassis only. Didn't see it being reported on pizza box for the same test. Going to work on this issue next week to see if it's sonic repo issue or vendor issue.