search

sonic-net / sonic-buildimage

Scripts which perform an installable binary image build for SONiC
Other
739 stars 1.43k forks source link

Too many open files: '/proc/stat', Error message seen along with snmp errors #13499

Open judyjoseph opened 1 year ago

judyjoseph commented 1 year ago

Description

Too many open files: '/proc/stat', Error message seen along with snmp errors as below on a VoQ linecard running 202205 image.

Jan 25 00:24:19.963935 -lc1 ERR snmp#snmp-subagent [ax_interface] ERROR: MIBUpdater.start() caught an unexpected exception during update_data()#012Traceback (most recent call last):#012  File "/usr/local/lib/python3.9/dist-packages/ax_interface/mib.py", line 37, in start#012    self.reinit_data()#012  File "/usr/local/lib/python3.9/dist-packages/sonic_ax_impl/mibs/ietf/rfc2737.py", line 290, in reinit_data#012  File "/usr/local/lib/python3.9/dist-packages/sonic_ax_impl/mibs/__init__.py", line 474, in get_device_metadata#012  File "/usr/lib/python3/dist-packages/swsscommon/swsscommon.py", line 1514, in connect#012RuntimeError: Unable to connect to redis (unix-socket): Cannot assign requested address
Jan 25 00:24:19.964218 -lc1 ERR snmp#snmp-subagent [ax_interface] ERROR: MIBUpdater.start() caught an unexpected exception during update_data()#012Traceback (most recent call last):#012  File "/usr/local/lib/python3.9/dist-packages/ax_interface/mib.py", line 37, in start#012    self.reinit_data()#012  File "/usr/local/lib/python3.9/dist-packages/sonic_ax_impl/mibs/ieee802_1ab.py", line 182, in reinit_data#012  File "/usr/local/lib/python3.9/dist-packages/sonic_ax_impl/mibs/__init__.py", line 669, in get_sync_d_from_all_namespace#012  File "/usr/local/lib/python3.9/dist-packages/sonic_ax_impl/mibs/__init__.py", line 268, in init_sync_d_interface_tables#012  File "/usr/local/lib/python3.9/dist-packages/swsssdk/port_util.py", line 73, in get_interface_oid_map#012  File "/usr/lib/python3/dist-packages/swsscommon/swsscommon.py", line 1514, in connect#012RuntimeError: Unable to connect to redis (unix-socket): Cannot assign requested address
Jan 25 00:24:20.963964 -lc1 ERR snmp#snmp-subagent [ax_interface] ERROR: MIBUpdater.start() caught an unexpected exception during update_data()#012Traceback (most recent call last):#012  File "/usr/local/lib/python3.9/dist-packages/ax_interface/mib.py", line 37, in start#012    self.reinit_data()#012  File "/usr/local/lib/python3.9/dist-packages/sonic_ax_impl/mibs/ietf/rfc1213.py", line 716, in reinit_data#012  File "/usr/lib/python3/dist-packages/swsscommon/swsscommon.py", line 1514, in connect#012RuntimeError: Unable to connect to redis: Cannot assign requested address
Jan 25 00:24:20.964144 -lc1 ERR snmp#snmp-subagent [ax_interface] ERROR: MIBUpdater.start() caught an unexpected exception during update_data()#012Traceback (most recent call last):#012  File "/usr/local/lib/python3.9/dist-packages/ax_interface/mib.py", line 37, in start#012    self.reinit_data()#012  File "/usr/local/lib/python3.9/dist-packages/sonic_ax_impl/mibs/ieee802_1ab.py", line 126, in reinit_data#012  File "/usr/local/lib/python3.9/dist-packages/sonic_ax_impl/mibs/__init__.py", line 598, in connect_all_dbs#012  File "/usr/lib/python3/dist-packages/swsscommon/swsscommon.py", line 1514, in connect#012RuntimeError: Unable to connect to redis (unix-socket): Cannot assign requested address
Jan 25 00:24:20.964424 -lc1 ERR snmp#snmp-subagent [ax_interface] ERROR: MIBUpdater.start() caught an unexpected exception during update_data()#012Traceback (most recent call last):#012  File "/usr/local/lib/python3.9/dist-packages/ax_interface/mib.py", line 43, in start#012    self.update_data()#012  File "/usr/local/lib/python3.9/dist-packages/sonic_ax_impl/mibs/vendor/__init__.py", line 57, in update_data#012  File "/usr/local/lib/python3.9/dist-packages/psutil/__init__.py", line 1752, in cpu_percent#012  File "/usr/local/lib/python3.9/dist-packages/psutil/__init__.py", line 1613, in cpu_times#012  File "/usr/local/lib/python3.9/dist-packages/psutil/_pslinux.py", line 586, in cpu_times#012  File "/usr/local/lib/python3.9/dist-packages/psutil/_common.py", line 728, in open_binary#012OSError: [Errno 24] Too many open files: '/proc/stat'
Jan 25 00:24:21.963019 -lc1 ERR snmp#snmp-subagent [ax_interface] ERROR: MIBUpdater.start() caught an unexpected exception during update_data()#012Traceback (most recent call last):#012  File "/usr/local/lib/python3.9/dist-packages/ax_interface/mib.py", line 37, in start#012    self.reinit_data()#012  File "/usr/local/lib/python3.9/dist-packages/sonic_ax_impl/mibs/ietf/rfc1213.py", line 223, in reinit_data#012  File "/usr/local/lib/python3.9/dist-packages/sonic_ax_impl/mibs/__init__.py", line 669, in get_sync_d_from_all_namespace#012  File "/usr/local/lib/python3.9/dist-packages/sonic_ax_impl/mibs/__init__.py", line 268, in init_sync_d_interface_tables#012  File "/usr/local/lib/python3.9/dist-packages/swsssdk/port_util.py", line 73, in get_interface_oid_map#012  File "/usr/lib/python3/dist-packages/swsscommon/swsscommon.py", line 1514, in connect#012RuntimeError: Unable to connect to redis (unix-socket): Cannot assign requested address

Steps to reproduce the issue:

  1. Boot the device in 202205 image, the logs appear in syslog

Describe the results you received:

Too many open files error message in syslog

Describe the results you expected:

These snmp errors need to be checked, even if there are snmp failures there should not be Too many open files error message.

Output of show version:

-lc1:~$ show version 

SONiC Software Version: SONiC.20220531.14
Distribution: Debian 11.6
Kernel: 5.10.0-18-2-amd64
Build commit: 47473541fa
Build date: Fri Dec 23 05:44:02 UTC 2022
Built by: cloudtest@cb7cfe51c000000

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):

judyjoseph commented 1 year ago

@SuvarnaMeenakshi to take a look as there was a similar issue with too many files open