search

sonic-net / SONiC

Landing page for Software for Open Networking in the Cloud (SONiC) - https://sonic-net.github.io/SONiC/
2.27k stars 1.13k forks source link

Dell S6100: snmpwalk for IF-MIB and IP-MIB fails #118

Open chitra-raghavan opened 7 years ago

chitra-raghavan commented 7 years ago

Snmpwalk for IF-MIB and IP-MIB provided in library files are failing in S6100,whereas System mibs are passing . In S6000 Snmpwalk for all the provided MIBs are passing.

Working mibs in s6100:

Snip from library file : /var/johnar/sonic-mgmt/ansible/library/snmp_facts.py

    # From SNMPv2-MIB
    self.sysDescr    = dp + "1.3.6.1.2.1.1.1.0"
    self.sysObjectId = dp + "1.3.6.1.2.1.1.2.0"
    self.sysUpTime   = dp + "1.3.6.1.2.1.1.3.0"
    self.sysContact  = dp + "1.3.6.1.2.1.1.4.0"
    self.sysName     = dp + "1.3.6.1.2.1.1.5.0"
    self.sysLocation = dp + "1.3.6.1.2.1.1.6.0"

Sample o/p in S6100: 

rchitra@login-sjc-01[3:28am] : SONIC >  snmpwalk -v 2c -c public 10.11.150.100 1.3.6.1.2.1.1.1.0
SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.HEAD.361-f7cdca4 - HwSku:        Force10-S6100 - Distribution: Debian 8.9 - Kernel: 3.16.0-4-amd64

rchitra@login-sjc-01[3:28am] : SONIC >   snmpwalk -v 2c -c public 10.11.150.100 1.3.6.1.2.1.1.2.0
SNMPv2-MIB::sysObjectID.0 = OID: NET-SNMP-MIB::netSnmpAgentOIDs.10

Failing mibs in s6100 :

The below oid’s are failing in 6100 .All these OIDs are running fine in S6000

Snip from library file : 

   # From IF-MIB
    self.ifIndex       = dp + "1.3.6.1.2.1.2.2.1.1"
    self.ifDescr       = dp + "1.3.6.1.2.1.2.2.1.2"
    self.ifMtu         = dp + "1.3.6.1.2.1.2.2.1.4"
    self.ifSpeed       = dp + "1.3.6.1.2.1.2.2.1.5"
    self.ifPhysAddress = dp + "1.3.6.1.2.1.2.2.1.6"
    self.ifAdminStatus = dp + "1.3.6.1.2.1.2.2.1.7"
    self.ifOperStatus  = dp + "1.3.6.1.2.1.2.2.1.8"
    self.ifAlias       = dp + "1.3.6.1.2.1.31.1.1.1.18"
    self.ifInDiscards  = dp + "1.3.6.1.2.1.2.2.1.13"
    self.ifOutDiscards = dp + "1.3.6.1.2.1.2.2.1.19"
    self.ifInErrors    = dp + "1.3.6.1.2.1.2.2.1.14"
    self.ifOutErrors   = dp + "1.3.6.1.2.1.2.2.1.20"
    self.ifHCInOctets  = dp + "1.3.6.1.2.1.31.1.1.1.6"
    self.ifHCOutOctets = dp + "1.3.6.1.2.1.31.1.1.1.10"
    self.ifInUcastPkts = dp + "1.3.6.1.2.1.2.2.1.11"
    self.ifOutUcastPkts= dp + "1.3.6.1.2.1.2.2.1.17"

    # From IP-MIB
    self.ipAdEntAddr    = dp + "1.3.6.1.2.1.4.20.1.1"
    self.ipAdEntIfIndex = dp + "1.3.6.1.2.1.4.20.1.2"
    self.ipAdEntNetMask = dp + "1.3.6.1.2.1.4.20.1.3"

    # From Dell Private MIB
    self.ChStackUnitCpuUtil5sec = dp + "1.3.6.1.4.1.6027.3.10.1.2.9.1.2.1"

Sample o/p inS6100 : 

rchitra@login-sjc-01[3:30am] : SONIC > snmpwalk -v 2c -c public 10.11.150.100 1.3.6.1.2.1.2.2.1.1
IF-MIB::ifIndex = No Such Object available on this agent at this OID
1.3.6.1.2.1.2.2 (ifTable)
rchitra@login-sjc-01[3:30am] : SONIC > snmpwalk -v 2c -c public 10.11.150.100 1.3.6.1.2.1.2
IF-MIB::interfaces = No Such Object available on this agent at this OID
1.3.6.1.2.1.31 (ifMIB)
rchitra@login-sjc-01[3:32am] : SONIC > snmpwalk -v 2c -c public 10.11.150.100 1.3.6.1.2.1.31
IF-MIB::ifMIB = No Such Object available on this agent at this OID

1.3.6.1.2.1.4.20 (ipAddrTable)
rchitra@login-sjc-01[3:34am] : SONIC > snmpwalk -v 2c -c public 10.11.150.100 1.3.6.1.2.1.4.20
IP-MIB::ipAddrTable = No Such Object available on this agent at this OID
1.3.6.1.4.1.6027.3 (f10Mgmt)
rchitra@login-sjc-01[3:34am] : SONIC > snmpwalk -v 2c -c public 10.11.150.100 1.3.6.1.4.1.6027.3
SNMPv2-SMI::enterprises.6027.3 = No Such Object available on this agent at this OID

S6000 sample o/p :

rchitra@login-sjc-01[4:08am] : SONIC > snmpwalk -v 2c -c public 10.11.150.18 1.3.6.1.2.1.2.2.1.2
IF-MIB::ifDescr.1 = STRING: fortyGigE0/0
IF-MIB::ifDescr.5 = STRING: fortyGigE0/4
IF-MIB::ifDescr.9 = STRING: fortyGigE0/8
IF-MIB::ifDescr.13 = STRING: fortyGigE0/12
IF-MIB::ifDescr.17 = STRING: fortyGigE0/16
IF-MIB::ifDescr.21 = STRING: fortyGigE0/20
IF-MIB::ifDescr.25 = STRING: fortyGigE0/24
IF-MIB::ifDescr.29 = STRING: fortyGigE0/28
IF-MIB::ifDescr.33 = STRING: fortyGigE0/32
IF-MIB::ifDescr.37 = STRING: fortyGigE0/36
IF-MIB::ifDescr.41 = STRING: fortyGigE0/40
IF-MIB::ifDescr.45 = STRING: fortyGigE0/44
IF-MIB::ifDescr.49 = STRING: fortyGigE0/48
IF-MIB::ifDescr.53 = STRING: fortyGigE0/52
IF-MIB::ifDescr.57 = STRING: fortyGigE0/56
IF-MIB::ifDescr.61 = STRING: fortyGigE0/60
IF-MIB::ifDescr.65 = STRING: fortyGigE0/64
IF-MIB::ifDescr.69 = STRING: fortyGigE0/68
IF-MIB::ifDescr.73 = STRING: fortyGigE0/72
IF-MIB::ifDescr.77 = STRING: fortyGigE0/76
IF-MIB::ifDescr.81 = STRING: fortyGigE0/80
IF-MIB::ifDescr.85 = STRING: fortyGigE0/84
IF-MIB::ifDescr.89 = STRING: fortyGigE0/88
IF-MIB::ifDescr.93 = STRING: fortyGigE0/92
IF-MIB::ifDescr.97 = STRING: fortyGigE0/96
IF-MIB::ifDescr.101 = STRING: fortyGigE0/100
IF-MIB::ifDescr.105 = STRING: fortyGigE0/104
IF-MIB::ifDescr.109 = STRING: fortyGigE0/108
IF-MIB::ifDescr.113 = STRING: fortyGigE0/112
IF-MIB::ifDescr.117 = STRING: fortyGigE0/116
IF-MIB::ifDescr.121 = STRING: fortyGigE0/120
IF-MIB::ifDescr.125 = STRING: fortyGigE0/124
prsunny commented 7 years ago

Did you check it on the latest image? There were few fixes made in the context of SNMP agent. One is https://github.com/Azure/sonic-py-swsssdk/pull/17

You can check the logs to see if there are any failures/crashes in SNMP agent initialization.

stcheng commented 7 years ago

Hello, @chitra-raghavan we've run the SNMP test against the S6100 platform, and it passes fine. Please check if you're on the latest version of the image. Thanks.

root@s6100:/home/admin# show version
SONiC Software Version: SONiC.HEAD.294-3417e78 
Distribution: Debian 8.9                       
Kernel: 3.16.0-4-amd64                         
Build commit: 3417e78
chitra-raghavan commented 7 years ago

@prsunny , @stcheng I am using the latest version of the label and seeing the same issue. 

root@sonic:/var/log# show ver
SONiC Software Version: SONiC.HEAD.368-3417e78
Distribution: Debian 8.9
Kernel: 3.16.0-4-amd64
Build commit: 3417e78
Build date: Mon Oct 23 12:58:23 UTC 2017
Built by: johnar@jenkins-worker-3

Docker images:
REPOSITORY                TAG                 IMAGE ID            SIZE
docker-syncd-brcm         HEAD.368-3417e78    fbe33656cfd3        318.3 MB
docker-syncd-brcm         latest              fbe33656cfd3        318.3 MB
docker-orchagent-brcm     HEAD.368-3417e78    06b3e355a5b9        259 MB
docker-orchagent-brcm     latest              06b3e355a5b9        259 MB
docker-lldp-sv2           HEAD.368-3417e78    e532679f9de4        256.5 MB
docker-lldp-sv2           latest              e532679f9de4        256.5 MB
docker-dhcp-relay         HEAD.368-3417e78    75c8ace8196a        253.5 MB
docker-dhcp-relay         latest              75c8ace8196a        253.5 MB
docker-database           HEAD.368-3417e78    14320c14aef1        251.7 MB
docker-database           latest              14320c14aef1        251.7 MB
docker-snmp-sv2           HEAD.368-3417e78    9ac3dd770004        291.3 MB
docker-snmp-sv2           latest              9ac3dd770004        291.3 MB
docker-teamd              HEAD.368-3417e78    c8a66e1bb150        256.1 MB
docker-teamd              latest              c8a66e1bb150        256.1 MB
docker-platform-monitor   HEAD.368-3417e78    7b130159ec57        271 MB
docker-platform-monitor   latest              7b130159ec57        271 MB
docker-fpm-quagga         HEAD.368-3417e78    80774276af0d        262.6 MB
docker-fpm-quagga         latest              80774276af0d        262.6 MB

I see the below messages in syslog file.

Oct 24 06:35:20.505886 sonic-s6100-01 INFO supervisord: snmp-subagent Traceback (most recent call last):
Oct 24 06:35:20.505970 sonic-s6100-01 INFO supervisord: snmp-subagent   File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
Oct 24 06:35:20.506004 sonic-s6100-01 INFO supervisord: snmp-subagent     "__main__", mod_spec)
Oct 24 06:35:20.506038 sonic-s6100-01 INFO supervisord: snmp-subagent   File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
Oct 24 06:35:20.506073 sonic-s6100-01 INFO supervisord: snmp-subagent     exec(code, run_globals)
Oct 24 06:35:20.506107 sonic-s6100-01 INFO supervisord: snmp-subagent   File "/usr/local/lib/python3.6/dist-packages/sonic_ax_impl/__main__.py", line 74, in <module>
Oct 24 06:35:20.506157 sonic-s6100-01 INFO supervisord: snmp-subagent     from .main import main
Oct 24 06:35:20.506190 sonic-s6100-01 INFO supervisord: snmp-subagent   File "/usr/local/lib/python3.6/dist-packages/sonic_ax_impl/main.py", line 12, in <module>
Oct 24 06:35:20.506223 sonic-s6100-01 INFO supervisord: snmp-subagent     from sonic_ax_impl.mibs import ieee802_1ab
Oct 24 06:35:20.506257 sonic-s6100-01 INFO supervisord: snmp-subagent   File "/usr/local/lib/python3.6/dist-packages/sonic_ax_impl/mibs/ieee802_1ab.py", line 122, in <module>
Oct 24 06:35:20.506290 sonic-s6100-01 INFO supervisord: snmp-subagent     _lldp_updater = LLDPUpdater()
Oct 24 06:35:20.506323 sonic-s6100-01 INFO supervisord: snmp-subagent   File "/usr/local/lib/python3.6/dist-packages/sonic_ax_impl/mibs/ieee802_1ab.py", line 34, in __init__
Oct 24 06:35:20.506374 sonic-s6100-01 INFO supervisord: snmp-subagent     self.reinit_data()
Oct 24 06:35:20.506406 sonic-s6100-01 INFO supervisord: snmp-subagent   File "/usr/local/lib/python3.6/dist-packages/sonic_ax_impl/mibs/ieee802_1ab.py", line 50, in reinit_data
Oct 24 06:35:20.506440 sonic-s6100-01 INFO supervisord: snmp-subagent     self.oid_name_map = mibs.init_sync_d_interface_tables(self.db_conn)
Oct 24 06:35:20.506473 sonic-s6100-01 INFO supervisord: snmp-subagent   File "/usr/local/lib/python3.6/dist-packages/sonic_ax_impl/mibs/__init__.py", line 77, in init_sync_d_interface_tables
Oct 24 06:35:20.506524 sonic-s6100-01 INFO supervisord: snmp-subagent     if_name_map, if_id_map = port_util.get_interface_oid_map(db_conn)
Oct 24 06:35:20.506557 sonic-s6100-01 INFO supervisord: snmp-subagent   File "/usr/local/lib/python3.6/dist-packages/swsssdk/port_util.py", line 40, in get_interface_oid_map
Oct 24 06:35:20.506609 sonic-s6100-01 INFO supervisord: snmp-subagent     if_name_map = db.get_all('COUNTERS_DB', 'COUNTERS_PORT_NAME_MAP', blocking=True)
Oct 24 06:35:20.506643 sonic-s6100-01 INFO supervisord: snmp-subagent   File "/usr/local/lib/python3.6/dist-packages/swsssdk/interface.py", line 38, in wrapped
Oct 24 06:35:20.506676 sonic-s6100-01 INFO supervisord: snmp-subagent     ret_data = f(inst, db_name, *args, **kwargs)
Oct 24 06:35:20.506709 sonic-s6100-01 INFO supervisord: snmp-subagent   File "/usr/local/lib/python3.6/dist-packages/swsssdk/interface.py", line 303, in get_all
Oct 24 06:35:20.506742 sonic-s6100-01 INFO supervisord: snmp-subagent     raise UnavailableDataError(message, _hash)
Oct 24 06:35:20.506775 sonic-s6100-01 INFO supervisord: snmp-subagent swsssdk.exceptions.UnavailableDataError: Key 'COUNTERS_PORT_NAME_MAP' unavailable in database 'COUNTERS_DB'
Oct 24 06:35:20.552338 sonic-s6100-01 INFO snmp.sh[9294]: 2017-10-24 06:35:20,551 INFO exited: snmp-subagent (exit status 1; not expected)
Oct 24 06:35:29.123978 sonic-s6100-01 INFO supervisord 2017-10-24 06:35:20,551 INFO exited: snmp-subagent (exit status 1; not expected)
prsunny commented 7 years ago

Looks like it is due the following pull-requests.

https://github.com/Azure/sonic-sairedis/pull/234 https://github.com/Azure/sonic-swss/pull/331

We are tracking this!

stcheng commented 7 years ago

we've fixed this. please confirm.

chitra-raghavan commented 7 years ago

seeing same issue in build 376. snmpwalk doesnt return any output. version : 

root@sonic-s6100-01:~# show ver
SONiC Software Version: SONiC.HEAD.376-d7d6c41
Distribution: Debian 8.9
Kernel: 3.16.0-4-amd64
Build commit: d7d6c41
Build date: Tue Oct 31 13:13:47 UTC 2017
Built by: johnar@jenkins-worker-3

Docker images:
REPOSITORY                TAG                 IMAGE ID            SIZE
docker-syncd-brcm         HEAD.376-d7d6c41    c9c0917e1f09        318.3 MB
docker-syncd-brcm         latest              c9c0917e1f09        318.3 MB
docker-orchagent-brcm     HEAD.376-d7d6c41    d1040046a4c5        259.1 MB
docker-orchagent-brcm     latest              d1040046a4c5        259.1 MB
docker-lldp-sv2           HEAD.376-d7d6c41    4bf6be6d4f13        256.5 MB
docker-lldp-sv2           latest              4bf6be6d4f13        256.5 MB
docker-dhcp-relay         HEAD.376-d7d6c41    99e930f82d84        253.5 MB
docker-dhcp-relay         latest              99e930f82d84        253.5 MB
docker-database           HEAD.376-d7d6c41    72344548f653        251.7 MB
docker-database           latest              72344548f653        251.7 MB
docker-snmp-sv2           HEAD.376-d7d6c41    b0f9c9a36e9a        291.3 MB
docker-snmp-sv2           latest              b0f9c9a36e9a        291.3 MB
docker-teamd              HEAD.376-d7d6c41    c74659900c4d        256.1 MB
docker-teamd              latest              c74659900c4d        256.1 MB
docker-platform-monitor   HEAD.376-d7d6c41    58881d29a46e        271 MB
docker-platform-monitor   latest              58881d29a46e        271 MB
docker-fpm-quagga         HEAD.376-d7d6c41    19b478542bc7        262.7 MB
docker-fpm-quagga         latest              19b478542bc7        262.7 MB

snmp o/p :
version: 

rchitra@login-sjc-01[1:57am] : ~ > snmpwalk -v 2c -c public 10.11.150.100 1.3.6.1.2.1.1.1.0
SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.HEAD.376-d7d6c41 - HwSku: Force10-S6100 - Distribution: Debian 8.9 - Kernel: 3.16.0-4-amd64

IF-MIB : 

rchitra@login-sjc-01[1:57am] : ~ > snmpwalk -v 2c -c public 10.11.150.100 1.3.6.1.2.1.2.2.1.1
IF-MIB::ifIndex = No Such Object available on this agent at this OID

rchitra@login-sjc-01[1:57am] : ~ > snmpwalk -v 2c -c public 10.11.150.100 1.3.6.1.2.1.2
IF-MIB::interfaces = No Such Object available on this agent at this OID

rchitra@login-sjc-01[1:57am] : ~ > snmpwalk -v 2c -c public 10.11.150.100 1.3.6.1.2.1.31
IF-MIB::ifMIB = No Such Object available on this agent at this OID

IP-MIB:

rchitra@login-sjc-01[1:57am] : ~ > snmpwalk -v 2c -c public 10.11.150.100 1.3.6.1.2.1.4.20
IP-MIB::ipAddrTable = No Such Object available on this agent at this OID

rchitra@login-sjc-01[1:58am] : ~ >
qiluo-msft commented 7 years ago

@chitra-raghavan The syslog you pasted on Oct 23 is saying the 'COUNTERS_PORT_NAME_MAP' is not available in redis-db. You may double check by command.

redis-cli -n 2 -c hgetall COUNTERS_PORT_NAME_MAP

Normally it is due to very slow initialization on this platform. We are trying to solve this problem (@yxieca ). Please confirm if it is the case. If yes, mitigating solution is to restart snmp docker:

sudo systemctl restart snmp
chitra-raghavan commented 6 years ago

After restarting the snmp docker , snmpwalk for IF-MIB works fine. (sudo systemctl restart snmp)

chitra-raghavan commented 6 years ago

snmpwalk for IP-MIB still fails.

rchitra@login-sjc-01[1:57am] : ~ > snmpwalk -v 2c -c public 10.11.150.100 1.3.6.1.2.1.4.20 IP-MIB::ipAddrTable = No Such Object available on this agent at this OID

qiluo-msft commented 6 years ago

My original comment still applies if you met the same error in syslog.

chitra-raghavan commented 6 years ago

Even after restarting the snmp , IP-MIB doesnt work. IF_MIB alone works fine after restarting the snmp

qiluo-msft commented 6 years ago

@chitra-raghavan Sorry, I thought you were talking about IF-MIB. I double checked the code and lab device, that current snmpagent implemented partial IP-MIB. If you query "1.3.6.1.2.1.4", you will have some idea.

If you want to check code, you may read: class IpMib, class IpCidrRouteTable.

What do you really need in IP-MIB? Feel free to submit a pull request if you would like to contribute.

chitra-raghavan commented 6 years ago

it is fine if the oid polled in the sonic-mgmt script is enough for now

rraghav-cisco commented 4 years ago

Any resolution on this ? The latest sonic builds are still seeing this problem. On 2 different vendor platforms. Thanks, rraghav

scottlaird commented 4 years ago

I'm also seeing this, on a fresh install of build 306 from Jenkins (a day or two old), on an as7326_56x (Broadcom). snmpwalk shows IP-MIB but not IF-MIB.

scottlaird commented 4 years ago

Found in syslog:

./syslog.1:Jun 10 17:57:29.089708 sonic ERR snmp#snmp-subagent [ax_interface] ERROR: MIBUpdater.start() caught an unexpected exception during update_data()#012Traceback (most recent call last):#012 File "/usr/local/lib/python3.7/dist-packages/ax_interface/mib.py", line 43, in start#012 self.update_data()#012 File "/usr/local/lib/python3.7/dist-packages/sonic_ax_impl/mibs/ietf/rfc2863.py", line 98, in update_data#012 for sai_id in self.if_id_map}#012 File "/usr/local/lib/python3.7/dist-packages/sonic_ax_impl/mibs/ietf/rfc2863.py", line 98, in <dictcomp>#012 for sai_id in self.if_id_map}#012 File "/usr/local/lib/python3.7/dist-packages/swsssdk/dbconnector.py", line 308, in get_all#012 return super(SonicV2Connector, self).get_all(db_id, _hash, *args, **kwargs)#012 File "/usr/local/lib/python3.7/dist-packages/swsssdk/interface.py", line 38, in wrapped#012 ret_data = f(inst, db_id, *args, **kwargs)#012 File "/usr/local/lib/python3.7/dist-packages/swsssdk/interface.py", line 303, in get_all#012 raise UnavailableDataError(message, _hash)#012swsssdk.exceptions.UnavailableDataError: Key 'b'COUNTERS:oid:0x1000000000009'' unavailable in database '2'

I'm not that familiar with Redis, but it doesn't look like that key exists:


$ redis-cli -n 2 -c keys COUNTERS:oid:0x10*
1) "COUNTERS:oid:0x1000000000002"
2) "COUNTERS:oid:0x1000000000006"
3) "COUNTERS:oid:0x1000000000007"
4) "COUNTERS:oid:0x1000000000008"
5) "COUNTERS:oid:0x1000000000003"
6) "COUNTERS:oid:0x1000000000005"
7) "COUNTERS:oid:0x1000000000004"```
scottlaird commented 4 years ago

FWIW, after upgrading to build 310 and rebooting, IF-MIB entries now appear.