SNMP not working for any device on network, nothing blocking access and credentials correct.

Using Netdisco's latest version on an EC2 instance. All traffic is permitted to/from this instance thru port 161, 162, 22, and 5000. Netdisco is running as expected, not showing any errors in logs, and I can connect to the web interface just fine.

Attempting command ~netdisco/bin/netdisco-do discover -d 10.39.25.253 -DIQ where 10.39.25.253 is a Cisco Switch that the instance can successfully ping and SSH into. The EC2 instance's IP address has been listed as an SNMP collector on the switch. When running the command, I receive the following output: (It is worth noting that it hangs for about 10s before timing out, and if any auth is incorrect it immediately times out)

[netdisco@uvaadmmapp01ndc logs]$ ~netdisco/bin/netdisco-do discover -d 10.39.25.253 -DIQ [13238] 2023-02-13 18:26:44 info App::Netdisco version 2.060003 loaded. SELECT me.version, me.installed FROM dbix_class_schema_versions me WHERE 1 = 0 SELECT me.version FROM dbix_class_schema_versions me ORDER BY installed DESC LIMIT '1' SELECT me.ip, me.alias, me.subnet, me.port, me.dns, me.creation FROM device_ip me WHERE me.alias = '10.39.25.253' AND me.ip = '10.39.25.253' SELECT me.ip, me.alias, me.subnet, me.port, me.dns, me.creation FROM device_ip me WHERE alias = '10.39.25.253' SELECT me.ip, me.creation, me.dns, me.description, me.uptime, me.contact, me.name, me.location, me.layers, me.num_ports, me.mac, me.serial, me.chassis_id, me.model, me.ps1_type, me.ps2_type, me.ps1_status, me.ps2_status, me.fan, me.slots, me.vendor, me.os, me.os_ver, me.log, me.snmp_ver, me.snmp_comm, me.snmp_class, me.snmp_engineid, me.vtp_domain, me.last_discover, me.last_macsuck, me.last_arpnip, me.is_pseudo, me.pae_is_enabled, me.custom_fields, to_char( me.creation, 'YYYY-MM-DD HH24:MI' ), to_char( me.last_arpnip, 'YYYY-MM-DD HH24:MI' ), to_char( me.last_discover, 'YYYY-MM-DD HH24:MI' ), to_char( me.last_macsuck, 'YYYY-MM-DD HH24:MI' ), extract( epoch FROM age( now( ), me.creation ) ), extract( epoch FROM age( now( ), me.last_arpnip ) ), extract( epoch FROM age( now( ), me.last_discover ) ), extract( epoch FROM age( now( ), me.last_macsuck ) ), replace( age( timestamp 'epoch' + me.uptime / 100 * interval '1 second', timestamp '1970-01-01 00:00:00-00' ) ::text, 'mon', 'month' ) FROM device me WHERE me.ip = '10.39.25.253' [13238] 2023-02-13 18:26:45 info discover: [10.39.25.253] started at Mon Feb 13 13:26:45 2023 [13238] 2023-02-13 18:26:45 debug discover: running with timeout 600s [13238] 2023-02-13 18:26:45 debug => running workers for phase: check [13238] 2023-02-13 18:26:45 debug -> run worker check/0 "discover" [13238] 2023-02-13 18:26:45 debug Discover is able to run. [13238] 2023-02-13 18:26:45 debug => running workers for phase: early [13238] 2023-02-13 18:26:45 debug -> run worker early/100 "discover::properties" [13238] 2023-02-13 18:26:45 debug snmp reader cache warm: [10.39.25.253] SELECT me.ip, me.snmp_comm_rw, me.snmp_auth_tag_read, me.snmp_auth_tag_write FROM community me WHERE me.ip = '10.39.25.253' SELECT me.ip, me.snmp_comm_rw, me.snmp_auth_tag_read, me.snmp_auth_tag_write FROM community me WHERE me.ip = '10.39.25.253' [13238] 2023-02-13 18:26:45 debug [10.39.25.253:161] try_connect with ver: 3, class: SNMP::Info, comm: SNMP::Info::_global uptime : DISMAN-EVENT-MIB::sysUpTimeInstance : .1.3.6.1.2.1.1.3.0 SNMP::Info::_global(uptime) Timeout at /home/netdisco/perl5/lib/perl5/App/Netdisco/Transport/SNMP.pm line 263. SNMP::Info::_global hrSystemUptime : HOST-RESOURCES-MIB::hrSystemUptime.0 : .1.3.6.1.2.1.25.1.1.0 SNMP::Info::_global(hrSystemUptime) Timeout at /home/netdisco/perl5/lib/perl5/App/Netdisco/Transport/SNMP.pm line 263. SNMP::Info::_global sysUpTime : DISMAN-EVENT-MIB::sysUpTimeInstance : .1.3.6.1.2.1.1.3.0 SNMP::Info::_global(sysUpTime) Timeout at /home/netdisco/perl5/lib/perl5/App/Netdisco/Transport/SNMP.pm line 263. [13238] 2023-02-13 18:27:12 debug discover failed: could not SNMP connect to 10.39.25.253 [13238] 2023-02-13 18:27:12 debug -> run worker early/100 "discover::properties" [13238] 2023-02-13 18:27:12 debug -> run worker early/100 "discover::properties" [13238] 2023-02-13 18:27:12 debug -> run worker early/100 "discover::properties" [13238] 2023-02-13 18:27:12 debug -> run worker early/100 "discover::properties" [13238] 2023-02-13 18:27:12 debug => running workers for phase: main [13238] 2023-02-13 18:27:12 debug -> run worker main/100 "discover::canonicalip" [13238] 2023-02-13 18:27:12 debug -> run worker main/100 "discover::entities" [13238] 2023-02-13 18:27:12 debug -> run worker main/100 "discover::neighbors" [13238] 2023-02-13 18:27:12 debug -> run worker main/100 "discover::neighbors::docsis" [13238] 2023-02-13 18:27:12 debug -> run worker main/100 "discover::neighbors::routed" [13238] 2023-02-13 18:27:12 debug -> run worker main/100 "discover::portpower" [13238] 2023-02-13 18:27:12 debug -> run worker main/100 "discover::portproperties" [13238] 2023-02-13 18:27:12 debug -> run worker main/100 "discover::portproperties::portaccessentity" [13238] 2023-02-13 18:27:12 debug pae failed: could not SNMP connect to 10.39.25.253 [13238] 2023-02-13 18:27:12 debug -> run worker main/100 "discover::vlans" [13238] 2023-02-13 18:27:12 debug -> run worker main/100 "discover::wireless" [13238] 2023-02-13 18:27:12 debug -> run worker main/0 "discover::withnodes" [13238] 2023-02-13 18:27:12 debug => running workers for phase: late [13238] 2023-02-13 18:27:12 debug -> run worker late/0 "discover::hooks" [13238] 2023-02-13 18:27:12 debug [10.39.25.253] hooks - 0 queued [13238] 2023-02-13 18:27:12 info discover: finished at Mon Feb 13 13:27:12 2023 [13238] 2023-02-13 18:27:12 info discover: status defer: discover failed: could not SNMP connect to 10.39.25.253

The switch is using SNMPv3 and auth should be configured correctly in deployment.yml, as follows:

device_auth:

tag: snmpv3_sw user: myUser auth: pass: myAuthPass proto: SHA priv: pass: myPrivPass proto: AES256

This issue recurs when connecting to ANY network device.

Expected Behavior

SNMP Connection successful and netdisco scans

Possible Solution

I have tried configuring the device as a psuedo device to workaround the SNMP issue without success, being told it does not have an offline cache.

interesting... wanted to update our docs for snmpv3 and came across this.

let's cover the basics first. going on the output & description i don't see an issue, but let's check these just to be sure.

you say

This issue recurs when connecting to ANY network device.

for clarification: does this happen on different vendors / devices / os versions?
what does netdisco-do show -d 10.39.25.253 -e specify say?

[!CAUTION] if you paste that here be aware that it contains cleartext credentials , you might want to remove those first.

to verify, when you say

All traffic is permitted to/from this instance thru port 161, 162, 22, and 5000

does this includes udp/161 (not just tcp)? i'm guessing it does since you mention that incorrect credentials get rejected immediately.
the example device_auth you pasted, can you paste that again but using triple backticks example here, formatting is relevant for the config.

you do mention it's a cisco device & you are using aes-256. i don't have that specific config available for testing atm. i did run into this however: https://github.com/netdisco/netdisco/issues/962#issuecomment-1379001329 . tldr: try changing aes256 to aes256c.

dunno what access you have on that aws instance, but if you have cli access testing snmpwalk could also give insights (used values from your example): snmpwalk -v 3 -x AES256 -X myPrivPass -a SHA -A myAuthPass -u myUser -l authPriv 10.39.25.253 and snmpwalk -v 3 -x AES256C -X myPrivPass -a SHA -A myAuthPass -u myUser -l authPriv 10.39.25.253

if aes256 and/or aes256c aren't supported you will get the help blurb, if the device rejects it for whatever reason you get snmpwalk: Authentication failure (incorrect password, community or key), if it works - well - you get actual info :)

I did manage to resolve this, it was a few months ago but if I recall correctly the issue was that netdisco wasn't properly using snmpwalk or that snmpwalk wasn't working with snmpv3. I wound up reinstalling the library snmpwalk is from and installing a different version then reconfiguring that for snmpv3 and it wound up working.

On Sat, Feb 10, 2024, 7:10 PM nick n. @.***> wrote:

interesting... wanted to update our docs for snmpv3 and came across this.

let's cover the basics first. going on the output & description i don't see an issue, but let's check these just to be sure.

you say

This issue recurs when connecting to ANY network device.

1.

for clarification: does this happen on different vendors / devices / os versions? 2.

what does netdisco-do show -d 10.39.25.253 -e specify say?

Caution

if you paste that here be aware that it contains cleartext credentials , you might want to remove those first.

to verify, when you say

All traffic is permitted to/from this instance thru port 161, 162, 22, and 5000

1.

does this includes udp/161 (not just tcp)? i'm guessing it does since you mention that incorrect credentials get rejected immediately. 2.

the example device_auth you pasted, can you paste that again but using triple backticks example here https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax#quoting-code, formatting is relevant for the config.

you do mention it's a cisco device & you are using aes-256. i don't have that specific config available for testing atm. i did run into this however: netdisco/netdisco#962 (comment) https://github.com/netdisco/netdisco/issues/962#issuecomment-1379001329 . tldr: try changing aes256 to aes256c.

dunno what access you have on that aws instance, but if you have cli access testing snmpwalk could also give insights (used values from your example): snmpwalk -v 3 -x AES256 -X myPrivPass -a SHA -A myAuthPass -u myUser -l authPriv 10.39.25.253 and snmpwalk -v 3 -x AES256C -X myPrivPass -a SHA -A myAuthPass -u myUser -l authPriv 10.39.25.253

if aes256 and/or aes256c aren't supported you will get the help blurb, if the device rejects it for whatever reason you get snmpwalk: Authentication failure (incorrect password, community or key), if it works - well - you get actual info :)

— Reply to this email directly, view it on GitHub https://github.com/netdisco/snmp-info/issues/480#issuecomment-1937363709, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALGG5ZOCCPJPYGPBPJFV2A3YTAEBBAVCNFSM6AAAAAAU2TQA4WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZXGM3DGNZQHE . You are receiving this because you authored the thread.Message ID: @.***>

the issue was that netdisco wasn't properly using snmpwalk or that snmpwalk wasn't working with snmpv3

yeah, i was starting to update the snmpv3 documentation for snmp::info & netdisco for newer hashing algo's and quickly ran into all the bits that need to be in sync to make it work. troubleshooting my own setup wasn't that hard, but writing a generalized guide is a challenge.

do you perhaps recall the steps you took to fix it?

what i'm most curious about is the net-snmp version. specifically: was it 5.9.1 or newer, or was it an older version?

I believe the issue was related specifically to AES256. I used ChatGPT to recreate the issue and troubleshoot so you can find all the info here:

https://chat.openai.com/share/d8686d8d-1ebe-4e23-97a5-dac7d384f8af

Hopefully that's helpful and let me know if you have any other questions!

On Sat, Feb 10, 2024, 8:42 PM nick n. @.***> wrote:

the issue was that netdisco wasn't properly using snmpwalk or that snmpwalk wasn't working with snmpv3

yeah, i was starting to update the snmpv3 documentation for snmp::info & netdisco for newer hashing algo's and quickly ran into all the bits that need to be in sync to make it work. troubleshooting my own setup wasn't that hard, but writing a generalized guide is a challenge.

do you perhaps recall the steps you took to fix it?

what i'm most curious about is the net-snmp version. specifically: was it 5.9.1 or newer, or was it an older version?

— Reply to this email directly, view it on GitHub https://github.com/netdisco/snmp-info/issues/480#issuecomment-1937387097, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALGG5ZLJAA6G5JL3QZDNSVTYTAOWRAVCNFSM6AAAAAAU2TQA4WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZXGM4DOMBZG4 . You are receiving this because you authored the thread.Message ID: @.***>

i've opened netdisco/snmp-info#515 based on your troubleshooting workflow and my own observations.

it summarizes everything i think needs better documentation/handling. if there's something you think that should still be added just make a note in that ticket.

(i did already update the netdisco wiki with the basics.)

netdisco / snmp-info

SNMP not working for any device on network, nothing blocking access and credentials correct. #480

Expected Behavior

Possible Solution