search

prometheus / snmp_exporter

SNMP Exporter for Prometheus
Apache License 2.0
1.61k stars 605 forks source link

Error scraping target when happend "endOfMibView" #617

Open bubbleatgit opened 3 years ago

bubbleatgit commented 3 years ago

Host operating system: output of uname -a

             uptime: 1w15h1m53s
            version: 6.47.9 (long-term)
         build-time: Feb/08/2021 12:48:33
   factory-software: 6.43.10
        free-memory: 953.9MiB
       total-memory: 1024.0MiB
                cpu: ARMv7
          cpu-count: 4
      cpu-frequency: 1400MHz
           cpu-load: 2%
     free-hdd-space: 418.2MiB
    total-hdd-space: 512.3MiB
  architecture-name: arm
         board-name: RB4011iGS+
           platform: MikroTik

snmp_exporter version: output of snmp_exporter -version

msg="Starting snmp_exporter" version="(version=0.20.0, branch=HEAD, revision=c33572b6c8c8e43a479fde0f9fa8ac86e15598bc)"

What device/snmpwalk OID are you using?

 .1.3.6.1.4.1.14988

What did you do that produced an error?

level=debug ts=2021-02-23T08:58:59.373Z caller=collector.go:131 module=mikrotik target=172.20.31.254 msg="Getting OIDs" oids=2
level=debug ts=2021-02-23T08:58:59.375Z caller=collector.go:140 module=mikrotik target=172.20.31.254 msg="Get of OIDs completed" oids=2 duration_seconds=2.042977ms
level=debug ts=2021-02-23T08:58:59.375Z caller=collector.go:164 module=mikrotik target=172.20.31.254 msg="Walking subtree" oid=1.3.6.1.2.1.2
level=debug ts=2021-02-23T08:58:59.419Z caller=collector.go:177 module=mikrotik target=172.20.31.254 msg="Walk of subtree completed" oid=1.3.6.1.2.1.2 duration_seconds=43.644082ms
level=debug ts=2021-02-23T08:58:59.419Z caller=collector.go:164 module=mikrotik target=172.20.31.254 msg="Walking subtree" oid=1.3.6.1.2.1.25
level=debug ts=2021-02-23T08:58:59.425Z caller=collector.go:177 module=mikrotik target=172.20.31.254 msg="Walk of subtree completed" oid=1.3.6.1.2.1.25 duration_seconds=5.999824ms
level=debug ts=2021-02-23T08:58:59.425Z caller=collector.go:164 module=mikrotik target=172.20.31.254 msg="Walking subtree" oid=1.3.6.1.2.1.31
level=debug ts=2021-02-23T08:58:59.469Z caller=collector.go:177 module=mikrotik target=172.20.31.254 msg="Walk of subtree completed" oid=1.3.6.1.2.1.31 duration_seconds=44.123004ms
level=debug ts=2021-02-23T08:58:59.469Z caller=collector.go:164 module=mikrotik target=172.20.31.254 msg="Walking subtree" oid=1.3.6.1.4.1.14988
level=info ts=2021-02-23T08:59:09.489Z caller=collector.go:224 module=mikrotik target=172.20.31.254 msg="Error scraping target" err="scrape canceled (possible timeout) walking target 172.20.31.254"
level=debug ts=2021-02-23T08:59:09.489Z caller=main.go:113 module=mikrotik target=172.20.31.254 msg="Finished scrape" duration_seconds=10.117271226

What did you see instead?

when use snmpwalk test the OIDs，get this result:

iso.3.6.1.4.1.14988.1.1.17.1.1.4.1 = STRING: "RouterOS v6.47.9 Feb/08/2021 12:48:33"
iso.3.6.1.4.1.14988.1.1.17.1.1.5.1 = INTEGER: 1
iso.3.6.1.4.1.14988.1.1.17.1.1.6.1 = INTEGER: 1
iso.3.6.1.4.1.14988.1.1.19.1.1.2.40 = STRING: "sfp-sfpplus1"
iso.3.6.1.4.1.14988.1.1.19.1.1.3.40 = INTEGER: 0
iso.3.6.1.4.1.14988.1.1.19.1.1.4.40 = INTEGER: 0
iso.3.6.1.4.1.14988.1.1.19.1.1.5.40 = Gauge32: 85000
iso.3.6.1.4.1.14988.1.1.19.1.1.6.40 = Gauge32: 50
iso.3.6.1.4.1.14988.1.1.19.1.1.7.40 = Gauge32: 3289
iso.3.6.1.4.1.14988.1.1.19.1.1.8.40 = Gauge32: 8
iso.3.6.1.4.1.14988.1.1.19.1.1.9.40 = INTEGER: -2633
iso.3.6.1.4.1.14988.1.1.19.1.1.10.40 = INTEGER: -2586
iso.3.6.1.4.1.14988.1.1.19.1.1.10.40 = No more variables left in this MIB View (It is past the end of the MIB tree)

If annotation this index, this will be ok:

mikrotik:
  walk:
  - 1.3.6.1.2.1.2
  - 1.3.6.1.2.1.25
  - 1.3.6.1.2.1.31
  #- 1.3.6.1.4.1.14988
  - 1.3.6.1.4.1.2021.10.1.1
  - 1.3.6.1.4.1.2021.10.1.2
  version: 3
carlosedp commented 2 years ago

I recently started to get errors on 14988 OID in my Mikrotik where it produced timeouts. I manually edited snmp.yml adding version: 1 to the mikrotik config and it worked (it uses get next instead of get bulk).

Then I checked with snmpwalk and found that the culprit was an ascii art I was playing with in the "notes" section of Mikrotik. Once I removed it, it started fetching SNMP OID 14988 again using default v2 (bulk)

candlerb commented 3 months ago

I have a similar issue here, possibly the same one. snmp_exporter 0.25.0, Mikrotik running 7.15alpha255, using the out-of-box snmp.yml and its mikrotik mib.

The first thing I noticed is that even though the whole mikrotik MIB is being walked (1.3.6.1.4.1.14988), only the interface stats are being returned by snmp_exporter:

# curl -fsS 'localhost:9116/snmp?target=gw1&module=mikrotik&auth=workshop_v3' | grep -v '^#' | sed 's/[{ ].*$//' | uniq -c
     14 mtxrInterfaceStatsDriverRxBytes
     14 mtxrInterfaceStatsDriverRxPackets
     14 mtxrInterfaceStatsDriverTxBytes
     14 mtxrInterfaceStatsDriverTxPackets
     14 mtxrInterfaceStatsIndex
     14 mtxrInterfaceStatsLinkDowns
     14 mtxrInterfaceStatsName
     14 mtxrInterfaceStatsRx1024To1518
     14 mtxrInterfaceStatsRx128To255
     14 mtxrInterfaceStatsRx1519ToMax
     14 mtxrInterfaceStatsRx256To511
     14 mtxrInterfaceStatsRx512To1023
     14 mtxrInterfaceStatsRx64
     14 mtxrInterfaceStatsRx65To127
     14 mtxrInterfaceStatsRxAlignError
     14 mtxrInterfaceStatsRxBroadcast
     14 mtxrInterfaceStatsRxBytes
     14 mtxrInterfaceStatsRxCarrierError
     14 mtxrInterfaceStatsRxCodeError
     14 mtxrInterfaceStatsRxControl
     14 mtxrInterfaceStatsRxDrop
     14 mtxrInterfaceStatsRxFCSError
     14 mtxrInterfaceStatsRxFragment
     14 mtxrInterfaceStatsRxJabber
     14 mtxrInterfaceStatsRxLengthError
     14 mtxrInterfaceStatsRxMulticast
     14 mtxrInterfaceStatsRxOverflow
     14 mtxrInterfaceStatsRxPackets
     14 mtxrInterfaceStatsRxPause
     14 mtxrInterfaceStatsRxTooLong
     14 mtxrInterfaceStatsRxTooShort
     14 mtxrInterfaceStatsRxUnknownOp
     14 mtxrInterfaceStatsTx1024To1518
     14 mtxrInterfaceStatsTx128To255
     14 mtxrInterfaceStatsTx1519ToMax
     14 mtxrInterfaceStatsTx256To511
     14 mtxrInterfaceStatsTx512To1023
     14 mtxrInterfaceStatsTx64
     14 mtxrInterfaceStatsTx65To127
     14 mtxrInterfaceStatsTxBroadcast
     14 mtxrInterfaceStatsTxBytes
     14 mtxrInterfaceStatsTxCollision
     14 mtxrInterfaceStatsTxControl
     14 mtxrInterfaceStatsTxDeferred
     14 mtxrInterfaceStatsTxDrop
     14 mtxrInterfaceStatsTxExcessiveCollision
     14 mtxrInterfaceStatsTxExcessiveDeferred
     14 mtxrInterfaceStatsTxFCSError
     14 mtxrInterfaceStatsTxFragment
     14 mtxrInterfaceStatsTxJabber
     14 mtxrInterfaceStatsTxLateCollision
     14 mtxrInterfaceStatsTxMulticast
     14 mtxrInterfaceStatsTxMultipleCollision
     14 mtxrInterfaceStatsTxPackets
     14 mtxrInterfaceStatsTxPause
     14 mtxrInterfaceStatsTxPauseHonored
     14 mtxrInterfaceStatsTxRx1024To1518
     14 mtxrInterfaceStatsTxRx128To255
     14 mtxrInterfaceStatsTxRx1519ToMax
     14 mtxrInterfaceStatsTxRx256To511
     14 mtxrInterfaceStatsTxRx512To1023
     14 mtxrInterfaceStatsTxRx64
     14 mtxrInterfaceStatsTxRx65To127
     14 mtxrInterfaceStatsTxSingleCollision
     14 mtxrInterfaceStatsTxTooLong
     14 mtxrInterfaceStatsTxTooShort
     14 mtxrInterfaceStatsTxTotalCollision
     14 mtxrInterfaceStatsTxUnderrun
      1 snmp_scrape_duration_seconds
      1 snmp_scrape_packets_retried
      1 snmp_scrape_packets_sent
      1 snmp_scrape_pdus_returned
      1 snmp_scrape_walk_duration_seconds
      1 sysDescr

Compare to snmpbulkwalk, where a bunch of additional info is being returned such as POE status:

# snmpbulkwalk -m+MIKROTIK-MIB 10.12.250.2 1.3.6.1.4.1.14988 | cut -f1 -d. | uniq -c
      1 MIKROTIK-MIB::mtxrWlRtabEntryCount
      1 MIKROTIK-MIB::mtxrWlCMRtabEntryCount
      1 MIKROTIK-MIB::mtxrWlCMREntryCount
      1 MIKROTIK-MIB::mtxrHlVoltage
      1 MIKROTIK-MIB::mtxrHlActiveFan
      1 MIKROTIK-MIB::mtxrHlTemperature
      1 MIKROTIK-MIB::mtxrHlProcessorFrequency
      2 MIKROTIK-MIB::mtxrGaugeName
      2 MIKROTIK-MIB::mtxrGaugeValue
      2 MIKROTIK-MIB::mtxrGaugeUnit
      1 MIKROTIK-MIB::mtxrLicSoftwareId
      1 MIKROTIK-MIB::mtxrLicUpgrUntil
      1 MIKROTIK-MIB::mtxrLicLevel
      1 MIKROTIK-MIB::mtxrLicVersion
      1 MIKROTIK-MIB::mtxrLicUpgradableTo
      1 MIKROTIK-MIB::mtxrDHCPLeaseCount
      1 MIKROTIK-MIB::mtxrSystemReboot
      1 MIKROTIK-MIB::mtxrUSBPowerReset
      1 MIKROTIK-MIB::mtxrSerialNumber
      1 MIKROTIK-MIB::mtxrFirmwareVersion
      1 MIKROTIK-MIB::mtxrNote
      1 MIKROTIK-MIB::mtxrBuildTime
      1 MIKROTIK-MIB::mtxrFirmwareUpgradeVersion
      1 MIKROTIK-MIB::mtxrDisplayName
      1 MIKROTIK-MIB::mtxrBoardName
      1 MIKROTIK-MIB::mtxrNeighborIpAddress
      1 MIKROTIK-MIB::mtxrNeighborMacAddress
      1 MIKROTIK-MIB::mtxrNeighborVersion
      1 MIKROTIK-MIB::mtxrNeighborPlatform
      1 MIKROTIK-MIB::mtxrNeighborIdentity
      1 MIKROTIK-MIB::mtxrNeighborSoftwareID
      1 MIKROTIK-MIB::mtxrNeighborInterfaceID
      1 MIKROTIK-MIB::mtxrDate
      1 MIKROTIK-MIB::mtxrLongtitude
      1 MIKROTIK-MIB::mtxrLatitude
      1 MIKROTIK-MIB::mtxrAltitude
      1 MIKROTIK-MIB::mtxrSpeed
      1 MIKROTIK-MIB::mtxrSattelites
      1 MIKROTIK-MIB::mtxrValid
      1 MIKROTIK-MIB::mtxrWirelessModemSignalStrength
      1 MIKROTIK-MIB::mtxrWirelessModemSignalECIO
     14 MIKROTIK-MIB::mtxrInterfaceStatsIndex
     14 MIKROTIK-MIB::mtxrInterfaceStatsName
     14 MIKROTIK-MIB::mtxrInterfaceStatsDriverRxBytes
     14 MIKROTIK-MIB::mtxrInterfaceStatsDriverRxPackets
     14 MIKROTIK-MIB::mtxrInterfaceStatsDriverTxBytes
     14 MIKROTIK-MIB::mtxrInterfaceStatsDriverTxPackets
     14 MIKROTIK-MIB::mtxrInterfaceStatsTxRx64
     14 MIKROTIK-MIB::mtxrInterfaceStatsTxRx65To127
     14 MIKROTIK-MIB::mtxrInterfaceStatsTxRx128To255
     14 MIKROTIK-MIB::mtxrInterfaceStatsTxRx256To511
     14 MIKROTIK-MIB::mtxrInterfaceStatsTxRx512To1023
     14 MIKROTIK-MIB::mtxrInterfaceStatsTxRx1024To1518
     14 MIKROTIK-MIB::mtxrInterfaceStatsTxRx1519ToMax
     14 MIKROTIK-MIB::mtxrInterfaceStatsRxBytes
     14 MIKROTIK-MIB::mtxrInterfaceStatsRxPackets
     14 MIKROTIK-MIB::mtxrInterfaceStatsRxTooShort
     14 MIKROTIK-MIB::mtxrInterfaceStatsRx64
     14 MIKROTIK-MIB::mtxrInterfaceStatsRx65To127
     14 MIKROTIK-MIB::mtxrInterfaceStatsRx128To255
     14 MIKROTIK-MIB::mtxrInterfaceStatsRx256To511
     14 MIKROTIK-MIB::mtxrInterfaceStatsRx512To1023
     14 MIKROTIK-MIB::mtxrInterfaceStatsRx1024To1518
     14 MIKROTIK-MIB::mtxrInterfaceStatsRx1519ToMax
     14 MIKROTIK-MIB::mtxrInterfaceStatsRxTooLong
     14 MIKROTIK-MIB::mtxrInterfaceStatsRxBroadcast
     14 MIKROTIK-MIB::mtxrInterfaceStatsRxPause
     14 MIKROTIK-MIB::mtxrInterfaceStatsRxMulticast
     14 MIKROTIK-MIB::mtxrInterfaceStatsRxFCSError
     14 MIKROTIK-MIB::mtxrInterfaceStatsRxAlignError
     14 MIKROTIK-MIB::mtxrInterfaceStatsRxFragment
     14 MIKROTIK-MIB::mtxrInterfaceStatsRxOverflow
     14 MIKROTIK-MIB::mtxrInterfaceStatsRxControl
     14 MIKROTIK-MIB::mtxrInterfaceStatsRxUnknownOp
     14 MIKROTIK-MIB::mtxrInterfaceStatsRxLengthError
     14 MIKROTIK-MIB::mtxrInterfaceStatsRxCodeError
     14 MIKROTIK-MIB::mtxrInterfaceStatsRxCarrierError
     14 MIKROTIK-MIB::mtxrInterfaceStatsRxJabber
     14 MIKROTIK-MIB::mtxrInterfaceStatsRxDrop
     14 MIKROTIK-MIB::mtxrInterfaceStatsTxBytes
     14 MIKROTIK-MIB::mtxrInterfaceStatsTxPackets
     14 MIKROTIK-MIB::mtxrInterfaceStatsTxTooShort
     14 MIKROTIK-MIB::mtxrInterfaceStatsTx64
     14 MIKROTIK-MIB::mtxrInterfaceStatsTx65To127
     14 MIKROTIK-MIB::mtxrInterfaceStatsTx128To255
     14 MIKROTIK-MIB::mtxrInterfaceStatsTx256To511
     14 MIKROTIK-MIB::mtxrInterfaceStatsTx512To1023
     14 MIKROTIK-MIB::mtxrInterfaceStatsTx1024To1518
     14 MIKROTIK-MIB::mtxrInterfaceStatsTx1519ToMax
     14 MIKROTIK-MIB::mtxrInterfaceStatsTxTooLong
     14 MIKROTIK-MIB::mtxrInterfaceStatsTxBroadcast
     14 MIKROTIK-MIB::mtxrInterfaceStatsTxPause
     14 MIKROTIK-MIB::mtxrInterfaceStatsTxMulticast
     14 MIKROTIK-MIB::mtxrInterfaceStatsTxUnderrun
     14 MIKROTIK-MIB::mtxrInterfaceStatsTxCollision
     14 MIKROTIK-MIB::mtxrInterfaceStatsTxExcessiveCollision
     14 MIKROTIK-MIB::mtxrInterfaceStatsTxMultipleCollision
     14 MIKROTIK-MIB::mtxrInterfaceStatsTxSingleCollision
     14 MIKROTIK-MIB::mtxrInterfaceStatsTxExcessiveDeferred
     14 MIKROTIK-MIB::mtxrInterfaceStatsTxDeferred
     14 MIKROTIK-MIB::mtxrInterfaceStatsTxLateCollision
     14 MIKROTIK-MIB::mtxrInterfaceStatsTxTotalCollision
     14 MIKROTIK-MIB::mtxrInterfaceStatsTxPauseHonored
     14 MIKROTIK-MIB::mtxrInterfaceStatsTxDrop
     14 MIKROTIK-MIB::mtxrInterfaceStatsTxJabber
     14 MIKROTIK-MIB::mtxrInterfaceStatsTxFCSError
     14 MIKROTIK-MIB::mtxrInterfaceStatsTxControl
     14 MIKROTIK-MIB::mtxrInterfaceStatsTxFragment
     14 MIKROTIK-MIB::mtxrInterfaceStatsLinkDowns
     14 MIKROTIK-MIB::mtxrInterfaceStatsTxRx1024ToMax
      4 MIKROTIK-MIB::mtxrPOEInterfaceIndex
      4 MIKROTIK-MIB::mtxrPOEName
      4 MIKROTIK-MIB::mtxrPOEStatus
      4 MIKROTIK-MIB::mtxrPOEVoltage
      4 MIKROTIK-MIB::mtxrPOECurrent
      4 MIKROTIK-MIB::mtxrPOEPower
      1 MIKROTIK-MIB::mtxrIkeSACount
      4 MIKROTIK-MIB::mtxrPOECurrent

However, snmpbulkwalk is giving some errors about walking off the end of the MIB:

# time snmpbulkwalk -m+MIKROTIK-MIB 10.12.250.2 1.3.6.1.4.1.14988 | tail
MIKROTIK-MIB::mtxrPOECurrent.5 = INTEGER: 0
MIKROTIK-MIB::mtxrPOEPower.2 = INTEGER: .0
MIKROTIK-MIB::mtxrPOEPower.3 = INTEGER: .0
MIKROTIK-MIB::mtxrPOEPower.4 = INTEGER: 3.7
MIKROTIK-MIB::mtxrPOEPower.5 = INTEGER: .0
MIKROTIK-MIB::mtxrIkeSACount.1.0 = Gauge32: 0
MIKROTIK-MIB::mtxrPOECurrent.5 = No more variables left in this MIB View (It is past the end of the MIB tree)
MIKROTIK-MIB::mtxrPOECurrent.5 = No more variables left in this MIB View (It is past the end of the MIB tree)
MIKROTIK-MIB::mtxrPOECurrent.5 = No more variables left in this MIB View (It is past the end of the MIB tree)
MIKROTIK-MIB::mtxrPOECurrent.5 = No more variables left in this MIB View (It is past the end of the MIB tree)

real    0m4.841s
user    0m0.132s
sys 0m0.066s

The problem goes away with -Cr1 or -Cr2 (max repetitions), or using snmpwalk instead of snmpbulkwalk, but it still persists with -Cr3:

# time snmpbulkwalk -Cr3 -m+MIKROTIK-MIB 10.12.250.2 1.3.6.1.4.1.14988 | tail
MIKROTIK-MIB::mtxrPOECurrent.3 = INTEGER: 0
MIKROTIK-MIB::mtxrPOECurrent.4 = INTEGER: 145
MIKROTIK-MIB::mtxrPOECurrent.5 = INTEGER: 0
MIKROTIK-MIB::mtxrPOEPower.2 = INTEGER: .0
MIKROTIK-MIB::mtxrPOEPower.3 = INTEGER: .0
MIKROTIK-MIB::mtxrPOEPower.4 = INTEGER: 3.4
MIKROTIK-MIB::mtxrPOEPower.5 = INTEGER: .0
MIKROTIK-MIB::mtxrIkeSACount.1.0 = Gauge32: 0
MIKROTIK-MIB::mtxrIkeSATableEntry = No more variables left in this MIB View (It is past the end of the MIB tree)
MIKROTIK-MIB::mtxrIkeSATableEntry = No more variables left in this MIB View (It is past the end of the MIB tree)

real    0m3.643s
user    0m0.190s
sys 0m0.102s

An indeed, setting max_repetitions: 2 under the mib definition in snmp.yml fixes the problem:

# curl -fsS 'localhost:9116/snmp?target=gw1&module=mikrotik&auth=workshop_v3' | grep -v '^#' | sed 's/[{ ].*$//' | grep -v mtxrInterfaceStats | uniq -c
      1 mtxrAltitude
      1 mtxrBoardName
      1 mtxrBuildTime
      1 mtxrDHCPLeaseCount
      1 mtxrDate
      1 mtxrFirmwareUpgradeVersion
      1 mtxrFirmwareVersion
      2 mtxrGaugeName
      2 mtxrGaugeUnit
      2 mtxrGaugeValue
      1 mtxrHlActiveFan
      1 mtxrHlProcessorFrequency
      1 mtxrHlTemperature
      1 mtxrHlVoltage
      1 mtxrIkeSACount
      1 mtxrLatitude
      1 mtxrLicLevel
      1 mtxrLicSoftwareId
      1 mtxrLicUpgrUntil
      1 mtxrLicUpgradableTo
      1 mtxrLicVersion
      1 mtxrLongtitude
      1 mtxrNeighborIdentity
      1 mtxrNeighborInterfaceID
      1 mtxrNeighborIpAddress
      1 mtxrNeighborMacAddress
      1 mtxrNeighborPlatform
      1 mtxrNeighborSoftwareID
      1 mtxrNeighborVersion
      1 mtxrNote
      4 mtxrPOECurrent
      4 mtxrPOEInterfaceIndex
      4 mtxrPOEName
      4 mtxrPOEPower
      4 mtxrPOEStatus
      4 mtxrPOEVoltage
      1 mtxrSattelites
      1 mtxrSerialNumber
      1 mtxrSpeed
      1 mtxrSystemReboot
      1 mtxrUSBPowerReset
      1 mtxrValid
      1 mtxrWirelessModemSignalECIO
      1 mtxrWirelessModemSignalStrength
      1 mtxrWlCMREntryCount
      1 mtxrWlCMRtabEntryCount
      1 mtxrWlRtabEntryCount
      1 snmp_scrape_duration_seconds
      1 snmp_scrape_packets_retried
      1 snmp_scrape_packets_sent
      1 snmp_scrape_pdus_returned
      1 snmp_scrape_walk_duration_seconds
      1 sysDescr

Therefore this seems likely to be a protocol error that I can raise with Mikrotik support, if I can point to exactly where in the packet it's broken.

Aside: there's something I don't understand in the generator.yml for mikrotik. It says to walk laIndex, which gets mapped to 1.3.6.1.4.1.2021.10.1.1 and 1.3.6.1.4.1.2021.10.1.2 (laIndex and laNames) in snmp.yml. I can't see anything in mikrotik.mib which references laIndex or laNames. In any case, my Mikrotik device returns nothing under the entire laTable (1.3.6.1.4.1.2021.10). And even if it did, the interesting info about load averages is returned in other columns.

candlerb commented 3 months ago

FYI, here's the final packet exchange:

18:24:51.679212 IP 10.12.255.33.57499 > 10.12.250.2.161:  F=ar U="admin" E=_80_00_3a_8c_04 C="" GetBulk(34)  N=0 M=10 .1.3.6.1.4.1.14988.1.1.15.1.1.5.5
        0x0000:  4500 008c 13b0 4000 4011 1975 0a0c ff21  E.....@.@..u...!
        0x0010:  0a0c fa02 e09b 00a1 0078 0dc6 306e 0201  .........x..0n..
        0x0020:  0330 1002 0471 5c3d 6102 0205 c004 0105  .0...q\=a.......
        0x0030:  0201 0304 2830 2604 0580 003a 8c04 0201  ....(0&....:....
        0x0040:  0502 0301 a82a 0405 6164 6d69 6e04 0cbf  .....*..admin...
        0x0050:  55d0 f62b 52d5 5b30 5d26 d204 0030 2d04  U..+R.[0]&...0-.
        0x0060:  0580 003a 8c04 0400 a522 0204 6525 38dc  ...:....."..e%8.
        0x0070:  0201 0002 010a 3014 3012 060e 2b06 0104  ......0.0...+...
        0x0080:  01f5 0c01 010f 0101 0505 0500            ............
18:24:51.754948 IP 10.12.250.2.161 > 10.12.255.33.57499:  F=a U="admin" E=_80_00_3a_8c_04 C="" GetResponse(220)  .1.3.6.1
.4.1.14988.1.1.15.1.1.6.2=0 .1.3.6.1.4.1.14988.1.1.15.1.1.6.3=0 .1.3.6.1.4.1.14988.1.1.15.1.1.6.4=42 .1.3.6.1.4.1.14988.1
.1.15.1.1.6.5=0 .1.3.6.1.4.1.14988.1.1.20.1.1.0=0 .1.3.6.1.6.3.10.2.1.1.0=80_00_3a_8c_04 .1.3.6.1.4.1.14988.1.1.15.1.1.5.
5=[endOfMibView] .1.3.6.1.4.1.14988.1.1.15.1.1.5.5=[endOfMibView] .1.3.6.1.4.1.14988.1.1.15.1.1.5.5=[endOfMibView] .1.3.6
.1.4.1.14988.1.1.15.1.1.5.5=[endOfMibView]
        0x0000:  4500 014a 7289 4000 3f11 badd 0a0c fa02  E..Jr.@.?.......
        0x0010:  0a0c ff21 00a1 e09b 0136 9941 3082 012a  ...!.....6.A0..*
        0x0020:  0201 0330 1002 0471 5c3d 6102 0205 c004  ...0...q\=a.....
        0x0030:  0101 0201 0304 2830 2604 0580 003a 8c04  ......(0&....:..
        0x0040:  0201 0502 0301 a82a 0405 6164 6d69 6e04  .......*..admin.
        0x0050:  0c50 224b 9d7f 289a 195e 965e 4504 0030  .P"K..(..^.^E..0
        0x0060:  81e8 0405 8000 3a8c 0404 00a2 81dc 0204  ......:.........
        0x0070:  6525 38dc 0201 0002 0100 3081 cd30 1306  e%8.......0..0..
        0x0080:  0e2b 0601 0401 f50c 0101 0f01 0106 0202  .+..............
        0x0090:  0100 3013 060e 2b06 0104 01f5 0c01 010f  ..0...+.........
        0x00a0:  0101 0603 0201 0030 1306 0e2b 0601 0401  .......0...+....
        0x00b0:  f50c 0101 0f01 0106 0402 012a 3013 060e  ...........*0...
        0x00c0:  2b06 0104 01f5 0c01 010f 0101 0605 0201  +...............
        0x00d0:  0030 1206 0d2b 0601 0401 f50c 0101 1401  .0...+..........
        0x00e0:  0100 4201 0030 1306 0a2b 0601 0603 0a02  ..B..0...+......
        0x00f0:  0101 0004 0580 003a 8c04 3012 060e 2b06  .......:..0...+.
        0x0100:  0104 01f5 0c01 010f 0101 0505 8200 3012  ..............0.
        0x0110:  060e 2b06 0104 01f5 0c01 010f 0101 0505  ..+.............
        0x0120:  8200 3012 060e 2b06 0104 01f5 0c01 010f  ..0...+.........
        0x0130:  0101 0505 8200 3012 060e 2b06 0104 01f5  ......0...+.....
        0x0140:  0c01 010f 0101 0505 8200                 ..........

It's basically run off past the end of the vendor MIB tree (1.3.6.1.4.1), returns the next OID (.1.3.6.1.6.3.10.2.1.1.0 = SNMP-FRAMEWORK-MIB::snmpEngineID), and then it happens to be the end of the MIB so remaining response slots are labelled [endOfMibView]

To me this looks like it could be legitimate according to RFC3416 section 4.2.3 (although that's pretty complicated!). I'll need to dig into how gosnmp interoperates with this. Running off the end of the entire MIB is an edge case which I guess is rarely seen in practice.

candlerb commented 3 months ago

Ergh, it has started working now.

The changes I was making were:

But now I've put things back how they were, and it's all working just fine. The response packets still have endOfMibView (lots, with 25 repetitions per packet).

20:34:28.499800 IP 10.12.250.1.161 > 10.12.255.33.48058:  F=ar U="admin" E=_80_00_3a_8c_04 C="" GetResponse(565)  .1.3.6.1.4.1.14988.1.1.15.1.1.6.10=0 .1.3.6.1.4.1.14988.1.1.17.1.1.2.1="part0" .1.3.6.1.4.1.14988.1.1.17.1.1.3.1=512 .1.3.6.1.4.1.14988.1.1.17.1.1.4.1="RouterOS v6.49.14 Apr/03/2024 14:26:16" .1.3.6.1.4.1.14988.1.1.17.1.1.5.1=1 .1.3.6.1.4.1.14988.1.1.17.1.1.6.1=1 .1.3.6.1.4.1.14988.1.1.20.1.1.0=0 .1.3.6.1.6.3.10.2.1.1.0=80_00_3a_8c_04 .1.3.6.1.4.1.14988.1.1.15.1.1.5.10=[endOfMibView] .1.3.6.1.4.1.14988.1.1.15.1.1.5.10=[endOfMibView] .1.3.6.1.4.1.14988.1.1.15.1.1.5.10=[endOfMibView] .1.3.6.1.4.1.14988.1.1.15.1.1.5.10=[endOfMibView] .1.3.6.1.4.1.14988.1.1.15.1.1.5.10=[endOfMibView] .1.3.6.1.4.1.14988.1.1.15.1.1.5.10=[endOfMibView] .1.3.6.1.4.1.14988.1.1.15.1.1.5.10=[endOfMibView] .1.3.6.1.4.1.14988.1.1.15.1.1.5.10=[endOfMibView] .1.3.6.1.4.1.14988.1.1.15.1.1.5.10=[endOfMibView] .1.3.6.1.4.1.14988.1.1.15.1.1.5.10=[endOfMibView] .1.3.6.1.4.1.14988.1.1.15.1.1.5.10=[endOfMibView] .1.3.6.1.4.1.14988.1.1.15.1.1.5.10=[endOfMibView] .1.3.6.1.4.1.14988.1.1.15.1.1.5.10=[endOfMibView] .1.3.6.1.4.1.14988.1.1.15.1.1.5.10=[endOfMibView] .1.3.6.1.4.1.14988.1.1.15.1.1.5.10=[endOfMibView] .1.3.6.1.4.1.14988.1.1.15.1.1.5.10=[endOfMibView] .1.3.6.1.4.1.14988.1.1.15.1.1.5.10=[endOfMibView]
    0x0000:  4500 02a6 0000 4000 4011 2b0c 0a0c fa01  E.....@.@.+.....
    0x0010:  0a0c ff21 00a1 bbba 0292 465e 3082 0286  ...!......F^0...
    0x0020:  0201 0330 1102 043c 86b7 0e02 0300 ffff  ...0...<........
    0x0030:  0401 0502 0103 0428 3026 0405 8000 3a8c  .......(0&....:.
    0x0040:  0402 013f 0203 009b 7f04 0561 646d 696e  ...?.......admin
    0x0050:  040c c6e8 e093 d080 4d5b b82d 3973 0400  ........M[.-9s..
    0x0060:  3082 0242 0405 8000 3a8c 0404 00a2 8202  0..B....:.......
    0x0070:  3502 043c 86b7 0e02 0100 0201 0030 8202  5..<.........0..
    0x0080:  2530 1306 0e2b 0601 0401 f50c 0101 0f01  %0...+..........
    0x0090:  0106 0a02 0100 3017 060e 2b06 0104 01f5  ......0...+.....
    0x00a0:  0c01 0111 0101 0201 0405 7061 7274 3030  ..........part00
    0x00b0:  1406 0e2b 0601 0401 f50c 0101 1101 0103  ...+............
    0x00c0:  0102 0202 0030 3806 0e2b 0601 0401 f50c  .....08..+......
    0x00d0:  0101 1101 0104 0104 2652 6f75 7465 724f  ........&RouterO
    0x00e0:  5320 7636 2e34 392e 3134 2041 7072 2f30  S.v6.49.14.Apr/0
    0x00f0:  332f 3230 3234 2031 343a 3236 3a31 3630  3/2024.14:26:160
    0x0100:  1306 0e2b 0601 0401 f50c 0101 1101 0105  ...+............
    0x0110:  0102 0101 3013 060e 2b06 0104 01f5 0c01  ....0...+.......
    0x0120:  0111 0101 0601 0201 0130 1206 0d2b 0601  .........0...+..
    0x0130:  0401 f50c 0101 1401 0100 4201 0030 1306  ..........B..0..
    0x0140:  0a2b 0601 0603 0a02 0101 0004 0580 003a  .+.............:
    0x0150:  8c04 3012 060e 2b06 0104 01f5 0c01 010f  ..0...+.........
    0x0160:  0101 050a 8200 3012 060e 2b06 0104 01f5  ......0...+.....
    0x0170:  0c01 010f 0101 050a 8200 3012 060e 2b06  ..........0...+.
    0x0180:  0104 01f5 0c01 010f 0101 050a 8200 3012  ..............0.
    0x0190:  060e 2b06 0104 01f5 0c01 010f 0101 050a  ..+.............
    0x01a0:  8200 3012 060e 2b06 0104 01f5 0c01 010f  ..0...+.........
    0x01b0:  0101 050a 8200 3012 060e 2b06 0104 01f5  ......0...+.....
    0x01c0:  0c01 010f 0101 050a 8200 3012 060e 2b06  ..........0...+.
    0x01d0:  0104 01f5 0c01 010f 0101 050a 8200 3012  ..............0.
    0x01e0:  060e 2b06 0104 01f5 0c01 010f 0101 050a  ..+.............
    0x01f0:  8200 3012 060e 2b06 0104 01f5 0c01 010f  ..0...+.........
    0x0200:  0101 050a 8200 3012 060e 2b06 0104 01f5  ......0...+.....
    0x0210:  0c01 010f 0101 050a 8200 3012 060e 2b06  ..........0...+.
    0x0220:  0104 01f5 0c01 010f 0101 050a 8200 3012  ..............0.
    0x0230:  060e 2b06 0104 01f5 0c01 010f 0101 050a  ..+.............
    0x0240:  8200 3012 060e 2b06 0104 01f5 0c01 010f  ..0...+.........
    0x0250:  0101 050a 8200 3012 060e 2b06 0104 01f5  ......0...+.....
    0x0260:  0c01 010f 0101 050a 8200 3012 060e 2b06  ..........0...+.
    0x0270:  0104 01f5 0c01 010f 0101 050a 8200 3012  ..............0.
    0x0280:  060e 2b06 0104 01f5 0c01 010f 0101 050a  ..+.............
    0x0290:  8200 3012 060e 2b06 0104 01f5 0c01 010f  ..0...+.........
    0x02a0:  0101 050a 8200                           ......

Before that, I was doing some gosnmp tests locally from macOS, and those were working fine too:

==> main2.go <==
// Copyright 2012 The GoSNMP Authors. All rights reserved.  Use of this
// source code is governed by a BSD-style license that can be found in the
// LICENSE file.

// This program demonstrates BulkWalk.
package main

import (
    "flag"
    "fmt"
    "os"
    "time"

    "github.com/gosnmp/gosnmp"
)

func main() {
    flag.Usage = func() {
        fmt.Printf("Usage:\n")
        fmt.Printf("     host      - the host to walk/scan\n")
        fmt.Printf("     oid       - the MIB/Oid defining a subtree of values\n\n")
        flag.PrintDefaults()
    }

    flag.Parse()

    if len(flag.Args()) < 1 {
        flag.Usage()
        os.Exit(1)
    }
    target := flag.Args()[0]
    var oid string = "1.3.6.1.2.1"
    if len(flag.Args()) > 1 {
        oid = flag.Args()[1]
    }

        // build our own GoSNMP struct, rather than using gosnmp.Default
        params := &gosnmp.GoSNMP{
                Target:        target,
                Port:          161,
                Version:       gosnmp.Version3,
                SecurityModel: gosnmp.UserSecurityModel,
                MsgFlags:      gosnmp.AuthNoPriv,
                Timeout:       time.Duration(30) * time.Second,
                MaxRepetitions: 10,
                SecurityParameters: &gosnmp.UsmSecurityParameters{UserName: "admin",
                        AuthenticationProtocol:   gosnmp.SHA,
                        AuthenticationPassphrase: "NetManage",
                        //PrivacyProtocol:          gosnmp.DES,
                        //PrivacyPassphrase:        "password",
                },
        }

    err := params.Connect()
    if err != nil {
        fmt.Printf("Connect err: %v\n", err)
        os.Exit(1)
    }
    defer params.Conn.Close()

    results, err := params.BulkWalkAll(oid)
    if err != nil {
        fmt.Printf("Walk Error: %v\n", err)
        os.Exit(1)
    }
    for _, v := range results {
        fmt.Printf("%v\n", v)
    }
}

(This is based on examples/walkexample and examples/example3 from the gosnmp repo)

% go run main2.go 10.12.250.2 1.3.6.1.4.1.14988 | tail -5
{0 .1.3.6.1.4.1.14988.1.1.15.1.1.6.2 Integer}
{0 .1.3.6.1.4.1.14988.1.1.15.1.1.6.3 Integer}
{34 .1.3.6.1.4.1.14988.1.1.15.1.1.6.4 Integer}
{0 .1.3.6.1.4.1.14988.1.1.15.1.1.6.5 Integer}
{0 .1.3.6.1.4.1.14988.1.1.20.1.1.0 Gauge32}
%

That uses BulkWalkAll() like snmp_exporter does, but I get the same with BulkWalk() and a callback function.

It has pulled in gosnmp v1.37.0, but that seems to be the same as snmp_exporter 0.25.0 was built with:

% go version -m snmp_exporter | grep gosnmp
    dep github.com/gosnmp/gosnmp    v1.37.0 h1:/Tf8D3b9wrnNuf/SfbvO+44mPrjVphBhRtcGg22V07Y=

So either the response from the Mikrotik has changed in some subtle way, or the behaviour of the gosnmp library has changed in a subtle way, like there's some uninitialized state.

I'll have to see if it fails again, and then debug further. I can't reboot the mikrotik right now.

SuperQ commented 3 months ago

So many cases where the max repetitions default is too high. IIRC net-snmp defaults to 10.