No SNMP response received before timeout With AsyncIO

[barrycarey]

Hello!

I've been using PySNMP synchronously for the last year. I'm attempting to convert to asyncio, however, all requests I make are coming back with no response received.

Below is the log output when running the exact example located at: http://snmplabs.com/pysnmp/examples/hlapi/asyncio/contents.html

I've also attempted with confirmed good hosts with the same result. The same hosts get a response when using the sync version.

Debug output: https://gist.github.com/barrycarey/8f5fd73db5ae0db8d08450c07f9f8592

[etingof]

This is curious. The linked example works for me as-is.

Which Python do you use? I use 3.7.

Can you check it out with tcpdump - does your script send out SNMP message? Is there any response coming and when?

Thanks!

[barrycarey]

I'll grab a dump for you tomorrow.

I'm using Python 3.6.

It's bizarre. If I used the normal example, it works. If I use the AsyncIO example it times out.

[BRJ-zz]

Hello,

I am seeing exactly the same behavior as barrycarey using the asyncio module ! and Python 3.7.0. Here is my script:

import asyncio
from pysnmp.hlapi.asyncio import *
from pysnmp import debug 

@asyncio.coroutine
def getone(snmpEngine, hostname):
    errorIndication, errorStatus, errorIndex, varBinds = yield from getCmd(
        snmpEngine,
        CommunityData('public, mpModel=1'),
        UdpTransportTarget(hostname),
        ContextData(),
        ObjectType(ObjectIdentity('SNMPv2-MIB', 'sysDescr', 0))
    )

    if errorIndication:
        print(errorIndication)
    elif errorStatus:
        print('%s at %s' % (
            errorStatus.prettyPrint(),
            errorIndex and varBinds[int(errorIndex) - 1][0] or '?'
        )
              )
    else:
        for varBind in varBinds:
            print(' = '.join([x.prettyPrint() for x in varBind]))

debug.setLogger(debug.Debug('io', 'msgproc'))

snmpEngine = SnmpEngine()

loop = asyncio.get_event_loop()
loop.run_until_complete(
    asyncio.wait([getone(snmpEngine, ('192.168.212.50', 161)),
                  getone(snmpEngine, ('192.168.212.50', 161)),
                  getone(snmpEngine, ('192.168.212.50', 161))])
)

The commands gets queued, but are never sent on the wire (tested with wireshark) ?

C:\Python>python testasync2.py 2018-10-08 10:34:54,124 pysnmp: running pysnmp version 4.4.6 2018-10-08 10:34:54,124 pysnmp: debug category 'io' enabled 2018-10-08 10:34:54,124 pysnmp: debug category 'msgproc' enabled 2018-10-08 10:34:54,349 pysnmp: prepareOutgoingMessage: PDU request-id 7778370 replaced with unique ID 14710918 2018-10-08 10:34:54,350 pysnmp: prepareOutgoingMessage: using contextEngineId <SnmpEngineID value object at 0x402ec30 ta gSet <TagSet object at 0x382ba90 tags 0:0:4> subtypeSpec <ConstraintsIntersection object at 0x3f6cbd0 consts <ValueSizeC onstraint object at 0x37d9670 consts 0, 65535>, <ValueSizeConstraint object at 0x3f6cc10 consts 5, 32>> encoding iso-885 9-1 payload [0x80004fb8054fe8e760]> contextName b'' 2018-10-08 10:34:54,351 pysnmp: generateRequestMsg: Message: version=1 community=public, mpModel=1 data=PDUs: get-request=GetRequestPDU: request-id=14710918 error-status=noError error-index=0 variable-bindings=VarBindList: VarBind: name=1.3.6.1.2.1.1.1.0 =_BindValue: unSpecified=

2018-10-08 10:34:54,351 pysnmp: sendMessage: queuing transportAddress ('192.168.212.50', 161) outgoingMessage 00000: 30 34 02 01 01 04 11 70 75 62 6C 69 63 2C 20 6D 00016: 70 4D 6F 64 65 6C 3D 31 A0 1C 02 04 00 E0 78 86 00032: 02 01 00 02 01 00 30 0E 30 0C 06 08 2B 06 01 02 00048: 01 01 01 00 05 00 2018-10-08 10:34:55,853 pysnmp: StatusInformation: {'errorIndication': RequestTimedOut('No SNMP response received before timeout')}

Any ideas ?

[etingof]

Please make sure you are using correct SNMP community name. The script you've pasted seems to have wrong quotes:

    CommunityData('public, mpModel=1'),

If I fix the quotes, the script works for me:

    CommunityData('public', mpModel=1),

import asyncio
from pysnmp.hlapi.asyncio import *
from pysnmp import debug

@asyncio.coroutine
def getone(snmpEngine, hostname):
    errorIndication, errorStatus, errorIndex, varBinds = yield from getCmd(
        snmpEngine,
        CommunityData('public', mpModel=1),
        UdpTransportTarget(hostname),
        ContextData(),
        ObjectType(ObjectIdentity('SNMPv2-MIB', 'sysDescr', 0))
    )

    if errorIndication:
        print(errorIndication)
    elif errorStatus:
        print('%s at %s' % (
            errorStatus.prettyPrint(),
            errorIndex and varBinds[int(errorIndex) - 1][0] or '?'
        )
              )
    else:
        for varBind in varBinds:
            print(' = '.join([x.prettyPrint() for x in varBind]))

debug.setLogger(debug.Debug('io', 'msgproc'))

snmpEngine = SnmpEngine()

loop = asyncio.get_event_loop()
loop.run_until_complete(
    asyncio.wait([getone(snmpEngine, ('104.236.166.95', 161)),
                  getone(snmpEngine, ('104.236.166.95', 161)),
                  getone(snmpEngine, ('104.236.166.95', 161))])
)

[etingof]

It's bizarre. If I used the normal example, it works. If I use the AsyncIO example it times out.

I am running this example on Mac. Do you guys run it on Windows? May be it has something to do with the platform...?

[barrycarey]

Windows 10 1709 for me.

When checking with Wireshark nothing hits the wire .

I just tested both code examples again. Synchronous works, Async does not. Direct copy and past from the website. Also tried both with a known good host on my network.

[barrycarey]

I'm also working remote today. I'll test again at the office tomorrow to eliminate any weirdness with the VPN connection.

[etingof]

The commands gets queued, but are never sent on the wire (tested with wireshark) ?

Actually, my (working) example logs that connection_made call is invoked while in @barrycarey's debug it seems not:

2018-10-08 23:19:31,049 pysnmp: sendMessage: queuing transportAddress ('104.236.166.95', 161) outgoingMessage 
00000: 30 29 02 01 01 04 06 70 75 62 6C 69 63 A0 1C 02 
00016: 04 00 C1 FD 83 02 01 00 02 01 00 30 0E 30 0C 06 
00032: 08 2B 06 01 02 01 01 01 00 05 00
2018-10-08 23:19:31,050 pysnmp: connection_made: invoked
2018-10-08 23:19:31,051 pysnmp: connection_made: transportAddress ('104.236.166.95', 161) outgoingMessage 
00000: 30 29 02 01 01 04 06 70 75 62 6C 69 63 A0 1C 02 
00016: 04 00 C1 FD 81 02 01 00 02 01 00 30 0E 30 0C 06 
00032: 08 2B 06 01 02 01 01 01 00 05 00
2018-10-08 23:19:31,052 pysnmp: connection_made: transportAddress ('104.236.166.95', 161) outgoingMessage 
00000: 30 29 02 01 01 04 06 70 75 62 6C 69 63 A0 1C 02 
00016: 04 00 C1 FD 82 02 01 00 02 01 00 30 0E 30 0C 06 
00032: 08 2B 06 01 02 01 01 01 00 05 00
2018-10-08 23:19:31,052 pysnmp: connection_made: transportAddress ('104.236.166.95', 161) outgoingMessage 
00000: 30 29 02 01 01 04 06 70 75 62 6C 69 63 A0 1C 02 
00016: 04 00 C1 FD 83 02 01 00 02 01 00 30 0E 30 0C 06 
00032: 08 2B 06 01 02 01 01 01 00 05 00

[etingof]

May be we should use different event loop when running asyncio app on Windows? That should not be the case at least for sockets. But who knows.

[BRJ-zz]

CommunityData('public', mpModel=1),

I fixed the community string problem, but still the same problem.

I have tested both on WIN7 build 7601, and WIN10 1803.

I also thought of the platform problem, so I tested the script in a WSL Ubuntu instance on my WIN10 box, and here it works ! It executes the queries concurrently as expected against my target:

I am new to asyncio, but I think you are on to the problem with changing the event loop. Have to read more about it :-) I would be happy to help test any changes.

Thanks. Brian

[BRJ-zz]

Btw. i get this messages upon script completion Is this normal ?

Tried wrapping the execution in a Try - Finally:

try: loop.run_until_complete( asyncio.wait([getone(snmpEngine, ('192.168.212.50', 161)), getone(snmpEngine, ('192.168.212.50', 161)), getone(snmpEngine, ('192.168.212.50', 161))]) ) finally: loop.close()

but did not solve the message

[lioncui]

i have same problem. if i use asyncio to run the pysnmp, it response timeout ..

[etingof]

Is it also on Windows? Also the example code?

[lioncui]

@etingof yes, problem on Windows, example of using pysnmp official Asynchronous: asyncio code. it look like getCmd can not work in windows'asyncio

[honglei]

I find out that pysnmp does not send any packet to the agent under Win7/pysnmp 4.4.9/python3.7.2

import asyncio
from pysnmp.hlapi.asyncio import *

@asyncio.coroutine
def run():
    errorIndication, errorStatus, errorIndex, varBinds = yield from getCmd(
        SnmpEngine(),
        CommunityData('public'),
        UdpTransportTarget(('demo.snmplabs.com', 161)),
        #UdpTransportTarget(('192.168.2.1', 161)),
        ContextData(),
        ObjectType(ObjectIdentity('SNMPv2-MIB', 'sysDescr', 0))
    )
    print(errorIndication, errorStatus, errorIndex, varBinds)

asyncio.get_event_loop().run_until_complete(run())

[honglei]

pysnmp/carrier/asyncio/dgram/base.py/DgramAsyncioProtocol.openClientMode loop.create_datagram_endpoint has an ignored Exception!

       def openClientMode(self, iface=None):
        try:
            #!!! iface== None, then remote_addr should be setted !!!
            c = self.loop.create_datagram_endpoint(  
                lambda: self, local_addr=iface, family=self.sockFamily
            )

 File "E:\Beidou\CiscoMIB\asyncio_snmp.py", line 22, in <module>
  asyncio.get_event_loop().run_until_complete(run())
File "D:\ProgramData\Anaconda3\envs\dmaProxy\Lib\asyncio\base_events.py", line 571, in run_until_complete
  self.run_forever()
File "D:\ProgramData\Anaconda3\envs\dmaProxy\Lib\asyncio\base_events.py", line 539, in run_forever
  self._run_once()
File "D:\ProgramData\Anaconda3\envs\dmaProxy\Lib\asyncio\base_events.py", line 1775, in _run_once
  handle._run()
File "D:\ProgramData\Anaconda3\envs\dmaProxy\Lib\asyncio\events.py", line 88, in _run
  self._context.run(self._callback, *self._args)
File "D:\ProgramData\Anaconda3\envs\dmaProxy\Lib\asyncio\base_events.py", line 1250, in create_datagram_endpoint
  sock, protocol, r_addr, waiter)
File "D:\ProgramData\Anaconda3\envs\dmaProxy\Lib\asyncio\selector_events.py", line 79, in _make_datagram_transport
  address, waiter, extra)
File "D:\ProgramData\Anaconda3\envs\dmaProxy\Lib\asyncio\selector_events.py", line 936, in __init__
  super().__init__(loop, sock, protocol, extra)
File "D:\ProgramData\Anaconda3\envs\dmaProxy\Lib\asyncio\selector_events.py", line 581, in __init__
  self._extra['sockname'] = sock.getsockname()

builtins.OSError: [WinError 10022] 提供了一个无效的参数。

[honglei]

like example tcp-echo-client ,

1) the coroutine loop.create_datagram_endpoint shouled be await or yield from.

2) argument remote_addr or local_addr is needed by create_datagram_endpoint .

    transport, protocol = await loop.create_datagram_endpoint(
        lambda: EchoClientProtocol(message, loop),
        remote_addr=('127.0.0.1', 9999))

   transport, protocol = await loop.create_datagram_endpoint(
        lambda: EchoServerProtocol(),
        local_addr=('127.0.0.1', 9999))

[honglei]

The problem can be solved by add remote_addr to create_datagram_endpoint:

pysnmp\hlapi\transport.py

    def openClientMode(self):
        self.transport = self.protoTransport().openClientMode(self.iface,self.transportAddr)
        return self.transport

pysnmp\carrier\asyncio\dgram\base.py

    def openClientMode(self, iface=None, remote_addr=None):
        try:
            c = self.loop.create_datagram_endpoint(
                lambda: self, local_addr=iface, remote_addr=remote_addr, family=self.sockFamily
            )

[etingof]

Thank you for debugging this problem!

I fear that setting remote address to a socket makes it dedicated to this target. So if you got 1000 targets to poll, we will need 1000 sockets to open and maintain.

WDYT?

[honglei]

I prefer the following way, not use the concept of Protocol/Transport

import socket
def read_handler(*args):
    try:
        data, address = sock.recvfrom(2500)
        print (data,address)
    except Exception as e:
        print (e)

import asyncio
if __name__ == '__main__':
    sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, 0)
    sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    sock.setblocking(0)

    io_loop = asyncio.get_event_loop() 
    io_loop.add_reader(sock.fileno(), read_handler)
    sock.sendto( b"xxxxx", ('127.0.0.1', 9999) ) # used to send data
    sock.sendto( b"yyyyy", ('127.0.0.1', 9999) )
    sock.sendto( b"zzzzz", ('127.0.0.1', 9999) )    
    io_loop.run_forever()

[tehki]

Hi mates, I have just tried to migrate from sync to async and face same issue on Windows 10 - errorIndication: No SNMP response received before timeout, both snmp v1, v2, any timeout any oid any host;

There are some limitations of asyncio for Windows platforms: https://docs.python.org/3/library/asyncio-platforms.html

ProactorEventLoop has the following limitations: The loop.create_datagram_endpoint() method is not supported. The loop.add_reader() and loop.add_writer() methods are not supported.

May be that's the root cause...

[etingof]

@legrand If your goal is to have pysnmp running asynchronously, may be you can just use good-old asyncore interface...

I do not see that pysnmp invokes ProactorEventLoop explicitly. Do you think it's platform-default? Also, ProactorEventLoop seems not to support UDP at all...

[tehki]

@etingof Regarding Windows it seems SelectorEventLoop is the default one, yet changing event loop type served no use.

Regarding asyncore that might be a workaround indeed, yet it says

The asyncore module is in Python standard library since ancient times.

• and it would be nice to use modern asyncio module instead, as it looks so simple to implement.

So, I've followed the solution provided by @honglei and added remote_addr to pysnmp\hlapi\transport.py and pysnmp\carrier\asyncio\dgram\base.py and it works now!

And in order limit socket spam I'll be using asyncio semaphore

Thank you very much for your library and credits to @honglei for resolving this issue for Windows platform.

p.s. soz to confuse with nickname change :-)

[tehki]

So far had to modify:

• pysnmp\hlapi\transport.py
• pysnmp\carrier\base.py
• pysnmp\carrier\asyncore\dgram\base.py
• pysnmp\carrier\asyncio\dgram\base.py

to def openClientMode(self, iface=None, remoteaddr=None) to fix TypeError: openClientMode() takes from 1 to 2 positional arguments but 3 were given