napalm-automation / napalm-salt

Modules for event-driven network automation and orchestration using Salt
Apache License 2.0
128 stars 36 forks source link

Proxy times out when calling function from Salt Engine #58

Open TheBirdsNest opened 5 years ago

TheBirdsNest commented 5 years ago

Hi,

I'm trying to use a Salt Engine running within a Proxy Minion process to periodically execute remote commands. Every 10 seconds the engine attempts to call net.ping on the router as follows:

while True:
   test = __salt__['net.ping']('8.8.8.8')
   log.info(test)
   time.sleep(10)

When executing, it causes the connection to end and the proxy restarts.. Looking at the debug, I see in the write_channel the command is issued but a reply is not ready for 60 seconds at which point the keepalive function is called and assumes the proxy is down.

TheBirdsNest commented 5 years ago

From looking at the command history, I can see when executing the above command via the engine, the following appears on the router:

%SSH-3-BAD_PACK_LEN: Bad packet length -1118152816

TheBirdsNest commented 5 years ago
[ERROR   ] Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/utils/napalm.py", line 178, in call
    out = getattr(napalm_device.get('DRIVER'), method)(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/napalm/ios/ios.py", line 2491, in ping
    output = self._send_command(command)
  File "/usr/lib/python2.7/site-packages/napalm/ios/ios.py", line 189, in _send_command
    raise ConnectionClosedException(str(e))
ConnectionClosedException
TheBirdsNest commented 5 years ago
[DEBUG   ] read_channel:
[DEBUG   ] write_channel: ping 8.8.8.8 timeout 2 size 100 repeat 5

[DEBUG   ] read_channel:
[DEBUG   ] read_channel:
[DEBUG   ] read_channel:
[DEBUG   ] read_channel:
TheBirdsNest commented 5 years ago

Note this also does not work if I call the module directly:

 test = __proxy__['napalm.call'](
                'ping',
                **{
                'destination': params['opts']['target'],
                'source': None,
                'ttl': None,
                'timeout': 2,
                'size': 100,
                'count': 5
                }
            )
mirceaulinic commented 5 years ago

Hey @TheBirdsNest. Have you looked at the Scheduler: https://docs.saltstack.com/en/latest/topics/jobs/. Unless you need to dynamically change the targets, you can schedule the example you shared in https://github.com/napalm-automation/napalm-salt/issues/58#issue-424915327, as follows:

schedule:
  run_pings:
    function: net.ping
    args:
       - 8.8.8.8
    seconds: 10

Alternatively, if you need to run towards various targets, you can indeed have an Engine (although you can equally have a scheduled custom Execution Module, or Runner on the Master). I'd recommend you to use __salt__ instead of __proxy__, as the latter is rather unpredictable and not sufficiently flexible (especially if you're running in mixed environments). Something like this should work:

 test = __salt__['net.ping'](
    params['opts']['target'],
    timeout=2,
    size=100,
    count=5
)