SSH Connection Closed While Looping over a command list

[mountaintop1]

With respect to below, ssh connection closes after the first command. This works with Juniper device but failed with Cisco devices.

Am I missing something? It is looking like there is an exit after the first command.

async def get_show_using_asynssh(dict_data: dict): async with asyncssh.connect(host=dict_data["ip_address"], username=dict_data["username"], password=dict_data["password"], known_hosts=None) as conn: output = {} commands = ['show mac address-table', 'show lldp neighbors detail', 'show vlan'] try:
for cmd in commands: # this is a multiple command coroutine out = (await conn.run(cmd, term_size=(5000000000, 2400000), check=True)).stdout if out: output[cmd.split()[1]] = out else: output[cmd.split()[1]] = '' print('Finished send command string.............')

    except Exception as e:
        print(e)
        return f"Error: {str(e)}"

[ronf]

The code you posted got mangled, so it's hard to see what the indentation was. However, assuming that had the calls to conn.run() properly inside the async with asyncssh.connect(), you're probably running into an issue that many embedded SSH servers only support a single session per connection. An attempt to start additional connections will fail on such servers.

Your options are to either do a separate asyncssh.connect() for each command you want to send, which means paying the cost of authenticating each time, or you need to open an interactive shell and send commands one after another on a single session. This is tricky to do, though, as many of these servers only accept input when they are done running each command, and unless you know the expected output of the command, it's hard to know when to stop reading one command's output and when to start sending the next command.

If you look through the closed issues here, you'll see a number of examples about different ways of handling this. One example is #121. Another example is #227, and there are probably others as well as this topic comes up repeatedly. Unfortunately, there's no simple answer here. It really depends on the device and your ability to get the device to output something unique enough at the end of each command's output to give you a reliable indicator of when it is safe to send the next command. For instance, if you can configure a unique prompt on the device which won't show up in the command output, you can do a readuntil() on that.

[mountaintop1]

I tried the #121 below it didnt work. The device is a Cisco IOS XE. I need your guidance please. My plan is to capture the output of each command in the list and save as a key to a dictionary. example is output = {"mac": "", "vlan": "", "lldp": ""}

`async def get_show_using_asynssh(dict_data: dict):

async with asyncssh.connect(host=dict_data["ip_address"], username=dict_data["username"], 
                                                      password=dict_data["password"], known_hosts=None) as conn:
    output = {}

    try:
    commands =  ['show mac address-table', 'show lldp neighbors detail', 'show vlan']
    process = await conn.create_process(command='<remote process, eg: bc>')

    await asyncio.wait_for(process.stdout.readuntil(PROMPT), timeout=10)
    for cmd in commands:
        process.stdin.write(cmd + '\n')
        result = ''
        try:
            result+=await\
                    asyncio.wait_for(
                        process.stdout.readuntil(PROMPT),timeout=10)
        except Exception as err:
            logging.warning("prompt timeout step {}".format(err))
        print(result, end='')
    except Exception as e:
        return f"Error: {str(e)}"

Below is the error I got.`

1941 bytes read on a total of undefined expected bytes 1935 bytes read on a total of undefined expected bytes 1909 bytes read on a total of undefined expected bytes 1935 bytes read on a total of undefined expected bytes 1935 bytes read on a total of undefined expected bytes 1942 bytes read on a total of undefined expected bytes

Thanks for your help!

[ronf]

The error you're seeing generally indicates that you received an end-of-file from the remote system before getting the prompt you told it to look for. I don't see the definition of PROMPT in the sample code above, but make sure that the remote system actually does output a prompt, and that you set PROMPT to match whatever it prints when command output ends and it is ready to receive a new command.

Also, for this to work, you shouldn't be providing a command. You need it to open an interactive shell, which occurs when you call create_process() with no command set.

[mountaintop1]

It seems there is no straight forward solution for this just like we use Netmiko or Paramiko for Cisco device. Asynssh works with Juniper but this issue with Cisco is major one. It will be good if you have a solution for this. I have edited the code. I now get some data from the command output but not all. I dont the the output of show vlan command. Am I missing anything here?

PROMPT = r'#'

async def get_show_using_asynssh(dict_data: dict):
    async with asyncssh.connect(host=dict_data["ip_address"], username=dict_data["username"], 
                                                          password=dict_data["password"], known_hosts=None) as conn:
        output = {}

        try:
                commands =  ['show mac address-table', 'show lldp neighbors detail',  'show vlan']
                process = await conn.create_process()
                process.stdin.write('terminal lenght 0' + '\n')

                await asyncio.wait_for(process.stdout.readuntil(PROMPT), timeout=30)
                for cmd in commands:
                    try:
                        process.stdin.write(cmd + '\n')
                        output[cmd.split()[1]] await asyncio.wait_for(process.stdout.readuntil(PROMPT), timeout=30)
                    except Exception as err:
                        logging.warning("prompt timeout step {}".format(err))
        except Exception as e:
            return f"Error: {str(e)}"

[ronf]

By any chance, does any of the output in any of the commands contain '#' in the output? If so, your prompt isn't unique enough to work reliably.

I also see what looks like a typo in the "terminal length 0" command string. That might prevent you from getting to a prompt if the amount of output is larger than the default terminal length.

[mountaintop1]

Below is the prompt after login and after command output.

cisco_switch_prompt#

'terminal length 0' in cisco allows you to display ouput without breaks

below is an example of command and output;

device_name#sh vlan

VLAN Name                             Status    Ports
---- -------------------------------- --------- -------------------------------
1    default                          active    Te1/1/3, Te1/1/4, Ap1/0/1
60   MGMT                             active
1002 fddi-default                     act/unsup
1003 token-ring-default               act/unsup
1004 fddinet-default                  act/unsup
1005 trnet-default                    act/unsup
1181 1181_41ra0551                    active    Gi1/0/1, Gi1/0/2, Gi1/0/3, Gi1/0/4, Gi1/0/5, Gi1/0/6
                                                Gi1/0/7, Gi1/0/8, Gi1/0/9, Gi1/0/10, Gi1/0/11, Gi1/0/12
                                                Gi1/0/13, Gi1/0/14, Gi1/0/15, Gi1/0/16, Te1/0/17, Te1/0/18
                                                Te1/0/19, Te1/0/20, Te1/0/21, Te1/0/22, Te1/0/23
1394 1394_41ra0551                    active    Te1/0/24
2181 2181_41ra0551                    active
3181 3181_41ra0551                    active
4001 blackhole                        active

VLAN Type  SAID       MTU   Parent RingNo BridgeNo Stp  BrdgMode Trans1 Trans2
---- ----- ---------- ----- ------ ------ -------- ---- -------- ------ ------
1    enet  100001     1500  -      -      -        -    -        0      0
60   enet  100060     1500  -      -      -        -    -        0      0
1002 fddi  101002     1500  -      -      -        -    -        0      0
1003 tr    101003     1500  -      -      -        -    -        0      0
1004 fdnet 101004     1500  -      -      -        ieee -        0      0
1005 trnet 101005     1500  -      -      -        ibm  -        0      0
1181 enet  101181     1500  -      -      -        -    -        0      0
1394 enet  101394     1500  -      -      -        -    -        0      0
2181 enet  102181     1500  -      -      -        -    -        0      0
3181 enet  103181     1500  -      -      -        -    -        0      0
4001 enet  104001     1500  -      -      -        -    -        0      0

Remote SPAN VLANs
------------------------------------------------------------------------------

Primary Secondary Type              Ports
------- --------- ----------------- ------------------------------------------

device_name#

thanks for your time

[ronf]

'terminal length 0' in cisco allows you to display ouput without breaks

Yes, I understand. My point was that your sample code included 'terminal lenght 0' instead of 'terminal length 0'. This would cause it to try and paginate the output, probably causing it to stop before it got back to the prompt. This would make the readuntil() time out.

I would also set PROMPT to something more than just '#', unless you're sure that none of the commands you were running would ever include a '#' in their output.

[mountaintop1]

Sorry I didnt get you here "My point was that your sample code included 'terminal lenght 0' instead of 'terminal length 0'"

[mountaintop1]

By the way I see this as part of the output

'device_name#') WARNING:root:prompt timeout step WARNING:root:prompt timeout step WARNING:root:prompt timeout step WARNING:root:prompt timeout step WARNING:root:prompt timeout step ('show mac address-table\r\n' ' Mac Address Table\r\n' '-------------------------------------------\r\n' '\r\n'

[ronf]

Take a look at your sample code above. One of the lines is:

                process.stdin.write('terminal lenght 0' + '\n')

This appears to be a typo in the command you are trying to run to disable pagination.

[mountaintop1]

Oh I see it now ht vs th........ Thanks so much I got it to work now by adding 'terminal length 0' to the list of command

async def get_show_using_asynssh(dict_data: dict):
    PROMPT = r'#'
    async with asyncssh.connect(host=dict_data["ip_address"], username=dict_data["username"] 
                          password=dict_data["password"], known_hosts=None,) as conn:
        output = {}
        commands =  ['terminal length 0', 'show mac address-table', 'show lldp neighbors detail', 'sh vlan brief']
         process = await conn.create_process()

          await asyncio.wait_for(process.stdout.readuntil(PROMPT), timeout=30)
           for cmd in commands:
                    try:
                        process.stdin.write(cmd + '\n')
                        output[cmd.split()[1]] = await asyncio.wait_for(process.stdout.readuntil(PROMPT), timeout=20)
                    except Exception as err:
                        logging.warning("prompt timeout step {}".format(err))

Thanks for helping me see this through. By the way, I am testing the speed of Asyncio (asynssh) vs Concurrent futures (Netmiko) for collecting data and configuring both cisco and Juniper switches.

I performed the operation on 150 switches and my result is Concurrent Future took about 75 secs Asynssh took 18 secs

Asynssh is way better and faster. Thanks so much.

[ronf]

Glad to hear you got it working!

Also, thanks for sharing the results of your performance tests... That's a pretty nice improvement!

Opening many connections at once is definitely one of AsyncSSH's strengths, as it doesn't require separate processes (or threads per connection (or even per session) the way most SSH client libraries do. If you are running on a multi-core client, you can improve performance even more by running multiple event loops at once, but you only want a small number of threads (one per available core) in that case, rather than opening a separate process or thread for each connection.

[mountaintop1]

How can I run multiple event loops at once? example

Thanks

[ronf]

I don't have an example handy, and it can get pretty involved depending on how independent your tasks are, but it basically involves using the "multiprocessing" library to create a pool of processes, letting each process create its own asyncio event loop, and then dividing up the work you are doing so that is gets spread across the multiple threads.

For example, let's say that you knew up front what the 150 servers were you wanted to pull data from, and you wanted to spread the work across 5 cores. You could simply break the list of servers into 5 lists of 30 servers each, and then you could create 5 separate processes and feed each process one of the 5 lists. In your main process, you'd call join() on each of the 5 processes, collecting the results from each until you had all 150 responses.

Fancier versions of this could involve having a multiprocess queue, where you have your main task add work items to the queue and let the child processes pull work off the queue, writing the results back to another multiprocess queue, where you limit the number of asyncio tasks each process starts pulling work from the pool until it hits some limit on the number of concurrent tasks you want it to start. After that, each task that completes would trigger it to get another one until the queue was exhausted.

I don't have any direct experience with it, but you might want to experiment with https://aiomultiprocess.omnilib.dev/en/stable/guide.html to do some of the work for you, providing an easy way to schedule async tasks across multiple processes.