SSHClient can't handle large output from executed command

[JeffreyDevloo]

Problem description

The SSHClient.run can and will hang forever on large outputs.

Possible root of the problem

The run method in SSHClient reads the whole stdout in a single go. The issue is, is that this buffer can overflow if a large output from a command is expected.

stdout = ''.join(stdout.readlines()).replace(u'\u2018', u'"').replace(u'\u2019', u'"')

Possible solution

Read stdout partially.

stdin, stdout, stderr = ssh_client.client.exec_command('python {0}'.format(LogFileTimeParser.TEMP_FILE_PATH))
                channel = stdout.channel

                stdin.close()  # we do not need stdin.
                channel.shutdown_write()  # indicate that we're not going to write to that channel anymore
                # read stdout/stderr in order to prevent read block hangs
                stdout_chunks = []
                # chunked read to prevent stalls
                while not channel.closed:  # stop if channel was closed prematurely
                    readq, _, _ = select.select([stdout.channel], [], [], LogFileTimeParser.TIME_OUT)
                    for c in readq:
                        if c.recv_ready():
                            stdout_chunks.append(stdout.channel.recv(len(c.in_buffer)))                          
                        if c.recv_stderr_ready():
                            # make sure to read stderr to prevent stall
                            stderr.channel.recv_stderr(len(c.in_stderr_buffer))
                    if stdout.channel.exit_status_ready() and not stderr.channel.recv_stderr_ready() \
                            and not stdout.channel.recv_ready():
                        # indicate that we're not going to read from this channel anymore
                        stdout.channel.shutdown_read()
                        # close the channel
                        stdout.channel.close()
                        break  # exit as remote side is finished and our buffers are empty

                # close all the pseudofiles
                stdout.close()
                stderr.close()
                ssh_client.client.close()

Additional information

Setup

Hyperconverged setup

• Three nodes with each three disks for the back-end

  Package information

• openvstorage 2.7.2-rev.3859.42b3488-1 amd64 openvStorage
• openvstorage-backend 1.7.2-rev.675.37ca5b8-1 amd64 openvStorage Backend plugin
• openvstorage-backend-core 1.7.2-rev.675.37ca5b8-1 amd64 openvStorage Backend plugin core
• openvstorage-backend-webapps 1.7.2-rev.675.37ca5b8-1 amd64 openvStorage Backend plugin Web Applications
• openvstorage-cinder-plugin 1.2.2-rev.32.948a8c1-1 amd64 OpenvStorage Cinder plugin for OpenStack
• openvstorage-core 2.7.2-rev.3859.42b3488-1 amd64 openvStorage core
• openvstorage-hc 1.7.2-rev.675.37ca5b8-1 amd64 openvStorage Backend plugin HyperConverged
• openvstorage-health-check 2.0.0-rev.117.3212ea9-1 amd64 Open vStorage HealthCheck
• openvstorage-sdm 1.6.2-rev.330.f06c8de-1 amd64 Open vStorage Backend ASD Manager
• openvstorage-test 2.7.2-rev.980.0dbc80f-1 amd64 openvStorage autotest suite
• openvstorage-webapps 2.7.2-rev.3859.42b3488-1 amd64 openvStorage Web Applications

[wimpers]

Can your create a PR for this?

[JeffreyDevloo]

Steps

• Installed OVS using the improved SSHClient
• Started Ipython
• ran:

from ovs.extensions.generic.sshclient import SSHClient
client = SSHClient('10.100.199.152', 'root')
client.run('lsof')

The lsof was one of the commands that always hung when started.

Output

Got the output from command as expected.

u"COMMAND     PID   TID       USER   FD      TYPE             DEVICE SIZE/OFF       NODE NAME\ninit          1             root  cwd       DIR              252,0     4096          2 /\ninit          1             root  rtd       DIR              252,0     4096          2 /\ninit          1             root  txt       REG              252,0   265848     260233 /sbin/init\ninit          1             root  mem       REG              252,0    43616     390375 /lib/x86_64-linux-gnu/libnss_files-2.19.so\ninit          1             root  mem       REG              252,0    47760     390379 /lib/x86_64-linux-gnu/libnss_nis-2.19.so\ninit          1             root  mem       REG              252,0    97296     390369 /lib/x86_64-linux-gnu/libnsl-2.19.so\ninit          1             root  mem       REG              252,0    39824     390371 /lib/x86_64-linux-gnu/libnss_compat-2.19.so\ninit          1             root  mem       REG              252,0    14664     390332 /lib/x86_64-linux-gnu/libdl-2.19.so\ninit          1             root  mem       REG              252,0   252032     390391 /lib/x86_64-linux-gnu/libpcre.so.3.13.1\ninit          1             root  mem       REG              252,0   141574     390406 /lib/x86_64-linux-gnu/libpthread-2.19.so\ninit          1             root  mem       REG              252,0  1840928     390316 /lib/x86_64-linux-gnu/libc-2.19.so\ninit          1             root  mem       REG

Test result

Test passed.