paramiko / paramiko

The leading native Python SSHv2 protocol library.
http://paramiko.org
GNU Lesser General Public License v2.1
9.08k stars 2.01k forks source link

[SUPPORT] - <Paramiko Hangs forever> #2304

Closed saiaprameya closed 1 year ago

saiaprameya commented 1 year ago

Are you using paramiko as a client or server?

Client

What feature(s) aren't working right?

SSH

What version(s) of paramiko are you using?

3.3.1

What version(s) of Python are you using?

3.10.13

What operating system and version are you using?

RHEL 8.6

If you're connecting as a client, which SSH server are you connecting to?

No response

If you're using paramiko as part of another tool, which tool/version?

No response

What are you trying to do with paramiko?

We have a test automation framework which uses paramiko as the underlying library for ssh connections to the cisco product. We have the framework running multiprocess env and it occurs intermittently and hangs.

We use atfork() to close the transport channel whenever we fork a new child process because the existing connection could be used by the parent process.

Here 148 is the cisco box and connecting from RHEL 82 where our tests are being run using paramiko ssh.

Can you please let me know is there a mutex lock or somewhere thats not cleaned up. How do we debug this further?

Regards Sai

How are you trying to do it, and what's happening instead?

Following is the back trace:

retval = self.client.connect(*args, **kwargs)
args = ('10.193.169.148',)
kwargs = {'username': 'xxx', 'password': 'xxx', 'allow_agent': False, 'key_filename': None}
self = <io.Client object at 0x7f84e034e430>

File "lib/python3.8/site-packages/paramiko/client.py", line 451, in connect t.start_client(timeout=timeout) addr = ('10.193.169.148', 22) af = <AddressFamily.AF_INET: 2> allow_agent = False auth_strategy = None auth_timeout = None banner_timeout = None channel_timeout = None compress = False disabled_algorithms = None errors = {} gss_auth = False gss_deleg_creds = True gss_host = None gss_kex = False gss_trust_dns = True hostname = '10.193.169.148' key_filename = None look_for_keys = True our_server_keys = None passphrase = None password = 'xxx' pkey = None port = 22 self = <paramiko.client.SSHClient object at 0x7f84e0113070> server_hostkey_name = '10.193.169.148' sock = <socket.socket fd=191, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.193.170.82', 33688), raddr=('10.193.169.148', 22)> t = <paramiko.Transport at 0xe05c9a60 (cipher aes128-ctr, 128 bits) (connecting)> timeout = None to_try = [(<AddressFamily.AF_INET: 2>, ('10.193.169.148', 22))] transport_factory = <class 'paramiko.transport.Transport'> username = 'xxx' File "lib/python3.8/site-packages/paramiko/transport.py", line 718, in start_client event.wait(0.1) event = <threading.Event object at 0x7f84e05034c0> max_time = None self = <paramiko.Transport at 0xe05c9a60 (cipher aes128-ctr, 128 bits) (connecting)> timeout = None File "/usr/local/lib/python3.8/threading.py", line 558, in wait signaled = self._cond.wait(timeout) self = <threading.Event object at 0x7f84e05034c0> signaled = False timeout = 0.1 File "/usr/local/lib/python3.8/threading.py", line 306, in wait gotit = waiter.acquire(True, timeout) gotit = False saved_state = None self = <Condition(<unlocked _thread.lock object at 0x7f84e0503390>, 1)> timeout = 0.1 waiter = <locked _thread.lock object at 0x7f84c9ef3030>

Anything else?

No response

saiaprameya commented 1 year ago

I used pyrasite-shell to fetch more frames and here is the output if this can help: >>> for thread_id, frame in sys._current_frames().items(): print('Stack for thread {}'.format(thread_id)) stack = traceback.StackSummary.extract(traceback.walk_stack(frame), capture_locals=True) stack.reverse() traceback.print_list(stack) print('')... ... ... ... ... ... Stack for thread 140209503274752 File "/usr/local/lib/python3.8/threading.py", line 890, in _bootstrap self._bootstrap_inner() self = <ReversePythonShell(Thread-148, started 140209503274752)> File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner self.run() self = <ReversePythonShell(Thread-148, started 140209503274752)> File "<string>", line 167, in run af = <AddressFamily.AF_INET6: 10> canonname = '' proto = 6 res = (<AddressFamily.AF_INET6: 10>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('::1', 47618, 0, 0)) sa = ('::1', 47618, 0, 0) self = <ReversePythonShell(Thread-148, started 140209503274752)> socktype = <SocketKind.SOCK_STREAM: 1> File "/usr/local/lib/python3.8/code.py", line 232, in interact more = self.push(line) banner = None cprt = 'Type "help", "copyright", "credits" or "license" for more information.' exitmsg = None line = '' more = True prompt = '... ' self = <__main__.DistantInteractiveConsole object at 0x7f84e0080910> File "/usr/local/lib/python3.8/code.py", line 258, in push more = self.runsource(source, self.filename) line = '' self = <__main__.DistantInteractiveConsole object at 0x7f84e0080910> source = "for thread_id, frame in sys._current_frames().items():\n print('Stack for thread {}'.format(thread_id))\n stack = traceback.StackSummary.extract(traceback.walk_stack(frame), capture_locals=True)\n stack.reverse()\n traceback.print_list(stack)\n print('')\n" File "/usr/local/lib/python3.8/code.py", line 74, in runsource self.runcode(code) code = <code object <module> at 0x7f8510de5be0, file "<console>", line 1> filename = '<console>' self = <__main__.DistantInteractiveConsole object at 0x7f84e0080910> source = "for thread_id, frame in sys._current_frames().items():\n print('Stack for thread {}'.format(thread_id))\n stack = traceback.StackSummary.extract(traceback.walk_stack(frame), capture_locals=True)\n stack.reverse()\n traceback.print_list(stack)\n print('')\n" symbol = 'single' File "/usr/local/lib/python3.8/code.py", line 90, in runcode exec(code, self.locals) code = <code object <module> at 0x7f8510de5be0, file "<console>", line 1> self = <__main__.DistantInteractiveConsole object at 0x7f84e0080910> File "<console>", line 3, in <module> DistantInteractiveConsole = <class '__main__.DistantInteractiveConsole'> InteractiveConsole = <class 'code.InteractiveConsole'> ReverseConnection = <class '__main__.ReverseConnection'> ReversePythonConnection = <class '__main__.ReversePythonConnection'> ReversePythonShell = <class '__main__.ReversePythonShell'> StringIO = <class '_io.StringIO'> __annotations__ = {} __builtins__ = <module 'builtins' (built-in)> __cached__ = None __doc__ = '\n:mod:pyrasite.reverse` - Pyrasite Reverse Connection Payload\n=============================================================\n' file = 'runtests' loader = <_frozen_importlib_external.SourceFileLoader object at 0x7f8541dc34c0> name = 'main' package = None requires = 'qali' spec = None warningregistry = {'version': 335} debug_deadlock = <function debug_deadlock at 0x7f8510f208b0> distribution = <function distribution at 0x7f853a4d0b80> frame = <frame at 0x7f85136149a0, file '', line 3, code > importlib_load_entry_point = <function importlib_load_entry_point at 0x7f853a596280> load_entry_point = <function importlib_load_entry_point at 0x7f853a596280> lock = <_MainThread(MainThread, started 140210312636224)> pyrasite = <module 'pyrasite' from 'lib/python3.8/site-packages/pyrasite/init.py'> re = <module 're' from 'lib/python3.8/re.py'> socket = <module 'socket' from '/usr/local/lib/python3.8/socket.py'> stack = [<FrameSummary file runtests, line 33 in >, <FrameSummary file line 2951 in main>, <FrameSummary file .py, line 1301 in run>, <FrameSummary file /usr/local/lib/python3.8/asyncio/runners.py, line 44 in run>, <FrameSummary file /usr/local/lib/python3.8/asyncio/base_events.py, line 603 in run_until_complete>, <FrameSummary file /usr/local/lib/python3.8/asyncio/base_events.py, line 570 in run_forever>, <FrameSummary file /usr/local/lib/python3.8/asyncio/base_events.py, line 1859 in _run_once>, <FrameSummary file /usr/local/lib/python3.8/asyncio/events.py, line 81 in _run>, <FrameSummary file py, line 1164 in main>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/process.py, line 121 in start>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/context.py, line 224 in _Popen>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/context.py, line 277 in _Popen>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/popen_fork.py, line 19 in init>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/popen_fork.py, line 75 in _launch>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/process.py, line 315 in _bootstrap>, <FrameSummary file .py, line 761 in run>, <FrameSummary file py, line 660 in create_child>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/process.py, line 121 in start>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/context.py, line 224 in _Popen>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/context.py, line 277 in _Popen>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/popen_fork.py, line 19 in init>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/popen_fork.py, line 75 in _launch>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/process.py, line 315 in _bootstrap>, <FrameSummary file py, line 761 in run>, <FrameSummary file py, line 660 in create_child>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/process.py, line 121 in start>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/context.py, line 224 in _Popen>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/context.py, line 277 in _Popen>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/popen_fork.py, line 19 in init>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/popen_fork.py, line 75 in _launch>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/process.py, line 315 in _bootstrap>, <FrameSummary file py, line 761 in run>, <FrameSummary file py, line 660 in create_child>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/process.py, line 121 in start>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/context.py, line 224 in _Popen>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/context.py, line 277 in _Popen>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/popen_fork.py, line 19 in init>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/popen_fork.py, line 75 in _launch>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/process.py, line 315 in _bootstrap>, <FrameSummary file py, line 761 in run>, <FrameSummary file py, line 660 in create_child>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/process.py, line 121 in start>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/context.py, line 224 in _Popen>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/context.py, line 277 in _Popen>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/popen_fork.py, line 19 in init>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/popen_fork.py, line 75 in _launch>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/process.py, line 315 in _bootstrap>, <FrameSummary file py, line 761 in run>, <FrameSummary file py, line 693 in execute_method>, <FrameSummary file py, line 44 in run_test>, <FrameSummary file py, line 235 in run_tasks>, <FrameSummary file py, line 331 in wrapper>, <FrameSummary file py, line 5631 in verify_failover_server>, <FrameSummary file py, line 331 in wrapper>, <FrameSummary file py, line 5961 in verify_connectivity_failover>, <FrameSummary file py, line 5627 in vic_intersight_verify_connectivity>, <FrameSummary file py, line 456 in com>, <FrameSummary file py, line 648 in com>, <FrameSummary file py, line 226 in connect>, <FrameSummary file python3.8/site-packages/paramiko/client.py, line 451 in connect>, <FrameSummary file python3.8/site-packages/paramiko/transport.py, line 718 in start_client>, <FrameSummary file /usr/local/lib/python3.8/threading.py, line 558 in wait>, <FrameSummary file /usr/local/lib/python3.8/threading.py, line 306 in wait>] sys = <module 'sys' (built-in)> thread_id = 140209503274752 threading = <module 'threading' from '/usr/local/lib/python3.8/threading.py'> traceback = <module 'traceback' from '/usr/local/lib/python3.8/traceback.py'>

Stack for thread 140208309782272 File "/usr/local/lib/python3.8/threading.py", line 890, in _bootstrap self._bootstrap_inner() self = <paramiko.Transport at 0xe05c9a60 (cipher aes128-ctr, 128 bits) (connecting)> File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner self.run() self = <paramiko.Transport at 0xe05c9a60 (cipher aes128-ctr, 128 bits) (connecting)> File "lib/python3.8/site-packages/paramiko/transport.py", line 2151, in run self.kex_engine.parse_next(ptype, m) errormsg = None m = paramiko.Message(b'\x00\x00\x01\x17\x00\x00\x00\x07ssh-rsa\x00\x00\x00\x03\x01\x00\x01\x00\x00\x01\x01\x00\xc8\x97\xa8u\x80\t\x0b\xbdF>\x9b\xb5\xd4:2\xeb\xd1\xfa\xec\xe1\xdb\x9e\xe3i\xd8m\xf9\x0b\xe4\x9f\x94\xc2H\xa9\xf4Ns\x8eg\x078\xac\x81\xd7\xa0\xf1\xf2\xf8v\x8f\xc2pw\xb3rK\x82\x19\x7fT\xd9\xdc\x9b\x91N@`\xd1\x90\x1b\xb8S+FQ\x935\xab\x8f\xe3 \xe0\x0f\xa29\x8c\x0cg\x0c2a\x94v \x14Y\x14\xe1i\x85\x9b\x82J\xa64\x05\x9e\xd5\xd3\xcf\x9a\xa0\x9e\x8f\xdfHmz\x9b\xa0P?(j\xcf\xa1neR\xd3\x06/\'7Dv<\xc9\x8bYJ\xa0\x9f9/\xc3\xef\xaf@\xee\x8c\x1c4\xcb$\xa3\xb3\xbf\xfd\x04+4\xfb\x9a\xeb\x94\x9doCwT\xa0!\xd2r\xadL\xbaZ\x80\xe9cs\x08\x94&#\xb7\xf2\xed\xe0\x16\x81\x8e\xe0\xe8\xf0i\xd0\xb6\x87\xe3M\xdfye\xa8c\xc3)\xb0D\x14\xa5eF\x8d6&\xdc2]\xe0\xd18\x9fIp\xf0\x15\x9d\xa39y\xf4\xf25\xc3O\'\xd4\xa2\'\xec\x82Ug\xc2\xc5\x89[\xca-\xd3\x86\x9f\x00\x00\x00A\x04\r\r\xa1LL|2\x87\xed\xf1\xd4r\x1a,@\xd6\x87\xd5C!\xf7v\x0c\x8fI\xfb\nNS2\xf5\x8a\xad\xe4\x15tH\xf0\xc5\x84\x9e\xaasB\x8f\xe7\xfc\xcb\xb7\xde\x7f\xcf\x8f\x81\xf6\xe2\xf4\xa7n \xa5\xc0\x00\x00\x01\x14\x00\x00\x00\x0crsa-sha2-256\x00\x00\x01\x00\xc1\x12\x0bvy\xbb\xdc\xac\xb1|\xc3\xde\xc3\xcf\xc5\xb7\xb0\x1d\xc1\xb5.=\x04\xa1\x0cb\x8e\x0e\xce\x12"\xa2\xc8wsh{F\x04\xf4!\xc4YJ\xbfR:::h;\x12\xa8\xa9\xdb\xf7\xfd\xe3o\xde\x19\xad\xeb\x84l)H2\xdf7\xcf\xd3\xa3\xab)\x98gN@\xe76b~_}\xd6\xe3\x04)\xdbT\x8a"\x85Ve\xe8\xd8\x92[\xe0\xe5;#\xd0*8\xbd\x82\x14\xcb\xe4\xd7\x1e12\xae\xec\xed\x92\x9d<\x02\x942\xb3\xf4\x1b|\r\xfbV\xb7\x10U.cY4\xc2\x13\x93\xe0s\xc4\xfb\xdc\xeaE\xc1r\xb9@x>.a\xf2\xca\xe0\x8a\x9a\xdc\xcf\xd8\x11\x0b\xbdT\xe4\xcf\xd5\xd9xWD\xd3\xae6\x1e{L\x96\xc4\xc5\xf9\xc6\x89\x11ar*\x8fU\xdf\xd0a\xb2@\x90|"8\r\x90\x9aM\xc7z\xad\x08\xf6\xb3<\x9f\xf7\x18\x01\xf0!\xecfX\x19\x88\xf8-\xa2(\xe3\x93\x00\xf7\xd7\xa0;\xcf\xa0C\x17.K\xdb\x1d6\xb6I\xd9\x1c^\x9b\xaf\xb9\x1f,') ptype = 31 self = <paramiko.Transport at 0xe05c9a60 (cipher aes128-ctr, 128 bits) (connecting)> tid = '0xe05c9a60' File "lib/python3.8/site-packages/paramiko/kex_ecdh_nist.py", line 53, in parse_next return self._parse_kexecdhreply(m) m = paramiko.Message(b'\x00\x00\x01\x17\x00\x00\x00\x07ssh-rsa\x00\x00\x00\x03\x01\x00\x01\x00\x00\x01\x01\x00\xc8\x97\xa8u\x80\t\x0b\xbdF>\x9b\xb5\xd4:2\xeb\xd1\xfa\xec\xe1\xdb\x9e\xe3i\xd8m\xf9\x0b\xe4\x9f\x94\xc2H\xa9\xf4Ns\x8eg\x078\xac\x81\xd7\xa0\xf1\xf2\xf8v\x8f\xc2pw\xb3rK\x82\x19\x7fT\xd9\xdc\x9b\x91N@`\xd1\x90\x1b\xb8S+FQ\x935\xab\x8f\xe3 \xe0\x0f\xa29\x8c\x0cg\x0c2a\x94v \x14Y\x14\xe1i\x85\x9b\x82J\xa64\x05\x9e\xd5\xd3\xcf\x9a\xa0\x9e\x8f\xdfHmz\x9b\xa0P?(j\xcf\xa1neR\xd3\x06/\'7Dv<\xc9\x8bYJ\xa0\x9f9/\xc3\xef\xaf@\xee\x8c\x1c4\xcb$\xa3\xb3\xbf\xfd\x04+4\xfb\x9a\xeb\x94\x9doCwT\xa0!\xd2r\xadL\xbaZ\x80\xe9cs\x08\x94&#\xb7\xf2\xed\xe0\x16\x81\x8e\xe0\xe8\xf0i\xd0\xb6\x87\xe3M\xdfye\xa8c\xc3)\xb0D\x14\xa5eF\x8d6&\xdc2]\xe0\xd18\x9fIp\xf0\x15\x9d\xa39y\xf4\xf25\xc3O\'\xd4\xa2\'\xec\x82Ug\xc2\xc5\x89[\xca-\xd3\x86\x9f\x00\x00\x00A\x04\r\r\xa1LL|2\x87\xed\xf1\xd4r\x1a,@\xd6\x87\xd5C!\xf7v\x0c\x8fI\xfb\nNS2\xf5\x8a\xad\xe4\x15tH\xf0\xc5\x84\x9e\xaasB\x8f\xe7\xfc\xcb\xb7\xde\x7f\xcf\x8f\x81\xf6\xe2\xf4\xa7n \xa5\xc0\x00\x00\x01\x14\x00\x00\x00\x0crsa-sha2-256\x00\x00\x01\x00\xc1\x12\x0bvy\xbb\xdc\xac\xb1|\xc3\xde\xc3\xcf\xc5\xb7\xb0\x1d\xc1\xb5.=\x04\xa1\x0cb\x8e\x0e\xce\x12"\xa2\xc8wsh{F\x04\xf4!\xc4YJ\xbfR:::h;\x12\xa8\xa9\xdb\xf7\xfd\xe3o\xde\x19\xad\xeb\x84l)H2\xdf7\xcf\xd3\xa3\xab)\x98gN@\xe76b~_}\xd6\xe3\x04)\xdbT\x8a"\x85Ve\xe8\xd8\x92[\xe0\xe5;#\xd0*8\xbd\x82\x14\xcb\xe4\xd7\x1e12\xae\xec\xed\x92\x9d<\x02\x942\xb3\xf4\x1b|\r\xfbV\xb7\x10U.cY4\xc2\x13\x93\xe0s\xc4\xfb\xdc\xeaE\xc1r\xb9@x>.a\xf2\xca\xe0\x8a\x9a\xdc\xcf\xd8\x11\x0b\xbdT\xe4\xcf\xd5\xd9xWD\xd3\xae6\x1e{L\x96\xc4\xc5\xf9\xc6\x89\x11ar*\x8fU\xdf\xd0a\xb2@\x90|"8\r\x90\x9aM\xc7z\xad\x08\xf6\xb3<\x9f\xf7\x18\x01\xf0!\xecfX\x19\x88\xf8-\xa2(\xe3\x93\x00\xf7\xd7\xa0;\xcf\xa0C\x17.K\xdb\x1d6\xb6I\xd9\x1c^\x9b\xaf\xb9\x1f,') ptype = 31 self = <paramiko.kex_ecdh_nist.KexNistp256 object at 0x7f84e0595340> File "lib/python3.8/site-packages/paramiko/kex_ecdh_nist.py", line 117, in _parse_kexecdh_reply K = self.P.exchange(ec.ECDH(), self.Q_S) K_S = b"\x00\x00\x00\x07ssh-rsa\x00\x00\x00\x03\x01\x00\x01\x00\x00\x01\x01\x00\xc8\x97\xa8u\x80\t\x0b\xbdF>\x9b\xb5\xd4:2\xeb\xd1\xfa\xec\xe1\xdb\x9e\xe3i\xd8m\xf9\x0b\xe4\x9f\x94\xc2H\xa9\xf4Ns\x8eg\x078\xac\x81\xd7\xa0\xf1\xf2\xf8v\x8f\xc2pw\xb3rK\x82\x19\x7fT\xd9\xdc\x9b\x91N@\xd1\x90\x1b\xb8S+FQ\x935\xab\x8f\xe3 \xe0\x0f\xa29\x8c\x0cg\x0c2a\x94v \x14Y\x14\xe1i\x85\x9b\x82J\xa64\x05\x9e\xd5\xd3\xcf\x9a\xa0\x9e\x8f\xdfHmz\x9b\xa0P?(j\xcf\xa1neR\xd3\x06/'7Dv<\xc9\x8bYJ\xa0\x9f9/\xc3\xef\xaf@\xee\x8c\x1c4\xcb$\xa3\xb3\xbf\xfd\x04+4\xfb\x9a\xeb\x94\x9doCwT\xa0!\xd2r\xadL\xbaZ\x80\xe9cs\x08\x94&#\xb7\xf2\xed\xe0\x16\x81\x8e\xe0\xe8\xf0i\xd0\xb6\x87\xe3M\xdfye\xa8c\xc3)\xb0D\x14\xa5eF\x8d6&\xdc2]\xe0\xd18\x9fIp\xf0\x15\x9d\xa39y\xf4\xf25\xc3O'\xd4\xa2'\xec\x82Ug\xc2\xc5\x89[\xca-\xd3\x86\x9f" Q_S_bytes = b'\x04\r_\r\xa1LL|2\x87\xed\xf1\xd4r\x1a,@\xd6\x87\xd5C!\xf7v\x0c\x8fI\xfb\nNS2\xf5\x8a\xad\xe4\x15tH\xf0\xc5\x84\x9e\xaasB\x8f\xe7\xfc\xcb\xb7\xde\x7f\xcf\x8f\x81\xf6\xe2\xf4\xa7n \xa5\xc0' m = paramiko.Message(b'\x00\x00\x01\x17\x00\x00\x00\x07ssh-rsa\x00\x00\x00\x03\x01\x00\x01\x00\x00\x01\x01\x00\xc8\x97\xa8u\x80\t\x0b\xbdF>\x9b\xb5\xd4:2\xeb\xd1\xfa\xec\xe1\xdb\x9e\xe3i\xd8m\xf9\x0b\xe4\x9f\x94\xc2H\xa9\xf4Ns\x8eg\x078\xac\x81\xd7\xa0\xf1\xf2\xf8v\x8f\xc2pw\xb3rK\x82\x19\x7fT\xd9\xdc\x9b\x91N@\xd1\x90\x1b\xb8S+FQ\x935\xab\x8f\xe3 \xe0\x0f\xa29\x8c\x0cg\x0c2a\x94v \x14Y\x14\xe1i\x85\x9b\x82J\xa64\x05\x9e\xd5\xd3\xcf\x9a\xa0\x9e\x8f\xdfHmz\x9b\xa0P?(j\xcf\xa1neR\xd3\x06/\'7Dv<\xc9\x8bYJ\xa0\x9f9/\xc3\xef\xaf@\xee\x8c\x1c4\xcb$\xa3\xb3\xbf\xfd\x04+4\xfb\x9a\xeb\x94\x9doCwT\xa0!\xd2r\xadL\xbaZ\x80\xe9cs\x08\x94&#\xb7\xf2\xed\xe0\x16\x81\x8e\xe0\xe8\xf0i\xd0\xb6\x87\xe3M\xdfye\xa8c\xc3)\xb0D\x14\xa5eF\x8d6&\xdc2]\xe0\xd18\x9fIp\xf0\x15\x9d\xa39y\xf4\xf25\xc3O\'\xd4\xa2\'\xec\x82Ug\xc2\xc5\x89[\xca-\xd3\x86\x9f\x00\x00\x00A\x04\r_\r\xa1LL|2\x87\xed\xf1\xd4r\x1a,@\xd6\x87\xd5C!\xf7v\x0c\x8fI\xfb\nNS2\xf5\x8a\xad\xe4\x15tH\xf0\xc5\x84\x9e\xaasB\x8f\xe7\xfc\xcb\xb7\xde\x7f\xcf\x8f\x81\xf6\xe2\xf4\xa7n \xa5\xc0\x00\x00\x01\x14\x00\x00\x00\x0crsa-sha2-256\x00\x00\x01\x00\xc1\x12\x0bvy\xbb\xdc\xac\xb1|\xc3\xde\xc3\xcf\xc5\xb7\xb0\x1d\xc1\xb5.=\x04\xa1\x0cb\x8e\x0e\xce\x12"\xa2\xc8wsh{F\x04\xf4!\xc4YJ\xbfR:::h;\x12\xa8\xa9\xdb\xf7\xfd\xe3o\xde\x19\xad\xeb\x84l)H2\xdf7\xcf\xd3\xa3\xab)\x98gN@\xe76b~_}\xd6\xe3\x04)\xdbT\x8a"\x85Ve\xe8\xd8\x92[\xe0\xe5;#\xd08\xbd\x82\x14\xcb\xe4\xd7\x1e12\xae\xec\xed\x92\x9d\x02\x942\xb3\xf4\x1b|\r\xfbV\xb7\x10U.cY4\xc2\x13\x93\xe0s\xc4\xfb\xdc\xeaE\xc1r\xb9@x.a\xf2\xca\xe0\x8a\x9a\xdc\xcf\xd8\x11\x0b\xbdT\xe4\xcf\xd5\xd9xWD\xd3\xae6\x1e{L\x96\xc4\xc5\xf9\xc6\x89\x11ar\x8fU\xdf\xd0a\xb2@\x90|"8\r\x90\x9aM\xc7z\xad\x08\xf6\xb3<\x9f\xf7\x18\x01\xf0!\xecfX\x19\x88\xf8-\xa2(\xe3\x93\x00\xf7\xd7\xa0;\xcf\xa0C\x17.K\xdb\x1d6\xb6I\xd9\x1c^\x9b\xaf\xb9\x1f,') self = <paramiko.kex_ecdh_nist.KexNistp256 object at 0x7f84e0595340> sig = b'\x00\x00\x00\x0crsa-sha2-256\x00\x00\x01\x00\xc1\x12\x0bvy\xbb\xdc\xac\xb1|\xc3\xde\xc3\xcf\xc5\xb7\xb0\x1d\xc1\xb5.=\x04\xa1\x0cb\x8e\x0e\xce\x12"\xa2\xc8wsh{F\x04\xf4!\xc4YJ\xbfR:::h;\x12\xa8\xa9\xdb\xf7\xfd\xe3o\xde\x19\xad\xeb\x84l)H2\xdf7\xcf\xd3\xa3\xab)\x98gN@\xe76b~_}\xd6\xe3\x04)\xdbT\x8a"\x85Ve\xe8\xd8\x92[\xe0\xe5;#\xd0*8\xbd\x82\x14\xcb\xe4\xd7\x1e12\xae\xec\xed\x92\x9d<\x02\x942\xb3\xf4\x1b|\r\xfbV\xb7\x10U.cY4\xc2\x13\x93\xe0s\xc4\xfb\xdc\xeaE\xc1r\xb9@x>.a\xf2\xca\xe0\x8a\x9a\xdc\xcf\xd8\x11\x0b\xbdT\xe4\xcf\xd5\xd9xWD\xd3\xae6\x1e{L\x96\xc4\xc5\xf9\xc6\x89\x11ar*\x8fU\xdf\xd0a\xb2@\x90|"8\r\x90\x9aM\xc7z\xad\x08\xf6\xb3<\x9f\xf7\x18\x01\xf0!\xecfX\x19\x88\xf8-\xa2(\xe3\x93\x00\xf7\xd7\xa0;\xcf\xa0C\x17.K\xdb\x1d6\xb6I\xd9\x1c^\x9b\xaf\xb9\x1f,' File "lib/python3.8/site-packages/cryptography/hazmat/backends/openssl/ec.py", line 162, in exchange return _evp_pkey_derive(self._backend, self._evp_pkey, peer_public_key) algorithm = <cryptography.hazmat.primitives.asymmetric.ec.ECDH object at 0x7f84e0595610> peer_public_key = <cryptography.hazmat.backends.openssl.ec._EllipticCurvePublicKey object at 0x7f84e05955e0> self = <cryptography.hazmat.backends.openssl.ec._EllipticCurvePrivateKey object at 0x7f84e0595460> File "lib/python3.8/site-packages/cryptography/hazmat/backends/openssl/utils.py", line 18, in _evp_pkey_derive res = backend._lib.EVP_PKEY_derive_init(ctx) backend = <OpenSSLBackend(version: OpenSSL 3.0.3 3 May 2022, FIPS: False)> ctx = <cdata 'EVP_PKEY_CTX ' 0x7f8504007490> evp_pkey = <cdata 'EVP_PKEY ' 0x7f850400b1e0> peer_public_key = <cryptography.hazmat.backends.openssl.ec._EllipticCurvePublicKey object at 0x7f84e05955e0>

Stack for thread 140208334960384 File "/usr/local/lib/python3.8/threading.py", line 890, in _bootstrap self._bootstrap_inner() self = <Thread(QueueFeederThread, started daemon 140208334960384)> File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner self.run() self = <Thread(QueueFeederThread, started daemon 140208334960384)> File "/usr/local/lib/python3.8/threading.py", line 870, in run self._target(*self._args, **self._kwargs) self = <Thread(QueueFeederThread, started daemon 140208334960384)> File "/usr/local/lib/python3.8/multiprocessing/queues.py", line 227, in _feed nwait() bpopleft = <built-in method popleft of collections.deque object at 0x7f84f88cc8e0> buffer = deque([]) close = <bound method _ConnectionBase.close of <multiprocessing.connection.Connection object at 0x7f84f89a4c10>> ignore_epipe = False nacquire = <built-in method acquire of _thread.lock object at 0x7f84c9e7c5a0> notempty = <Condition(<unlocked _thread.lock object at 0x7f84c9e7c5a0>, 1)> nrelease = <built-in method release of _thread.lock object at 0x7f84c9e7c5a0> nwait = <bound method Condition.wait of <Condition(<unlocked _thread.lock object at 0x7f84c9e7c5a0>, 1)>> obj = <memory at 0x7f8510d12e80> onerror = <function Queue._on_queue_feeder_error at 0x7f84f8ab68b0> queue_sem = <BoundedSemaphore(value=2147483647, maxvalue=2147483647)> send_bytes = <bound method _ConnectionBase.send_bytes of <multiprocessing.connection.Connection object at 0x7f84f89a4c10>> sentinel = <object object at 0x7f85148e7670> wacquire = <built-in method acquire of _multiprocessing.SemLock object at 0x7f84f894af30> wrelease = <built-in method release of _multiprocessing.SemLock object at 0x7f84f894af30> writelock = <Lock(owner=None)> File "/usr/local/lib/python3.8/threading.py", line 302, in wait waiter.acquire() gotit = False saved_state = None self = <Condition(<unlocked _thread.lock object at 0x7f84c9e7c5a0>, 1)> timeout = None waiter = <locked _thread.lock object at 0x7f84e04a25a0> `

jun66j5 commented 1 year ago

@saiaprameya Could you reformat your codes and errors using code blocks? It is really hard to read without fixed-width code formatting.

saiaprameya commented 1 year ago

Here it is:

    retval = self.client.connect(*args, **kwargs)
    args = ('10.193.169.148',)
    kwargs = {'username': 'xxx', 'password': 'xxx', 'allow_agent': False, 'key_filename': None}
    self = <io.Client object at 0x7f84e034e430>
  File "lib/python3.8/site-packages/paramiko/client.py", line 451, in connect
    t.start_client(timeout=timeout)
    addr = ('10.193.169.148', 22)
    af = <AddressFamily.AF_INET: 2>
    allow_agent = False
    auth_strategy = None
    auth_timeout = None
    banner_timeout = None
    channel_timeout = None
    compress = False
    disabled_algorithms = None
    errors = {}
    gss_auth = False
    gss_deleg_creds = True
    gss_host = None
    gss_kex = False
    gss_trust_dns = True
    hostname = '10.193.169.148'
    key_filename = None
    look_for_keys = True
    our_server_keys = None
    passphrase = None
    password = 'xxx'
    pkey = None
    port = 22
    self = <paramiko.client.SSHClient object at 0x7f84e0113070>
    server_hostkey_name = '10.193.169.148'
    sock = <socket.socket fd=191, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.193.170.82', 33688), raddr=('10.193.169.148', 22)>
    t = <paramiko.Transport at 0xe05c9a60 (cipher aes128-ctr, 128 bits) (connecting)>
    timeout = None
    to_try = [(<AddressFamily.AF_INET: 2>, ('10.193.169.148', 22))]
    transport_factory = <class 'paramiko.transport.Transport'>
    username = 'xxx'
  File "lib/python3.8/site-packages/paramiko/transport.py", line 718, in start_client
    event.wait(0.1)
    event = <threading.Event object at 0x7f84e05034c0>
    max_time = None
    self = <paramiko.Transport at 0xe05c9a60 (cipher aes128-ctr, 128 bits) (connecting)>
    timeout = None
  File "/usr/local/lib/python3.8/threading.py", line 558, in wait
    signaled = self._cond.wait(timeout)
    self = <threading.Event object at 0x7f84e05034c0>
    signaled = False
    timeout = 0.1
  File "/usr/local/lib/python3.8/threading.py", line 306, in wait
    gotit = waiter.acquire(True, timeout)
    gotit = False
    saved_state = None
    self = <Condition(<unlocked _thread.lock object at 0x7f84e0503390>, 1)>
    timeout = 0.1
    waiter = <locked _thread.lock object at 0x7f84c9ef3030>
saiaprameya commented 1 year ago

pyrasite-shell to fetch more frames and here is the output if this can help:

 >>> for thread_id, frame in sys._current_frames().items():
   print('Stack for thread {}'.format(thread_id))
   stack = traceback.StackSummary.extract(traceback.walk_stack(frame), capture_locals=True)
   stack.reverse()
   traceback.print_list(stack)
   print('')... ... ... ... ... 
... 
Stack for thread 140209503274752
  File "/usr/local/lib/python3.8/threading.py", line 890, in _bootstrap
    self._bootstrap_inner()
    self = <ReversePythonShell(Thread-148, started 140209503274752)>
  File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
    self = <ReversePythonShell(Thread-148, started 140209503274752)>
  File "<string>", line 167, in run
    af = <AddressFamily.AF_INET6: 10>
    canonname = ''
    proto = 6
    res = (<AddressFamily.AF_INET6: 10>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('::1', 47618, 0, 0))
    sa = ('::1', 47618, 0, 0)
    self = <ReversePythonShell(Thread-148, started 140209503274752)>
    socktype = <SocketKind.SOCK_STREAM: 1>
  File "/usr/local/lib/python3.8/code.py", line 232, in interact
    more = self.push(line)
    banner = None
    cprt = 'Type "help", "copyright", "credits" or "license" for more information.'
    exitmsg = None
    line = ''
    more = True
    prompt = '... '
    self = <__main__.DistantInteractiveConsole object at 0x7f84e0080910>
  File "/usr/local/lib/python3.8/code.py", line 258, in push
    more = self.runsource(source, self.filename)
    line = ''
    self = <__main__.DistantInteractiveConsole object at 0x7f84e0080910>
    source = "for thread_id, frame in sys._current_frames().items():\n   print('Stack for thread {}'.format(thread_id))\n   stack = traceback.StackSummary.extract(traceback.walk_stack(frame), capture_locals=True)\n   stack.reverse()\n   traceback.print_list(stack)\n   print('')\n"
  File "/usr/local/lib/python3.8/code.py", line 74, in runsource
    self.runcode(code)
    code = <code object <module> at 0x7f8510de5be0, file "<console>", line 1>
    filename = '<console>'
    self = <__main__.DistantInteractiveConsole object at 0x7f84e0080910>
    source = "for thread_id, frame in sys._current_frames().items():\n   print('Stack for thread {}'.format(thread_id))\n   stack = traceback.StackSummary.extract(traceback.walk_stack(frame), capture_locals=True)\n   stack.reverse()\n   traceback.print_list(stack)\n   print('')\n"
    symbol = 'single'
  File "/usr/local/lib/python3.8/code.py", line 90, in runcode
    exec(code, self.locals)
    code = <code object <module> at 0x7f8510de5be0, file "<console>", line 1>
    self = <__main__.DistantInteractiveConsole object at 0x7f84e0080910>
  File "<console>", line 3, in <module>
    DistantInteractiveConsole = <class '__main__.DistantInteractiveConsole'>
    InteractiveConsole = <class 'code.InteractiveConsole'>
    ReverseConnection = <class '__main__.ReverseConnection'>
    ReversePythonConnection = <class '__main__.ReversePythonConnection'>
    ReversePythonShell = <class '__main__.ReversePythonShell'>
    StringIO = <class '_io.StringIO'>
    __annotations__ = {}
    __builtins__ = <module 'builtins' (built-in)>
    __cached__ = None
    __doc__ = '\n:mod:`pyrasite.reverse` - Pyrasite Reverse Connection Payload\n=============================================================\n'
    __file__ = 'runtests'
    __loader__ = <_frozen_importlib_external.SourceFileLoader object at 0x7f8541dc34c0>
    __name__ = '__main__'
    __package__ = None
    __requires__ = 'qali'
    __spec__ = None
    __warningregistry__ = {'version': 335}
    debug_deadlock = <function debug_deadlock at 0x7f8510f208b0>
    distribution = <function distribution at 0x7f853a4d0b80>
    frame = <frame at 0x7f85136149a0, file '<console>', line 3, code <module>>
    importlib_load_entry_point = <function importlib_load_entry_point at 0x7f853a596280>
    load_entry_point = <function importlib_load_entry_point at 0x7f853a596280>
    lock = <_MainThread(MainThread, started 140210312636224)>
    pyrasite = <module 'pyrasite' from 'lib/python3.8/site-packages/pyrasite/__init__.py'>
    re = <module 're' from 'lib/python3.8/re.py'>
    socket = <module 'socket' from '/usr/local/lib/python3.8/socket.py'>
    stack = [<FrameSummary file runtests, line 33 in <module>>, <FrameSummary file line 2951 in main>, <FrameSummary file .py, line 1301 in run>, <FrameSummary file /usr/local/lib/python3.8/asyncio/runners.py, line 44 in run>, <FrameSummary file /usr/local/lib/python3.8/asyncio/base_events.py, line 603 in run_until_complete>, <FrameSummary file /usr/local/lib/python3.8/asyncio/base_events.py, line 570 in run_forever>, <FrameSummary file /usr/local/lib/python3.8/asyncio/base_events.py, line 1859 in _run_once>, <FrameSummary file /usr/local/lib/python3.8/asyncio/events.py, line 81 in _run>, <FrameSummary file py, line 1164 in main>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/process.py, line 121 in start>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/context.py, line 224 in _Popen>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/context.py, line 277 in _Popen>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/popen_fork.py, line 19 in __init__>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/popen_fork.py, line 75 in _launch>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/process.py, line 315 in _bootstrap>, <FrameSummary file .py, line 761 in run>, <FrameSummary file py, line 660 in create_child>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/process.py, line 121 in start>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/context.py, line 224 in _Popen>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/context.py, line 277 in _Popen>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/popen_fork.py, line 19 in __init__>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/popen_fork.py, line 75 in _launch>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/process.py, line 315 in _bootstrap>, <FrameSummary file 
    py, line 761 in run>, <FrameSummary file py, line 660 in create_child>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/process.py, line 121 in start>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/context.py, line 224 in _Popen>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/context.py, line 277 in _Popen>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/popen_fork.py, line 19 in __init__>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/popen_fork.py, line 75 in _launch>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/process.py, line 315 in _bootstrap>, <FrameSummary file py, line 761 in run>, <FrameSummary file py, line 660 in create_child>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/process.py, line 121 in start>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/context.py, line 224 in _Popen>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/context.py, line 277 in _Popen>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/popen_fork.py, line 19 in __init__>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/popen_fork.py, line 75 in _launch>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/process.py, line 315 in _bootstrap>, <FrameSummary file py, line 761 in run>, <FrameSummary file py, line 660 in create_child>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/process.py, line 121 in start>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/context.py, line 224 in _Popen>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/context.py, line 277 in _Popen>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/popen_fork.py, line 19 in __init__>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/popen_fork.py, line 75 in _launch>, <FrameSummary file /usr/local/lib/python3.8/multiprocessing/process.py, line 315 in _bootstrap>, <FrameSummary file py, line 761 in run>, <FrameSummary file py, line 693 in execute_method>, <FrameSummary file py, line 44 in run_test>, <FrameSummary file py, line 235 in run_tasks>, <FrameSummary file py, line 331 in wrapper>, <FrameSummary file py, line 5631 in verify_failover_server>, <FrameSummary file py, line 331 in wrapper>, <FrameSummary file py, line 5961 in verify_connectivity_failover>, <FrameSummary file /home/rajraghu/dev/proj3/qali-test/teams/vic/intersight/library/common/server_utils.py, line 5627 in vic_intersight_verify_connectivity>, <FrameSummary file /home/rajraghu/dev/qali-infra/qali/ucs/nxos/__init__.py, line 456 in com>, <FrameSummary file /home/rajraghu/dev/qali-infra/qali/interface/io.py, line 648 in com>, <FrameSummary file /home/rajraghu/dev/qali-infra/qali/interface/io.py, line 226 in connect>, <FrameSummary file /home/rajraghu/Envs/proj3/lib/python3.8/site-packages/paramiko/client.py, line 451 in connect>, <FrameSummary file /home/rajraghu/Envs/proj3/lib/python3.8/site-packages/paramiko/transport.py, line 718 in start_client>, <FrameSummary file /usr/local/lib/python3.8/threading.py, line 558 in wait>, <FrameSummary file /usr/local/lib/python3.8/threading.py, line 306 in wait>]
    sys = <module 'sys' (built-in)>
    thread_id = 140209503274752
    threading = <module 'threading' from '/usr/local/lib/python3.8/threading.py'>
    traceback = <module 'traceback' from '/usr/local/lib/python3.8/traceback.py'>

Stack for thread 140208309782272
  File "/usr/local/lib/python3.8/threading.py", line 890, in _bootstrap
    self._bootstrap_inner()
    self = <paramiko.Transport at 0xe05c9a60 (cipher aes128-ctr, 128 bits) (connecting)>
  File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
    self = <paramiko.Transport at 0xe05c9a60 (cipher aes128-ctr, 128 bits) (connecting)>
  File "lib/python3.8/site-packages/paramiko/transport.py", line 2151, in run
    self.kex_engine.parse_next(ptype, m)
    error_msg = None
    m = paramiko.Message(b'\x00\x00\x01\x17\x00\x00\x00\x07ssh-rsa\x00\x00\x00\x03\x01\x00\x01\x00\x00\x01\x01\x00\xc8\x97\xa8u\x80\t\x0b\xbdF>\x9b\xb5\xd4:2\xeb\xd1\xfa\xec\xe1\xdb\x9e\xe3i\xd8m\xf9\x0b\xe4\x9f\x94\xc2H\xa9\xf4Ns\x8eg\x078\xac\x81\xd7\xa0\xf1\xf2\xf8v\x8f\xc2pw\xb3rK\x82\x19\x7fT\xd9\xdc\x9b\x91N@`\xd1\x90\x1b\xb8S+FQ\x935\xab\x8f\xe3 \xe0\x0f\xa29\x8c\x0cg\x0c2a\x94v \x14Y\x14\xe1i\x85\x9b\x82J\xa64\x05\x9e\xd5\xd3\xcf\x9a\xa0\x9e\x8f\xdfHmz\x9b\xa0P?(j\xcf\xa1neR\xd3\x06/\'7Dv<\xc9\x8bYJ\xa0\x9f9/\xc3\xef\xaf@\xee\x8c\x1c4\xcb$\xa3\xb3\xbf\xfd\x04+4\xfb\x9a\xeb\x94\x9doCwT\xa0!\xd2r\xadL\xbaZ\x80\xe9cs\x08\x94&#\xb7\xf2\xed\xe0\x16\x81\x8e\xe0\xe8\xf0i\xd0\xb6\x87\xe3M\xdfye\xa8c\xc3)\xb0D\x14\xa5eF\x8d6&\xdc2]\xe0\xd18\x9fIp\xf0\x15\x9d\xa39y\xf4\xf25\xc3O\'\xd4\xa2\'\xec\x82Ug\xc2\xc5\x89[\xca-\xd3\x86\x9f\x00\x00\x00A\x04\r_\r\xa1LL|2\x87\xed\xf1\xd4r\x1a,@\xd6\x87\xd5C!\xf7v\x0c\x8fI\xfb\nNS2\xf5\x8a\xad\xe4\x15tH`\xf0\xc5\x84\x9e\xaasB\x8f\xe7\xfc\xcb\xb7\xde\x7f\xcf\x8f\x81\xf6\xe2\xf4\xa7n \xa5\xc0\x00\x00\x01\x14\x00\x00\x00\x0crsa-sha2-256\x00\x00\x01\x00\xc1\x12\x0bvy\xbb\xdc\xac\xb1|\xc3\xde\xc3\xcf\xc5\xb7\xb0\x1d\xc1\xb5.=\x04\xa1\x0cb\x8e\x0e\xce\x12"\xa2\xc8wsh{F\x04\xf4!\xc4YJ\xbfR:::h;\x12\xa8\xa9\xdb\xf7\xfd\xe3o\xde\x19\xad\xeb\x84l)H2\xdf7\xcf\xd3\xa3\xab)\x98gN@\xe76b~_}\xd6\xe3\x04)\xdbT\x8a"\x85Ve\xe8\xd8\x92[\xe0\xe5;#\xd0*8\xbd\x82\x14\xcb\xe4\xd7\x1e12\xae\xec\xed\x92\x9d<\x02\x942\xb3\xf4\x1b|\r\xfbV\xb7\x10U.cY4\xc2\x13\x93\xe0s\xc4\xfb\xdc\xeaE\xc1r\xb9@x>.a\xf2\xca\xe0\x8a\x9a\xdc\xcf\xd8\x11\x0b\xbdT\xe4\xcf\xd5\xd9xWD\xd3\xae6\x1e{L\x96\xc4\xc5\xf9\xc6\x89\x11ar*\x8fU\xdf\xd0a`\xb2@\x90|"8\r\x90\x9aM\xc7z\xad\x08\xf6\xb3<\x9f\xf7\x18\x01\xf0!\xecfX\x19\x88\xf8-\xa2(\xe3\x93\x00\xf7\xd7\xa0;\xcf\xa0C\x17.K\xdb\x1d6\xb6I\xd9\x1c^\x9b\xaf\xb9\x1f,')
    ptype = 31
    self = <paramiko.Transport at 0xe05c9a60 (cipher aes128-ctr, 128 bits) (connecting)>
    tid = '0xe05c9a60'
  File "lib/python3.8/site-packages/paramiko/kex_ecdh_nist.py", line 53, in parse_next
    return self._parse_kexecdh_reply(m)
    m = paramiko.Message(b'\x00\x00\x01\x17\x00\x00\x00\x07ssh-rsa\x00\x00\x00\x03\x01\x00\x01\x00\x00\x01\x01\x00\xc8\x97\xa8u\x80\t\x0b\xbdF>\x9b\xb5\xd4:2\xeb\xd1\xfa\xec\xe1\xdb\x9e\xe3i\xd8m\xf9\x0b\xe4\x9f\x94\xc2H\xa9\xf4Ns\x8eg\x078\xac\x81\xd7\xa0\xf1\xf2\xf8v\x8f\xc2pw\xb3rK\x82\x19\x7fT\xd9\xdc\x9b\x91N@`\xd1\x90\x1b\xb8S+FQ\x935\xab\x8f\xe3 \xe0\x0f\xa29\x8c\x0cg\x0c2a\x94v \x14Y\x14\xe1i\x85\x9b\x82J\xa64\x05\x9e\xd5\xd3\xcf\x9a\xa0\x9e\x8f\xdfHmz\x9b\xa0P?(j\xcf\xa1neR\xd3\x06/\'7Dv<\xc9\x8bYJ\xa0\x9f9/\xc3\xef\xaf@\xee\x8c\x1c4\xcb$\xa3\xb3\xbf\xfd\x04+4\xfb\x9a\xeb\x94\x9doCwT\xa0!\xd2r\xadL\xbaZ\x80\xe9cs\x08\x94&#\xb7\xf2\xed\xe0\x16\x81\x8e\xe0\xe8\xf0i\xd0\xb6\x87\xe3M\xdfye\xa8c\xc3)\xb0D\x14\xa5eF\x8d6&\xdc2]\xe0\xd18\x9fIp\xf0\x15\x9d\xa39y\xf4\xf25\xc3O\'\xd4\xa2\'\xec\x82Ug\xc2\xc5\x89[\xca-\xd3\x86\x9f\x00\x00\x00A\x04\r_\r\xa1LL|2\x87\xed\xf1\xd4r\x1a,@\xd6\x87\xd5C!\xf7v\x0c\x8fI\xfb\nNS2\xf5\x8a\xad\xe4\x15tH`\xf0\xc5\x84\x9e\xaasB\x8f\xe7\xfc\xcb\xb7\xde\x7f\xcf\x8f\x81\xf6\xe2\xf4\xa7n \xa5\xc0\x00\x00\x01\x14\x00\x00\x00\x0crsa-sha2-256\x00\x00\x01\x00\xc1\x12\x0bvy\xbb\xdc\xac\xb1|\xc3\xde\xc3\xcf\xc5\xb7\xb0\x1d\xc1\xb5.=\x04\xa1\x0cb\x8e\x0e\xce\x12"\xa2\xc8wsh{F\x04\xf4!\xc4YJ\xbfR:::h;\x12\xa8\xa9\xdb\xf7\xfd\xe3o\xde\x19\xad\xeb\x84l)H2\xdf7\xcf\xd3\xa3\xab)\x98gN@\xe76b~_}\xd6\xe3\x04)\xdbT\x8a"\x85Ve\xe8\xd8\x92[\xe0\xe5;#\xd0*8\xbd\x82\x14\xcb\xe4\xd7\x1e12\xae\xec\xed\x92\x9d<\x02\x942\xb3\xf4\x1b|\r\xfbV\xb7\x10U.cY4\xc2\x13\x93\xe0s\xc4\xfb\xdc\xeaE\xc1r\xb9@x>.a\xf2\xca\xe0\x8a\x9a\xdc\xcf\xd8\x11\x0b\xbdT\xe4\xcf\xd5\xd9xWD\xd3\xae6\x1e{L\x96\xc4\xc5\xf9\xc6\x89\x11ar*\x8fU\xdf\xd0a`\xb2@\x90|"8\r\x90\x9aM\xc7z\xad\x08\xf6\xb3<\x9f\xf7\x18\x01\xf0!\xecfX\x19\x88\xf8-\xa2(\xe3\x93\x00\xf7\xd7\xa0;\xcf\xa0C\x17.K\xdb\x1d6\xb6I\xd9\x1c^\x9b\xaf\xb9\x1f,')
    ptype = 31
    self = <paramiko.kex_ecdh_nist.KexNistp256 object at 0x7f84e0595340>
  File "lib/python3.8/site-packages/paramiko/kex_ecdh_nist.py", line 117, in _parse_kexecdh_reply
    K = self.P.exchange(ec.ECDH(), self.Q_S)
    K_S = b"\x00\x00\x00\x07ssh-rsa\x00\x00\x00\x03\x01\x00\x01\x00\x00\x01\x01\x00\xc8\x97\xa8u\x80\t\x0b\xbdF>\x9b\xb5\xd4:2\xeb\xd1\xfa\xec\xe1\xdb\x9e\xe3i\xd8m\xf9\x0b\xe4\x9f\x94\xc2H\xa9\xf4Ns\x8eg\x078\xac\x81\xd7\xa0\xf1\xf2\xf8v\x8f\xc2pw\xb3rK\x82\x19\x7fT\xd9\xdc\x9b\x91N@`\xd1\x90\x1b\xb8S+FQ\x935\xab\x8f\xe3 \xe0\x0f\xa29\x8c\x0cg\x0c2a\x94v \x14Y\x14\xe1i\x85\x9b\x82J\xa64\x05\x9e\xd5\xd3\xcf\x9a\xa0\x9e\x8f\xdfHmz\x9b\xa0P?(j\xcf\xa1neR\xd3\x06/'7Dv<\xc9\x8bYJ\xa0\x9f9/\xc3\xef\xaf@\xee\x8c\x1c4\xcb$\xa3\xb3\xbf\xfd\x04+4\xfb\x9a\xeb\x94\x9doCwT\xa0!\xd2r\xadL\xbaZ\x80\xe9cs\x08\x94&#\xb7\xf2\xed\xe0\x16\x81\x8e\xe0\xe8\xf0i\xd0\xb6\x87\xe3M\xdfye\xa8c\xc3)\xb0D\x14\xa5eF\x8d6&\xdc2]\xe0\xd18\x9fIp\xf0\x15\x9d\xa39y\xf4\xf25\xc3O'\xd4\xa2'\xec\x82Ug\xc2\xc5\x89[\xca-\xd3\x86\x9f"
    Q_S_bytes = b'\x04\r_\r\xa1LL|2\x87\xed\xf1\xd4r\x1a,@\xd6\x87\xd5C!\xf7v\x0c\x8fI\xfb\nNS2\xf5\x8a\xad\xe4\x15tH`\xf0\xc5\x84\x9e\xaasB\x8f\xe7\xfc\xcb\xb7\xde\x7f\xcf\x8f\x81\xf6\xe2\xf4\xa7n \xa5\xc0'
    m = paramiko.Message(b'\x00\x00\x01\x17\x00\x00\x00\x07ssh-rsa\x00\x00\x00\x03\x01\x00\x01\x00\x00\x01\x01\x00\xc8\x97\xa8u\x80\t\x0b\xbdF>\x9b\xb5\xd4:2\xeb\xd1\xfa\xec\xe1\xdb\x9e\xe3i\xd8m\xf9\x0b\xe4\x9f\x94\xc2H\xa9\xf4Ns\x8eg\x078\xac\x81\xd7\xa0\xf1\xf2\xf8v\x8f\xc2pw\xb3rK\x82\x19\x7fT\xd9\xdc\x9b\x91N@`\xd1\x90\x1b\xb8S+FQ\x935\xab\x8f\xe3 \xe0\x0f\xa29\x8c\x0cg\x0c2a\x94v \x14Y\x14\xe1i\x85\x9b\x82J\xa64\x05\x9e\xd5\xd3\xcf\x9a\xa0\x9e\x8f\xdfHmz\x9b\xa0P?(j\xcf\xa1neR\xd3\x06/\'7Dv<\xc9\x8bYJ\xa0\x9f9/\xc3\xef\xaf@\xee\x8c\x1c4\xcb$\xa3\xb3\xbf\xfd\x04+4\xfb\x9a\xeb\x94\x9doCwT\xa0!\xd2r\xadL\xbaZ\x80\xe9cs\x08\x94&#\xb7\xf2\xed\xe0\x16\x81\x8e\xe0\xe8\xf0i\xd0\xb6\x87\xe3M\xdfye\xa8c\xc3)\xb0D\x14\xa5eF\x8d6&\xdc2]\xe0\xd18\x9fIp\xf0\x15\x9d\xa39y\xf4\xf25\xc3O\'\xd4\xa2\'\xec\x82Ug\xc2\xc5\x89[\xca-\xd3\x86\x9f\x00\x00\x00A\x04\r_\r\xa1LL|2\x87\xed\xf1\xd4r\x1a,@\xd6\x87\xd5C!\xf7v\x0c\x8fI\xfb\nNS2\xf5\x8a\xad\xe4\x15tH`\xf0\xc5\x84\x9e\xaasB\x8f\xe7\xfc\xcb\xb7\xde\x7f\xcf\x8f\x81\xf6\xe2\xf4\xa7n \xa5\xc0\x00\x00\x01\x14\x00\x00\x00\x0crsa-sha2-256\x00\x00\x01\x00\xc1\x12\x0bvy\xbb\xdc\xac\xb1|\xc3\xde\xc3\xcf\xc5\xb7\xb0\x1d\xc1\xb5.=\x04\xa1\x0cb\x8e\x0e\xce\x12"\xa2\xc8wsh{F\x04\xf4!\xc4YJ\xbfR:::h;\x12\xa8\xa9\xdb\xf7\xfd\xe3o\xde\x19\xad\xeb\x84l)H2\xdf7\xcf\xd3\xa3\xab)\x98gN@\xe76b~_}\xd6\xe3\x04)\xdbT\x8a"\x85Ve\xe8\xd8\x92[\xe0\xe5;#\xd0*8\xbd\x82\x14\xcb\xe4\xd7\x1e12\xae\xec\xed\x92\x9d<\x02\x942\xb3\xf4\x1b|\r\xfbV\xb7\x10U.cY4\xc2\x13\x93\xe0s\xc4\xfb\xdc\xeaE\xc1r\xb9@x>.a\xf2\xca\xe0\x8a\x9a\xdc\xcf\xd8\x11\x0b\xbdT\xe4\xcf\xd5\xd9xWD\xd3\xae6\x1e{L\x96\xc4\xc5\xf9\xc6\x89\x11ar*\x8fU\xdf\xd0a`\xb2@\x90|"8\r\x90\x9aM\xc7z\xad\x08\xf6\xb3<\x9f\xf7\x18\x01\xf0!\xecfX\x19\x88\xf8-\xa2(\xe3\x93\x00\xf7\xd7\xa0;\xcf\xa0C\x17.K\xdb\x1d6\xb6I\xd9\x1c^\x9b\xaf\xb9\x1f,')
    self = <paramiko.kex_ecdh_nist.KexNistp256 object at 0x7f84e0595340>
    sig = b'\x00\x00\x00\x0crsa-sha2-256\x00\x00\x01\x00\xc1\x12\x0bvy\xbb\xdc\xac\xb1|\xc3\xde\xc3\xcf\xc5\xb7\xb0\x1d\xc1\xb5.=\x04\xa1\x0cb\x8e\x0e\xce\x12"\xa2\xc8wsh{F\x04\xf4!\xc4YJ\xbfR:::h;\x12\xa8\xa9\xdb\xf7\xfd\xe3o\xde\x19\xad\xeb\x84l)H2\xdf7\xcf\xd3\xa3\xab)\x98gN@\xe76b~_}\xd6\xe3\x04)\xdbT\x8a"\x85Ve\xe8\xd8\x92[\xe0\xe5;#\xd0*8\xbd\x82\x14\xcb\xe4\xd7\x1e12\xae\xec\xed\x92\x9d<\x02\x942\xb3\xf4\x1b|\r\xfbV\xb7\x10U.cY4\xc2\x13\x93\xe0s\xc4\xfb\xdc\xeaE\xc1r\xb9@x>.a\xf2\xca\xe0\x8a\x9a\xdc\xcf\xd8\x11\x0b\xbdT\xe4\xcf\xd5\xd9xWD\xd3\xae6\x1e{L\x96\xc4\xc5\xf9\xc6\x89\x11ar*\x8fU\xdf\xd0a`\xb2@\x90|"8\r\x90\x9aM\xc7z\xad\x08\xf6\xb3<\x9f\xf7\x18\x01\xf0!\xecfX\x19\x88\xf8-\xa2(\xe3\x93\x00\xf7\xd7\xa0;\xcf\xa0C\x17.K\xdb\x1d6\xb6I\xd9\x1c^\x9b\xaf\xb9\x1f,'
  File "lib/python3.8/site-packages/cryptography/hazmat/backends/openssl/ec.py", line 162, in exchange
    return _evp_pkey_derive(self._backend, self._evp_pkey, peer_public_key)
    algorithm = <cryptography.hazmat.primitives.asymmetric.ec.ECDH object at 0x7f84e0595610>
    peer_public_key = <cryptography.hazmat.backends.openssl.ec._EllipticCurvePublicKey object at 0x7f84e05955e0>
    self = <cryptography.hazmat.backends.openssl.ec._EllipticCurvePrivateKey object at 0x7f84e0595460>
  File "lib/python3.8/site-packages/cryptography/hazmat/backends/openssl/utils.py", line 18, in _evp_pkey_derive
    res = backend._lib.EVP_PKEY_derive_init(ctx)
    backend = <OpenSSLBackend(version: OpenSSL 3.0.3 3 May 2022, FIPS: False)>
    ctx = <cdata 'EVP_PKEY_CTX *' 0x7f8504007490>
    evp_pkey = <cdata 'EVP_PKEY *' 0x7f850400b1e0>
    peer_public_key = <cryptography.hazmat.backends.openssl.ec._EllipticCurvePublicKey object at 0x7f84e05955e0>

Stack for thread 140208334960384
  File "/usr/local/lib/python3.8/threading.py", line 890, in _bootstrap
    self._bootstrap_inner()
    self = <Thread(QueueFeederThread, started daemon 140208334960384)>
  File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
    self = <Thread(QueueFeederThread, started daemon 140208334960384)>
  File "/usr/local/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
    self = <Thread(QueueFeederThread, started daemon 140208334960384)>
  File "/usr/local/lib/python3.8/multiprocessing/queues.py", line 227, in _feed
    nwait()
    bpopleft = <built-in method popleft of collections.deque object at 0x7f84f88cc8e0>
    buffer = deque([])
    close = <bound method _ConnectionBase.close of <multiprocessing.connection.Connection object at 0x7f84f89a4c10>>
    ignore_epipe = False
    nacquire = <built-in method acquire of _thread.lock object at 0x7f84c9e7c5a0>
    notempty = <Condition(<unlocked _thread.lock object at 0x7f84c9e7c5a0>, 1)>
    nrelease = <built-in method release of _thread.lock object at 0x7f84c9e7c5a0>
    nwait = <bound method Condition.wait of <Condition(<unlocked _thread.lock object at 0x7f84c9e7c5a0>, 1)>>
    obj = <memory at 0x7f8510d12e80>
    onerror = <function Queue._on_queue_feeder_error at 0x7f84f8ab68b0>
    queue_sem = <BoundedSemaphore(value=2147483647, maxvalue=2147483647)>
    send_bytes = <bound method _ConnectionBase.send_bytes of <multiprocessing.connection.Connection object at 0x7f84f89a4c10>>
    sentinel = <object object at 0x7f85148e7670>
    wacquire = <built-in method acquire of _multiprocessing.SemLock object at 0x7f84f894af30>
    wrelease = <built-in method release of _multiprocessing.SemLock object at 0x7f84f894af30>
    writelock = <Lock(owner=None)>
  File "/usr/local/lib/python3.8/threading.py", line 302, in wait
    waiter.acquire()
    gotit = False
    saved_state = None
    self = <Condition(<unlocked _thread.lock object at 0x7f84c9e7c5a0>, 1)>
    timeout = None
    waiter = <locked _thread.lock object at 0x7f84e04a25a0>
saiaprameya commented 1 year ago

Hi Jun Omae, one more observation that i see is that: we have Parent process and Child process. So while forking out the child process we tried closing the connection which would call the client.close. But when the child process is forked it uses the same handle that was created before and i did see it was using the same pssh log. This happens intermittently.

We did use atfork too but that also hangs at the paramiko as shown in the tracebacks above. Can you please help us with any debugging techniques on the paramiko side

Some times we see paramiko logs like this and it hangs:

paramiko.transport(1893): INFO     starting thread (client mode): 0xc1426070^M
paramiko.transport(1893): INFO     Local version/idstring: SSH-2.0-paramiko_3.3.1^M
paramiko.transport(1893): INFO     Remote version/idstring: SSH-2.0-OpenSSH_9.1 PKIX[13.5 FIPS]^M
paramiko.transport(1893): INFO     Connected (version 2.0, client OpenSSH_9.1)^M
paramiko.transport(1893): INFO     === Key exchange possibilities ===^M
paramiko.transport(1893): INFO     kex algos: ecdh-sha2-nistp256, ecdh-sha2-nistp384, ecdh-sha2-nistp521, diffie-hellman-group16-sha512, diffie-hellman-group14-sha1^M
paramiko.transport(1893): INFO     server key: ssh-rsa, rsa-sha2-256, rsa-sha2-512^M
paramiko.transport(1893): INFO     client encrypt: aes128-ctr, aes192-ctr, aes256-ctr, aes128-gcm@openssh.com, aes256-gcm@openssh.com^M
paramiko.transport(1893): INFO     server encrypt: aes128-ctr, aes192-ctr, aes256-ctr, aes128-gcm@openssh.com, aes256-gcm@openssh.com^M
paramiko.transport(1893): INFO     client mac: hmac-sha1, hmac-sha2-256, hmac-sha2-512^M
paramiko.transport(1893): INFO     server mac: hmac-sha1, hmac-sha2-256, hmac-sha2-512^M
paramiko.transport(1893): INFO     client compress: none, zlib@openssh.com^M
paramiko.transport(1893): INFO     server compress: none, zlib@openssh.com^M
paramiko.transport(1893): INFO     client lang: <none>^M
paramiko.transport(1893): INFO     server lang: <none>^M
paramiko.transport(1893): INFO     kex follows: False^M
paramiko.transport(1893): INFO     === Key exchange agreements ===^M
paramiko.transport(1893): INFO     Kex: ecdh-sha2-nistp256^M
paramiko.transport(1893): INFO     HostKey: rsa-sha2-256^M
paramiko.transport(1893): INFO     Cipher: aes128-ctr^M
paramiko.transport(1893): INFO     MAC: hmac-sha2-256^M
paramiko.transport(1893): INFO     Compression: none^M
paramiko.transport(1893): INFO     === End of kex handshake ===^M

Regards SaiAprameya

jun66j5 commented 1 year ago

we have Parent process and Child process. So while forking out the child process we tried closing the connection which would call the client.close. But when the child process is forked it uses the same handle that was created before and i did see it was using the same pssh log. This happens intermittently.

Do not invoke SSHClient.close() after the fork. MSG_DISCONNECT is sent by the close and the ssh session is terminated. It should connect to the ssh server after fork.

If the connecting after fork is difficult, try to use SSHClient.get_transport().stop_thread() instead of SSHClient.close() (not verified).

The same issue is found in stackoverflow: Paramiko Sessions Closes Transport in the Child Process

jun66j5 commented 1 year ago

Sorry. I missed you're using Transport.atfork() to close the socket on the child process.

Could you please provide minimal reproducible code, and paramiko logs with hexdump being enabled.

saiaprameya commented 1 year ago

Hi Jun Omae, It takes more than an hr to repro and since its the automation script working within the org will be tough to provide the same. But the gist is we just do a simple wrap on the paramiko and call the connect or close based on the api provided by the paramiko.

I tried writing simple steps to repro locally but its not reproducible. It only comes when we run those complete automation scripts in scale and since its intermittent issue sometimes it goes fine at times hangs as mentioned above.

For hex dump since it hangs at an intermediate state "=== End of kex handshake ===" i dont see any hex dump there.

So wondering if any other debugs will help in root causing this problem. Please let me know your thoughts on this.

1) More over when we use atfork before forking out and during the spawned process (child process) do you think the socket becomes shared? 2) Tried closing the connection before forking too but that too fails as i see same client connection handle being used by the child process.

Regards SaiAprameya

saiaprameya commented 1 year ago

Hi Jun Omae,

Here are the things that i see currently:

1) Parent process 16636 creates a connection and after some time it run atfork() on that connection before Forking 2 processes (Process 21326 and Process 21327)

2) Process 21326 starts to connect to the existing connection for which atfork() was run in the parent process and sees a "socket error". Then it tries to create a new connection which goes successfully.

3) In the mean while Process 21327 starts with the existing connection too and sees socket error. Now it tries to create a new connection but now it hangs at "End of key exchange.....". I dont see any hex dumps happening here in this scenario.

Here are few prints

Client object at 0x7f7df942a460> 16636
406827 paramiko.transport(1895): INFO     starting thread (client mode): 0xf942a790^M
406828 paramiko.transport(1895): INFO     Local version/idstring: SSH-2.0-paramiko_3.3.1^M
406829 paramiko.transport(1895): INFO     Remote version/idstring: SSH-2.0-OpenSSH_9.1 PKIX[13.5 FIPS]^M
406830 paramiko.transport(1895): INFO     Connected (version 2.0, client OpenSSH_9.1)^M
406831 paramiko.transport(1895): INFO     ############ CALLING WHILE SELF.ACTIVE ##############^M
406832 paramiko.transport(1895): INFO     ############ DONE WITH self.packetizer.read_message 16636 ##############^M
406833 paramiko.transport(1895): INFO     === Key exchange possibilities ===^M
406834 paramiko.transport(1895): INFO     kex algos: ecdh-sha2-nistp256, ecdh-sha2-nistp384, ecdh-sha2-nistp521, diffie-hellman-group16-sha512, diffie-hellman-group14-sha1^M
406835 paramiko.transport(1895): INFO     server key: ssh-rsa, rsa-sha2-256, rsa-sha2-512^M
406836 paramiko.transport(1895): INFO     client encrypt: aes128-ctr, aes192-ctr, aes256-ctr, aes128-gcm@openssh.com, aes256-gcm@openssh.com^M
406837 paramiko.transport(1895): INFO     server encrypt: aes128-ctr, aes192-ctr, aes256-ctr, aes128-gcm@openssh.com, aes256-gcm@openssh.com^M
406838 paramiko.transport(1895): INFO     client mac: hmac-sha1, hmac-sha2-256, hmac-sha2-512^M
406839 paramiko.transport(1895): INFO     server mac: hmac-sha1, hmac-sha2-256, hmac-sha2-512^M
406840 paramiko.transport(1895): INFO     client compress: none, zlib@openssh.com^M
406841 paramiko.transport(1895): INFO     server compress: none, zlib@openssh.com^M
406842 paramiko.transport(1895): INFO     client lang: <none>^M
406843 paramiko.transport(1895): INFO     server lang: <none>^M
406844 paramiko.transport(1895): INFO     kex follows: False^M
406845 paramiko.transport(1895): INFO     === Key exchange agreements ===^M
406846 paramiko.transport(1895): INFO     Kex: ecdh-sha2-nistp256^M
406847 paramiko.transport(1895): INFO     HostKey: rsa-sha2-256^M
406848 paramiko.transport(1895): INFO     Cipher: aes128-ctr^M
406849 paramiko.transport(1895): INFO     MAC: hmac-sha2-256^M
406850 paramiko.transport(1895): INFO     Compression: none^M
406851 paramiko.transport(1895): INFO     === End of kex handshake ===^M
406852 paramiko.transport(1895): INFO     ############ KEX_INIT FUNCTION DONE 16636  ##############^M
406853 paramiko.transport(1895): INFO     ############ KEX_ENGINE START FUNCTION DONE 16636  ##############^M
406854 paramiko.transport(1895): INFO     ############ FINISHED WHILE 16636 ##############^M
406855 paramiko.transport(1895): INFO     ############ DONE WITH self.packetizer.read_message 16636 ##############^M
406856 paramiko.transport(1895): INFO     kex engine KexNistp256 specified hash_algo <built-in function openssl_sha256>^M
406857 paramiko.transport(1895): INFO     ############ DONE WITH self.packetizer.read_message 16636 ##############^M
406858 paramiko.transport(1895): INFO     Switch to new keys ...^M
406859 paramiko.transport(1895): INFO     ############ FINISHED WHILE 16636 ##############^M
406860 paramiko.transport(1895): INFO     Adding ssh-rsa host key for 10.193.169.147: b'd8139858d7c1a19056429d766a39e069'^M
406861 paramiko.transport(1895): INFO     ############ DONE WITH self.packetizer.read_message 16636 ##############^M
406862 paramiko.transport(1895): INFO     Got EXT_INFO: {'publickey-algorithms@roumenpetrov.info': b'x509v3-ecdsa-sha2-nistp256,x509v3-ecdsa-sha2-nistp384,x509v3-ecdsa-sha2-nistp521,x509v3-ssh-rsa,x509v3-rsa2048       -sha256,ssh-ed25519,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-rsa,rsa-sha2-256,rsa-sha2-512', 'server-sig-algs': b'x509v3-ecdsa-sha2-nistp256,x509v3-ecdsa-sha2-nistp384,x509v       3-ecdsa-sha2-nistp521,x509v3-ssh-rsa,x509v3-rsa2048-sha256,ssh-ed25519,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-rsa,rsa-sha2-256,rsa-sha2-512'}^M
406863 paramiko.transport(1895): INFO     ############ FINISHED WHILE 16636 ##############^M
406864 paramiko.transport(1895): INFO     Trying discovered key b'7f9999abd685ec4a3f960e971c1e0690' in /home/rajraghu/.ssh/id_rsa^M
406865 paramiko.transport(1895): INFO     ############ DONE WITH self.packetizer.read_message 16636 ##############^M
406866 paramiko.transport(1895): INFO     userauth is OK^M
406867 paramiko.transport(1895): INFO     Finalizing pubkey algorithm for key of type 'ssh-rsa'^M
406868 paramiko.transport(1895): INFO     Our pubkey algorithm list: ['rsa-sha2-512', 'rsa-sha2-256', 'ssh-rsa']^M
406869 paramiko.transport(1895): INFO     Server-side algorithm list: ['x509v3-ecdsa-sha2-nistp256', 'x509v3-ecdsa-sha2-nistp384', 'x509v3-ecdsa-sha2-nistp521', 'x509v3-ssh-rsa', 'x509v3-rsa2048-sha256', 'ssh-ed       25519', 'ssh-dss', 'ecdsa-sha2-nistp256', 'ecdsa-sha2-nistp384', 'ecdsa-sha2-nistp521', 'ssh-rsa', 'rsa-sha2-256', 'rsa-sha2-512']^M
406870 paramiko.transport(1895): INFO     Agreed upon 'rsa-sha2-512' pubkey algorithm^M
406871 paramiko.transport(1895): INFO     ############ FINISHED WHILE 16636 ##############^M
406872 paramiko.transport(1895): INFO     ############ DONE WITH self.packetizer.read_message 16636 ##############^M
406873 paramiko.transport(1895): INFO     Auth banner: b'Cisco UCS 6400 Series Fabric Interconnect\n'^M
406874 paramiko.transport(1895): INFO     ############ FINISHED WHILE 16636 ##############^M
406875 paramiko.transport(1895): INFO     ############ DONE WITH self.packetizer.read_message 16636 ##############^M
406876 paramiko.transport(1895): INFO     Authentication (publickey) failed.^M
406877 paramiko.transport(1895): INFO     ############ FINISHED WHILE 16636 ##############^M
406878 paramiko.transport(1895): INFO     ############ DONE WITH self.packetizer.read_message 16636 ##############^M
406879 paramiko.transport(1895): INFO     userauth is OK^M
406880 paramiko.transport(1895): INFO     ############ FINISHED WHILE 16636 ##############^M
406881 paramiko.transport(1895): INFO     ############ DONE WITH self.packetizer.read_message 16636 ##############^M
406882 paramiko.transport(1895): INFO     Authentication (password) successful!^M
406883 paramiko.transport(1895): INFO     ############ FINISHED WHILE 16636 ##############^M
406884 {'H': b'\xa6x\t\xabD\xc7L\xc1^\xc8N\xe3\x7f\xc9&f0w\x10\xef\x03\xe93\x8c'^M
406885       b'g\xcee\x9aB\xe2\x00\xd3',^M
406886  'K': None,^M
406887  '_agreed_pubkey_algorithm': 'rsa-sha2-512',^M
406888  '_args': (),^M
406889  '_channel_counter': 0,^M
406890  '_channels': <paramiko.transport.ChannelMap object at 0x7f7df942abe0>,^M
406891  '_daemonic': True,^M
406892  '_expected_packet': (),^M
406893  '_forward_agent_handler': None,^M
406894  '_handler_table': {7: <bound method Transport._parse_ext_info of <paramiko.Transport at 0xf942a790 (cipher aes128-ctr, 128 bits) (active; 0 open channel(s))>>,^M
406895                     20: <bound method Transport._negotiate_keys of <paramiko.Transport at 0xf942a790 (cipher aes128-ctr, 128 bits) (active; 0 open channel(s))>>,^M
406896                     21: <bound method Transport._parse_newkeys of <paramiko.Transport at 0xf942a790 (cipher aes128-ctr, 128 bits) (active; 0 open channel(s))>>,^M
406897                     80: <bound method Transport._parse_global_request of <paramiko.Transport at 0xf942a790 (cipher aes128-ctr, 128 bits) (active; 0 open channel(s))>>,^M
406898                     81: <bound method Transport._parse_request_success of <paramiko.Transport at 0xf942a790 (cipher aes128-ctr, 128 bits) (active; 0 open channel(s))>>,^M
406899                     82: <bound method Transport._parse_request_failure of <paramiko.Transport at 0xf942a790 (cipher aes128-ctr, 128 bits) (active; 0 open channel(s))>>,^M
                                                                                                                                                                                                 406899,1      72%

406900                     90: <bound method Transport._parse_channel_open of <paramiko.Transport at 0xf942a790 (cipher aes128-ctr, 128 bits) (active; 0 open channel(s))>>,^M
406901                     91: <bound method Transport._parse_channel_open_success of <paramiko.Transport at 0xf942a790 (cipher aes128-ctr, 128 bits) (active; 0 open channel(s))>>,^M
406902                     92: <bound method Transport._parse_channel_open_failure of <paramiko.Transport at 0xf942a790 (cipher aes128-ctr, 128 bits) (active; 0 open channel(s))>>},^M
406903  '_ident': 140178924492544,^M
406904  '_initialized': True,^M
406905  '_invoke_excepthook': <function _make_invoke_excepthook.<locals>.invoke_excepthook at 0x7f7df9ed33a0>,^M
406906  '_is_stopped': False,^M
406907  '_kwargs': {},^M
406908  '_latest_kex_init': b"\x14\xa3\xb4\xb1:\xea:\x0f\x9f\xda\x0b'\xc2\xf8'^"^M
406909                      b'\x87\x00\x00\x01\x1bcurve25519-sha256@libssh.org,ecdh-s'^M
406910                      b'ha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffi'^M
406911                      b'e-hellman-group16-sha512,diffie-hellman-group-exchange-s'^M
406912                      b'ha256,diffie-hellman-group14-sha256,diffie-hellman-group'^M
406913                      b'-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellma'^M
406914                      b'n-group1-sha1,ext-info-c\x00\x00\x01\xe9rsa-sha2-256,rsa'^M
406915                      b'-sha2-512,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nis'^M
406916                      b'tp384,ecdsa-sha2-nistp521,rsa-sha2-512,rsa-sha2-256,ssh-'^M
406917                      b'rsa,ssh-dss,rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-5'^M
406918                      b'12-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com'^M
406919                      b',ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nis'^M
406920                      b'tp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@'^M
406921                      b'openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-2'^M
406922                      b'56-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh'^M
406923                      b'-dss-cert-v01@openssh.com\x00\x00\x00Jaes128-ctr,aes1'^M
406924                      b'92-ctr,aes256-ctr,aes128-cbc,aes192-cbc,aes256-cbc,3des-'^M
406925                      b'cbc\x00\x00\x00Jaes128-ctr,aes192-ctr,aes256-ctr,aes1'^M
406926                      b'28-cbc,aes192-cbc,aes256-cbc,3des-cbc\x00\x00\x00\x83hma'^M
406927                      b'c-sha2-256,hmac-sha2-512,hmac-sha2-256-etm@openssh.com,h'^M
406928                      b'mac-sha2-512-etm@openssh.com,hmac-sha1,hmac-md5,hmac-sha'^M
406929                      b'1-96,hmac-md5-96\x00\x00\x00\x83hmac-sha2-256,hmac-sha2-'^M
406930                      b'512,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@open'^M
406931                      b'ssh.com,hmac-sha1,hmac-md5,hmac-sha1-96,hmac-md5-96\x00'^M
406932                      b'\x00\x00\x04none\x00\x00\x00\x04none\x00\x00\x00\x00\x00'^M
406933                      b'\x00\x00\x00\x00\x00\x00\x00\x00',^M
406934  '_logged_hash_selection': True,^M
406935  '_name': 'Thread-74',^M
406936  '_native_id': 20778,^M
406937  '_preferred_compression': ('none',),^M
406938  '_remote_ext_info': None,^M
406939  '_started': <threading.Event object at 0x7f7df942aa60>,^M
406940  '_stderr': <_io.TextIOWrapper name='<stderr>' mode='w' encoding='utf-8'>,^M
406941  '_target': None,^M
406942  '_tcp_handler': None,^M
406943  '_tstate_lock': <locked _thread.lock object at 0x7f7df9444f60>,^M
406944  '_x11_handler': None,^M
406945  'active': True,^M
406946  'auth_handler': <paramiko.auth_handler.AuthHandler object at 0x7f7df9ef7c40>,^M
406947  'auth_timeout': 30,^M
406948  'authenticated': True,^M
406949  'banner_timeout': 15,^M
406950  'channel_events': {},^M
406951  'channel_timeout': 3600,^M
406952  'channels_seen': {},^M
406953  'clear_to_send': <threading.Event object at 0x7f7df942acd0>,^M
406954  'clear_to_send_lock': <unlocked _thread.lock object at 0x7f7df942adb0>,^M
406955  'clear_to_send_timeout': 30.0,^M
406956  'completion_event': <threading.Event object at 0x7f7df9444d90>,^M
406957  'default_max_packet_size': 32768,^M
406958  'default_window_size': 2097152,^M
406959  'disabled_algorithms': {},^M
406960  'global_response': None,^M
406961  'gss_host': None,^M
406962  'gss_kex_used': False,^M
406963  'handshake_timeout': 15,^M
406964  'host_key': PKey(alg=RSA, bits=2048, fp=SHA256:paKa8cWmnUU7P7Cv10NljlSdcfkYwakbiNY18WVnozw),^M
406965  'host_key_type': 'rsa-sha2-256',^M
406966  'hostname': None,^M
406967  'in_kex': False,^M
406968  'initial_kex_done': True,^M
406969  'kex_engine': None,^M
406970  'kexgss_ctxt': None,^M
406971  'local_cipher': 'aes128-ctr',^M
406972  'local_compression': 'none',^M
406973  'local_kex_init': None,^M
406974  'local_mac': 'hmac-sha2-256',^M
406975  'local_version': 'SSH-2.0-paramiko_3.3.1',^M
406976  'lock': <unlocked _thread.lock object at 0x7f7df942aba0>,^M
406977  'log_name': 'paramiko.transport',^M
406978  'logger': <RuntestsLogger paramiko.transport (INFO)>,^M
406979  'packetizer': <paramiko.packet.Packetizer object at 0x7f7df942ab50>,^M
406980  'remote_cipher': 'aes128-ctr',^M
406981  'remote_compression': 'none',^M
406982  'remote_kex_init': None,^M
406983  'remote_mac': 'hmac-sha2-256',^M
406984  'remote_version': 'SSH-2.0-OpenSSH_9.1 PKIX[13.5 FIPS]',^M
406985  'saved_exception': None,^M
406986  'server_accept_cv': <Condition(<unlocked _thread.lock object at 0x7f7df942aba0>, 0)>,^M
406987  'server_accepts': [],^M
406988  'server_extensions': {'publickey-algorithms@roumenpetrov.info': b'x509v3-ecdsa'^M
406989                                                                  b'-sha2-nistp2'^M
406990                                                                  b'56,x509v3-ec'^M
406991                                                                  b'dsa-sha2-nis'^M
406992                                                                  b'tp384,x509v3'^M
406993                                                                  b'-ecdsa-sha2-'^M
406994                                                                  b'nistp521,x50'^M
406995                                                                  b'9v3-ssh-rsa,'^M
406996                                                                  b'x509v3-rsa20'^M
406997                                                                  b'48-sha256,ss'^M
406998                                                                  b'h-ed25519,ss'^M
406999                                                                  b'h-dss,ecdsa-'^M
407000                                                                  b'sha2-nistp25'^M
407001                                                                  b'6,ecdsa-sha2'^M
407002                                                                  b'-nistp384,ec'^M
407003                                                                  b'dsa-sha2-nis'^M
407004                                                                  b'tp521,ssh-rs'^M
407005                                                                  b'a,rsa-sha2-2'^M
407006                                                                  b'56,rsa-sha2-'^M
407007                                                                  b'512',^M
407008                        'server-sig-algs': b'x509v3-ecdsa-sha2-nistp256,x509v'^M
407009                                           b'3-ecdsa-sha2-nistp384,x509v3-ecd'^M
407010                                           b'sa-sha2-nistp521,x509v3-ssh-rsa,'^M
407011                                           b'x509v3-rsa2048-sha256,ssh-ed2551'^M
407012                                           b'9,ssh-dss,ecdsa-sha2-nistp256,ec'^M
407013                                           b'dsa-sha2-nistp384,ecdsa-sha2-nis'^M
407014                                           b'tp521,ssh-rsa,rsa-sha2-256,rsa-s'^M
407015                                           b'ha2-512'},^M
407016  'server_key_dict': {},^M
407017  'server_mode': False,^M
407018  'server_object': None,^M
407019  'server_sig_algs': True,^M
407020  'session_id': b'\xa6x\t\xabD\xc7L\xc1^\xc8N\xe3\x7f\xc9&f0w\x10\xef'^M
407021                b'\x03\xe93\x8cg\xcee\x9aB\xe2\x00\xd3',^M
407022  'sock': <socket.socket fd=180, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.193.170.82', 33990), raddr=('10.193.169.147', 22)>,^M
407023  'subsystem_table': {},^M
407024  'sys': <module 'sys' (built-in)>,^M
407025  'use_gss_kex': False}^M
407071 paramiko.transport(1210): INFO     [chan 0] Max packet in: 32768 bytes^M
407072 paramiko.transport(1895): INFO     ############ DONE WITH self.packetizer.read_message 16636 ##############^M
407073 paramiko.transport(1895): INFO     Received global request "hostkeys-00@openssh.com"^M
407074 paramiko.transport(1895): INFO     Rejecting "hostkeys-00@openssh.com" global request from server.^M
407075 paramiko.transport(1895): INFO     ############ FINISHED WHILE 16636 ##############^M
407076 paramiko.transport(1895): INFO     ############ DONE WITH self.packetizer.read_message 16636 ##############^M
407077 paramiko.transport(1210): INFO     [chan 0] Max packet out: 32768 bytes^M
407078 paramiko.transport(1895): INFO     Secsh channel 0 opened.^M
407079 paramiko.transport(1895): INFO     ############ FINISHED WHILE 16636 ##############^M
407080 paramiko.transport(1895): INFO     ############ DONE WITH self.packetizer.read_message 16636 ##############^M
407081 paramiko.transport(1210): INFO     [chan 0] Sesch channel 0 request ok^M
407082 paramiko.transport(1895): INFO     ############ FINISHED WHILE 16636 ##############^M
407083 paramiko.transport(1895): INFO     ############ DONE WITH self.packetizer.read_message 16636 ##############^M
407084 paramiko.transport(1895): INFO     ############ FINISHED WHILE 16636 ##############^M
407085 paramiko.transport(1895): INFO     ############ DONE WITH self.packetizer.read_message 16636 ##############^M
407086 paramiko.transport(1210): INFO     [chan 0] Sesch channel 0 request ok^M
407087 paramiko.transport(1895): INFO     ############ FINISHED WHILE 16636 ##############^M

__init__.py:93* - ATFORK called on conn handle   >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    <qali.interface.io.Client object at 0x7f7df942a460>   16636

416773 Client object at 0x7f7df942a460> 21326

416777 {'H': b'\xa6x\t\xabD\xc7L\xc1^\xc8N\xe3\x7f\xc9&f0w\x10\xef\x03\xe93\x8c'^M
416778       b'g\xcee\x9aB\xe2\x00\xd3',^M
416779  'K': None,^M
416780  '_agreed_pubkey_algorithm': 'rsa-sha2-512',^M
416781  '_args': (),^M
416782  '_channel_counter': 1,^M
416783  '_channels': <paramiko.transport.ChannelMap object at 0x7f7df942abe0>,^M
416784  '_daemonic': True,^M
416785  '_expected_packet': (),^M
416786  '_forward_agent_handler': None,^M
416787  '_handler_table': {7: <bound method Transport._parse_ext_info of <paramiko.Transport at 0xf942a790 (unconnected)>>,^M
416788                     20: <bound method Transport._negotiate_keys of <paramiko.Transport at 0xf942a790 (unconnected)>>,^M
416789                     21: <bound method Transport._parse_newkeys of <paramiko.Transport at 0xf942a790 (unconnected)>>,^M
416790                     80: <bound method Transport._parse_global_request of <paramiko.Transport at 0xf942a790 (unconnected)>>,^M
416791                     81: <bound method Transport._parse_request_success of <paramiko.Transport at 0xf942a790 (unconnected)>>,^M
416792                     82: <bound method Transport._parse_request_failure of <paramiko.Transport at 0xf942a790 (unconnected)>>,^M
416793                     90: <bound method Transport._parse_channel_open of <paramiko.Transport at 0xf942a790 (unconnected)>>,^M
416794                     91: <bound method Transport._parse_channel_open_success of <paramiko.Transport at 0xf942a790 (unconnected)>>,^M
416795                     92: <bound method Transport._parse_channel_open_failure of <paramiko.Transport at 0xf942a790 (unconnected)>>},^M
416796  '_ident': 140178924492544,^M
416797  '_initialized': True,^M
416798  '_invoke_excepthook': <function _make_invoke_excepthook.<locals>.invoke_excepthook at 0x7f7df9ed33a0>,^M
416799  '_is_stopped': True,^M
416800  '_kwargs': {},^M
416801  '_latest_kex_init': b"\x14\xa3\xb4\xb1:\xea:\x0f\x9f\xda\x0b'\xc2\xf8'^"^M
416802                      b'\x87\x00\x00\x01\x1bcurve25519-sha256@libssh.org,ecdh-s'^M
416803                      b'ha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffi'^M
416804                      b'e-hellman-group16-sha512,diffie-hellman-group-exchange-s'^M
416805                      b'ha256,diffie-hellman-group14-sha256,diffie-hellman-group'^M
416806                      b'-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellma'^M
416807                      b'n-group1-sha1,ext-info-c\x00\x00\x01\xe9rsa-sha2-256,rsa'^M
416808                      b'-sha2-512,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nis'^M
416809                      b'tp384,ecdsa-sha2-nistp521,rsa-sha2-512,rsa-sha2-256,ssh-'^M
416810                      b'rsa,ssh-dss,rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-5'^M
416811                      b'12-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com'^M
416812                      b',ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nis'^M
416813                      b'tp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@'^M
416814                      b'openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-2'^M
416815                      b'56-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh'^M
416816                      b'-dss-cert-v01@openssh.com\x00\x00\x00Jaes128-ctr,aes1'^M
416817                      b'92-ctr,aes256-ctr,aes128-cbc,aes192-cbc,aes256-cbc,3des-'^M
416818                      b'cbc\x00\x00\x00Jaes128-ctr,aes192-ctr,aes256-ctr,aes1'^M
416819                      b'28-cbc,aes192-cbc,aes256-cbc,3des-cbc\x00\x00\x00\x83hma'^M
416820                      b'c-sha2-256,hmac-sha2-512,hmac-sha2-256-etm@openssh.com,h'^M
416821                      b'mac-sha2-512-etm@openssh.com,hmac-sha1,hmac-md5,hmac-sha'^M
416822                      b'1-96,hmac-md5-96\x00\x00\x00\x83hmac-sha2-256,hmac-sha2-'^M
416823                      b'512,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@open'^M
416824                      b'ssh.com,hmac-sha1,hmac-md5,hmac-sha1-96,hmac-md5-96\x00'^M
416825                      b'\x00\x00\x04none\x00\x00\x00\x04none\x00\x00\x00\x00\x00'^M
416826                      b'\x00\x00\x00\x00\x00\x00\x00\x00',^M
416827  '_logged_hash_selection': True,^M
416828  '_name': 'Thread-74',^M
416829  '_native_id': 20778,^M
416830  '_preferred_compression': ('none',),^M
416831  '_remote_ext_info': None,^M
416832  '_started': <threading.Event object at 0x7f7df942aa60>,^M
416833  '_stderr': <_io.TextIOWrapper name='<stderr>' mode='w' encoding='utf-8'>,^M
416834  '_target': None,^M
416835  '_tcp_handler': None,^M
416836  '_tstate_lock': None,^M
416837  '_x11_handler': None,^M
416838  'active': False,^M
416839  'auth_handler': <paramiko.auth_handler.AuthHandler object at 0x7f7df9ef7c40>,^M
416840  'auth_timeout': 30,^M
416841  'authenticated': True,^M
416842  'banner_timeout': 15,^M
416843  'channel_events': {},^M
416844  'channel_timeout': 3600,^M
416845  'channels_seen': {0: True},^M
416846  'clear_to_send': <threading.Event object at 0x7f7df942acd0>,^M
416847  'clear_to_send_lock': <unlocked _thread.lock object at 0x7f7df942adb0>,^M
416848  'clear_to_send_timeout': 30.0,^M
416849  'completion_event': <threading.Event object at 0x7f7df9444d90>,^M
416850  'default_max_packet_size': 32768,^M
416851  'default_window_size': 2097152,^M
416852  'disabled_algorithms': {},^M
416853  'global_response': None,^M
416854  'gss_host': None,^M
416855  'gss_kex_used': False,^M
416856  'handshake_timeout': 15,^M
416857  'host_key': PKey(alg=RSA, bits=2048, fp=SHA256:paKa8cWmnUU7P7Cv10NljlSdcfkYwakbiNY18WVnozw),^M
416858  'host_key_type': 'rsa-sha2-256',^M
416859  'hostname': None,^M
416860  'in_kex': False,^M
416861  'initial_kex_done': True,^M
416862  'kex_engine': None,^M
416863  'kexgss_ctxt': None,^M
416864  'local_cipher': 'aes128-ctr',^M
416865  'local_compression': 'none',^M
416866  'local_kex_init': None,^M
416867  'local_mac': 'hmac-sha2-256',^M
416868  'local_version': 'SSH-2.0-paramiko_3.3.1',^M
416869  'lock': <unlocked _thread.lock object at 0x7f7df942aba0>,^M
416870  'log_name': 'paramiko.transport',^M
416871  'logger': <RuntestsLogger paramiko.transport (INFO)>,^M
416872  'packetizer': <paramiko.packet.Packetizer object at 0x7f7df942ab50>,^M
416873  'remote_cipher': 'aes128-ctr',^M
416874  'remote_compression': 'none',^M
416875  'remote_kex_init': None,^M
416876  'remote_mac': 'hmac-sha2-256',^M
416877  'remote_version': 'SSH-2.0-OpenSSH_9.1 PKIX[13.5 FIPS]',^M
416878  'saved_exception': None,^M
416879  'server_accept_cv': <Condition(<unlocked _thread.lock object at 0x7f7df942aba0>, 0)>,^M
416880  'server_accepts': [],^M
416881  'server_extensions': {'publickey-algorithms@roumenpetrov.info': b'x509v3-ecdsa'^M
416882                                                                  b'-sha2-nistp2'^M
416883                                                                  b'56,x509v3-ec'^M
416884                                                                  b'dsa-sha2-nis'^M
416885                                                                  b'tp384,x509v3'^M
416886                                                                  b'-ecdsa-sha2-'^M
416887                                                                  b'nistp521,x50'^M
416888                                                                  b'9v3-ssh-rsa,'^M
416889                                                                  b'x509v3-rsa20'^M
416890                                                                  b'48-sha256,ss'^M
416891                                                                  b'h-ed25519,ss'^M
416892                                                                  b'h-dss,ecdsa-'^M
416893                                                                  b'sha2-nistp25'^M
416894                                                                  b'6,ecdsa-sha2'^M
416895                                                                  b'-nistp384,ec'^M
416896                                                                  b'dsa-sha2-nis'^M
416897                                                                  b'tp521,ssh-rs'^M
416898                                                                  b'a,rsa-sha2-2'^M
416899                                                                  b'56,rsa-sha2-'^M
416900                                                                  b'512',^M
416901                        'server-sig-algs': b'x509v3-ecdsa-sha2-nistp256,x509v'^M
416902                                           b'3-ecdsa-sha2-nistp384,x509v3-ecd'^M
416903                                           b'sa-sha2-nistp521,x509v3-ssh-rsa,'^M
416904                                           b'x509v3-rsa2048-sha256,ssh-ed2551'^M
416905                                           b'9,ssh-dss,ecdsa-sha2-nistp256,ec'^M
416906                                           b'dsa-sha2-nistp384,ecdsa-sha2-nis'^M
416907                                           b'tp521,ssh-rsa,rsa-sha2-256,rsa-s'^M
416908                                           b'ha2-512'},^M
416909  'server_key_dict': {},^M
416910  'server_mode': False,^M
416911  'server_object': None,^M
416912  'server_sig_algs': True,^M
416913  'session_id': b'\xa6x\t\xabD\xc7L\xc1^\xc8N\xe3\x7f\xc9&f0w\x10\xef'^M
416914                b'\x03\xe93\x8cg\xcee\x9aB\xe2\x00\xd3',^M
416915  'sock': <socket.socket [closed] fd=-1, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0>,^M
416916  'subsystem_table': {},^M
416917  'sys': <module 'sys' (built-in)>,^M
416918  'use_gss_kex': False}^M

416973   File "/home/rajraghu/Envs/proj3/lib/python3.8/site-packages/paramiko/file.py", line 389, in write^M
416974     self._write_all(data)^M
416975   File "/home/rajraghu/Envs/proj3/lib/python3.8/site-packages/paramiko/file.py", line 507, in _write_all^M
416976     count = self._write(data)^M
416977   File "/home/rajraghu/Envs/proj3/lib/python3.8/site-packages/paramiko/channel.py", line 1362, in _write^M
416978     self.channel.sendall(data)^M
416979   File "/home/rajraghu/Envs/proj3/lib/python3.8/site-packages/paramiko/channel.py", line 844, in sendall^M
416980     sent = self.send(s)^M
416981   File "/home/rajraghu/Envs/proj3/lib/python3.8/site-packages/paramiko/channel.py", line 799, in send^M
416982     return self._send(s, m)^M
416983   File "/home/rajraghu/Envs/proj3/lib/python3.8/site-packages/paramiko/channel.py", line 1196, in _send^M
416984     raise socket.error("Socket is closed")^M
416985 OSError: Socket is closed^M

416989 Client object at 0x7f7df9e29b50> 21326 --

416990 paramiko.transport(1895): INFO     starting thread (client mode): 0xf9dd7e50^M
416991 paramiko.transport(1895): INFO     Local version/idstring: SSH-2.0-paramiko_3.3.1^M

417632 paramiko.transport(1895): INFO     Remote version/idstring: SSH-2.0-OpenSSH_9.1 PKIX[13.5 FIPS]^M
417633 paramiko.transport(1895): INFO     Connected (version 2.0, client OpenSSH_9.1)^M
417634 paramiko.transport(1895): INFO     ############ CALLING WHILE SELF.ACTIVE ##############^M
417635 paramiko.transport(1895): INFO     ############ DONE WITH self.packetizer.read_message 21326 ##############^M
417636 paramiko.transport(1895): INFO     === Key exchange possibilities ===^M
417637 paramiko.transport(1895): INFO     kex algos: ecdh-sha2-nistp256, ecdh-sha2-nistp384, ecdh-sha2-nistp521, diffie-hellman-group16-sha512, diffie-hellman-group14-sha1^M
417638 paramiko.transport(1895): INFO     server key: ssh-rsa, rsa-sha2-256, rsa-sha2-512^M
417639 paramiko.transport(1895): INFO     client encrypt: aes128-ctr, aes192-ctr, aes256-ctr, aes128-gcm@openssh.com, aes256-gcm@openssh.com^M
417640 paramiko.transport(1895): INFO     server encrypt: aes128-ctr, aes192-ctr, aes256-ctr, aes128-gcm@openssh.com, aes256-gcm@openssh.com^M
417641 paramiko.transport(1895): INFO     client mac: hmac-sha1, hmac-sha2-256, hmac-sha2-512^M
417642 paramiko.transport(1895): INFO     server mac: hmac-sha1, hmac-sha2-256, hmac-sha2-512^M
417643 paramiko.transport(1895): INFO     client compress: none, zlib@openssh.com^M
417644 paramiko.transport(1895): INFO     server compress: none, zlib@openssh.com^M
417645 paramiko.transport(1895): INFO     client lang: <none>^M
417646 paramiko.transport(1895): INFO     server lang: <none>^M
417647 paramiko.transport(1895): INFO     kex follows: False^M
417648 paramiko.transport(1895): INFO     === Key exchange agreements ===^M
417649 paramiko.transport(1895): INFO     Kex: ecdh-sha2-nistp256^M
417650 paramiko.transport(1895): INFO     HostKey: rsa-sha2-256^M
417651 paramiko.transport(1895): INFO     Cipher: aes128-ctr^M
417652 paramiko.transport(1895): INFO     MAC: hmac-sha2-256^M
417653 paramiko.transport(1895): INFO     Compression: none^M
417654 paramiko.transport(1895): INFO     === End of kex handshake ===^M
417655 paramiko.transport(1895): INFO     ############ KEX_INIT FUNCTION DONE 21326  ##############^M
417656 paramiko.transport(1895): INFO     ############ KEX_ENGINE START FUNCTION DONE 21326  ##############^M
417657 paramiko.transport(1895): INFO     ############ FINISHED WHILE 21326 ##############^M
417662 paramiko.transport(1895): INFO     ############ DONE WITH self.packetizer.read_message 21326 ##############^M
417663 paramiko.transport(1895): INFO     kex engine KexNistp256 specified hash_algo <built-in function openssl_sha256>^M
417664 paramiko.transport(1895): INFO     ############ DONE WITH self.packetizer.read_message 21326 ##############^M
417665 paramiko.transport(1895): INFO     Switch to new keys ...^M
417666 paramiko.transport(1895): INFO     ############ FINISHED WHILE 21326 ##############^M
417667 paramiko.transport(1895): INFO     ############ DONE WITH self.packetizer.read_message 21326 ##############^M
417668 paramiko.transport(1895): INFO     Adding ssh-rsa host key for 10.193.169.147: b'd8139858d7c1a19056429d766a39e069'^M
417669 paramiko.transport(1895): INFO     Got EXT_INFO: {'publickey-algorithms@roumenpetrov.info': b'x509v3-ecdsa-sha2-nistp256,x509v3-ecdsa-sha2-nistp384,x509v3-ecdsa-sha2-nistp521,x509v3-ssh-rsa,x509v3-rsa2048       -sha256,ssh-ed25519,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-rsa,rsa-sha2-256,rsa-sha2-512', 'server-sig-algs': b'x509v3-ecdsa-sha2-nistp256,x509v3-ecdsa-sha2-nistp384,x509v       3-ecdsa-sha2-nistp521,x509v3-ssh-rsa,x509v3-rsa2048-sha256,ssh-ed25519,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-rsa,rsa-sha2-256,rsa-sha2-512'}^M
417670 paramiko.transport(1895): INFO     ############ FINISHED WHILE 21326 ##############^M
417671 paramiko.transport(1895): INFO     Trying discovered key b'7f9999abd685ec4a3f960e971c1e0690' in /home/rajraghu/.ssh/id_rsa^M
417672 paramiko.transport(1895): INFO     ############ DONE WITH self.packetizer.read_message 21326 ##############^M
417673 paramiko.transport(1895): INFO     userauth is OK^M
417674 paramiko.transport(1895): INFO     Finalizing pubkey algorithm for key of type 'ssh-rsa'^M
417675 paramiko.transport(1895): INFO     Our pubkey algorithm list: ['rsa-sha2-512', 'rsa-sha2-256', 'ssh-rsa']^M
417676 paramiko.transport(1895): INFO     Server-side algorithm list: ['x509v3-ecdsa-sha2-nistp256', 'x509v3-ecdsa-sha2-nistp384', 'x509v3-ecdsa-sha2-nistp521', 'x509v3-ssh-rsa', 'x509v3-rsa2048-sha256', 'ssh-ed       25519', 'ssh-dss', 'ecdsa-sha2-nistp256', 'ecdsa-sha2-nistp384', 'ecdsa-sha2-nistp521', 'ssh-rsa', 'rsa-sha2-256', 'rsa-sha2-512']^M
417677 paramiko.transport(1895): INFO     Agreed upon 'rsa-sha2-512' pubkey algorithm^M
417678 paramiko.transport(1895): INFO     ############ FINISHED WHILE 21326 ##############^M
417679 paramiko.transport(1895): INFO     ############ DONE WITH self.packetizer.read_message 21326 ##############^M
417680 paramiko.transport(1895): INFO     Auth banner: b'Cisco UCS 6400 Series Fabric Interconnect\n'^M
417681 paramiko.transport(1895): INFO     ############ FINISHED WHILE 21326 ##############^M
417682 paramiko.transport(1895): INFO     ############ DONE WITH self.packetizer.read_message 21326 ##############^M
417683 paramiko.transport(1895): INFO     Authentication (publickey) failed.^M
417684 paramiko.transport(1895): INFO     ############ FINISHED WHILE 21326 ##############^M
417685 paramiko.transport(1895): INFO     ############ DONE WITH self.packetizer.read_message 21326 ##############^M
417686 paramiko.transport(1895): INFO     userauth is OK^M
417687 paramiko.transport(1895): INFO     ############ FINISHED WHILE 21326 ##############^M
417688 paramiko.transport(1895): INFO     ############ DONE WITH self.packetizer.read_message 21326 ##############^M
417689 paramiko.transport(1895): INFO     Authentication (password) successful!^M
417690 paramiko.transport(1895): INFO     ############ FINISHED WHILE 21326 ##############^M
417691 {'H': b'[\xde\\p\n~Mlt\xf7\xd5\xb2\x07b\x80\x08Pe\x12\x94\xdc\t\xd5\x05#FJ\x1c'^M
417692       b'7\xcb0\xed',^M
417693  'K': None,^M
417694  '_agreed_pubkey_algorithm': 'rsa-sha2-512',^M
417695  '_args': (),^M
417696  '_channel_counter': 0,^M
417697  '_channels': <paramiko.transport.ChannelMap object at 0x7f7df9dd7dc0>,^M
417698  '_daemonic': True,^M
417699  '_expected_packet': (),^M
417700  '_forward_agent_handler': None,^M
417701  '_handler_table': {7: <bound method Transport._parse_ext_info of <paramiko.Transport at 0xf9dd7e50 (cipher aes128-ctr, 128 bits) (active; 0 open channel(s))>>,^M
417702                     20: <bound method Transport._negotiate_keys of <paramiko.Transport at 0xf9dd7e50 (cipher aes128-ctr, 128 bits) (active; 0 open channel(s))>>,^M
417703                     21: <bound method Transport._parse_newkeys of <paramiko.Transport at 0xf9dd7e50 (cipher aes128-ctr, 128 bits) (active; 0 open channel(s))>>,^M
417704                     80: <bound method Transport._parse_global_request of <paramiko.Transport at 0xf9dd7e50 (cipher aes128-ctr, 128 bits) (active; 0 open channel(s))>>,^M
417705                     81: <bound method Transport._parse_request_success of <paramiko.Transport at 0xf9dd7e50 (cipher aes128-ctr, 128 bits) (active; 0 open channel(s))>>,^M
417706                     82: <bound method Transport._parse_request_failure of <paramiko.Transport at 0xf9dd7e50 (cipher aes128-ctr, 128 bits) (active; 0 open channel(s))>>,^M
417707                     90: <bound method Transport._parse_channel_open of <paramiko.Transport at 0xf9dd7e50 (cipher aes128-ctr, 128 bits) (active; 0 open channel(s))>>,^M
                                                                                                                                                                                                 417707,88     74%
417708                     91: <bound method Transport._parse_channel_open_success of <paramiko.Transport at 0xf9dd7e50 (cipher aes128-ctr, 128 bits) (active; 0 open channel(s))>>,^M
417709                     92: <bound method Transport._parse_channel_open_failure of <paramiko.Transport at 0xf9dd7e50 (cipher aes128-ctr, 128 bits) (active; 0 open channel(s))>>},^M
417710  '_ident': 140178932885248,^M
417711  '_initialized': True,^M
417712  '_invoke_excepthook': <function _make_invoke_excepthook.<locals>.invoke_excepthook at 0x7f7df943fa60>,^M
417713  '_is_stopped': False,^M
417714  '_kwargs': {},^M
417715  '_latest_kex_init': b'\x14\xdd\n#\xf8\xa1\xfe0\x0e2\x1b\x8e{8\x10\x03'^M
417716                      b'\x80\x00\x00\x01\x1bcurve25519-sha256@libssh.org,ecdh-s'^M
417717                      b'ha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffi'^M
417718                      b'e-hellman-group16-sha512,diffie-hellman-group-exchange-s'^M
417719                      b'ha256,diffie-hellman-group14-sha256,diffie-hellman-group'^M
417720                      b'-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellma'^M
417721                      b'n-group1-sha1,ext-info-c\x00\x00\x01\xe9rsa-sha2-256,rsa'^M
417722                      b'-sha2-512,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nis'^M
417723                      b'tp384,ecdsa-sha2-nistp521,rsa-sha2-512,rsa-sha2-256,ssh-'^M
417724                      b'rsa,ssh-dss,rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-5'^M
417725                      b'12-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com'^M
417726                      b',ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nis'^M
417727                      b'tp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@'^M
417728                      b'openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-2'^M
417729                      b'56-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh'^M
417730                      b'-dss-cert-v01@openssh.com\x00\x00\x00Jaes128-ctr,aes1'^M
417731                      b'92-ctr,aes256-ctr,aes128-cbc,aes192-cbc,aes256-cbc,3des-'^M
417732                      b'cbc\x00\x00\x00Jaes128-ctr,aes192-ctr,aes256-ctr,aes1'^M
417733                      b'28-cbc,aes192-cbc,aes256-cbc,3des-cbc\x00\x00\x00\x83hma'^M
417734                      b'c-sha2-256,hmac-sha2-512,hmac-sha2-256-etm@openssh.com,h'^M
417735                      b'mac-sha2-512-etm@openssh.com,hmac-sha1,hmac-md5,hmac-sha'^M
417736                      b'1-96,hmac-md5-96\x00\x00\x00\x83hmac-sha2-256,hmac-sha2-'^M
417737                      b'512,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@open'^M
417738                      b'ssh.com,hmac-sha1,hmac-md5,hmac-sha1-96,hmac-md5-96\x00'^M
417739                      b'\x00\x00\x04none\x00\x00\x00\x04none\x00\x00\x00\x00\x00'^M
417740                      b'\x00\x00\x00\x00\x00\x00\x00\x00',^M
417741  '_logged_hash_selection': True,^M
417742  '_name': 'Thread-81',^M
417743  '_native_id': 21402,^M
417744  '_preferred_compression': ('none',),^M
417745  '_remote_ext_info': None,^M
417746  '_started': <threading.Event object at 0x7f7df9dd7ca0>,^M
417747  '_stderr': <_io.TextIOWrapper name='<stderr>' mode='w' encoding='utf-8'>,^M
417748  '_target': None,^M
417749  '_tcp_handler': None,^M
417750  '_tstate_lock': <locked _thread.lock object at 0x7f7df9de63c0>,^M
417751  '_x11_handler': None,^M
417752  'active': True,^M
417753  'auth_handler': <paramiko.auth_handler.AuthHandler object at 0x7f7df9deb070>,^M
417754  'auth_timeout': 30,^M
417755  'authenticated': True,^M
417756  'banner_timeout': 15,^M
417757  'channel_events': {},^M
417758  'channel_timeout': 3600,^M
417759  'channels_seen': {},^M
417760  'clear_to_send': <threading.Event object at 0x7f7df9de6070>,^M
417761  'clear_to_send_lock': <unlocked _thread.lock object at 0x7f7df9de6150>,^M
417762  'clear_to_send_timeout': 30.0,^M
417763  'completion_event': <threading.Event object at 0x7f7df9de6280>,^M
417764  'default_max_packet_size': 32768,^M
417765  'default_window_size': 2097152,^M
417766  'disabled_algorithms': {},^M
417767  'global_response': None,^M
417768  'gss_host': None,^M
417769  'gss_kex_used': False,^M
417770  'handshake_timeout': 15,^M
417771  'host_key': PKey(alg=RSA, bits=2048, fp=SHA256:paKa8cWmnUU7P7Cv10NljlSdcfkYwakbiNY18WVnozw),^M
417772  'host_key_type': 'rsa-sha2-256',^M
417773  'hostname': None,^M
417774  'in_kex': False,^M
417775  'initial_kex_done': True,^M
417776  'kex_engine': None,^M
417777  'kexgss_ctxt': None,^M
417778  'local_cipher': 'aes128-ctr',^M
417779  'local_compression': 'none',^M
417780  'local_kex_init': None,^M
417781  'local_mac': 'hmac-sha2-256',^M
417782  'local_version': 'SSH-2.0-paramiko_3.3.1',^M
417783  'lock': <unlocked _thread.lock object at 0x7f7df9dd7cf0>,^M
417784  'log_name': 'paramiko.transport',^M
417785  'logger': <RuntestsLogger paramiko.transport (INFO)>,^M
417786  'packetizer': <paramiko.packet.Packetizer object at 0x7f7df9dd7cd0>,^M
417787  'remote_cipher': 'aes128-ctr',^M
417788  'remote_compression': 'none',^M
417789  'remote_kex_init': None,^M
417790  'remote_mac': 'hmac-sha2-256',^M
417791  'remote_version': 'SSH-2.0-OpenSSH_9.1 PKIX[13.5 FIPS]',^M
417792  'saved_exception': None,^M
417793  'server_accept_cv': <Condition(<unlocked _thread.lock object at 0x7f7df9dd7cf0>, 0)>,^M
417794  'server_accepts': [],^M
417795  'server_extensions': {'publickey-algorithms@roumenpetrov.info': b'x509v3-ecdsa'^M
417796                                                                  b'-sha2-nistp2'^M
417797                                                                  b'56,x509v3-ec'^M
417798                                                                  b'dsa-sha2-nis'^M
417799                                                                  b'tp384,x509v3'^M
417800                                                                  b'-ecdsa-sha2-'^M
417801                                                                  b'nistp521,x50'^M
417802                                                                  b'9v3-ssh-rsa,'^M
417803                                                                  b'x509v3-rsa20'^M
417804                                                                  b'48-sha256,ss'^M
417805                                                                  b'h-ed25519,ss'^M
417806                                                                  b'h-dss,ecdsa-'^M
417807                                                                  b'sha2-nistp25'^M
417808                                                                  b'6,ecdsa-sha2'^M
417809                                                                  b'-nistp384,ec'^M
417810                                                                  b'dsa-sha2-nis'^M
417811                                                                  b'tp521,ssh-rs'^M
417812                                                                  b'a,rsa-sha2-2'^M
417813                                                                  b'56,rsa-sha2-'^M
417814                                                                  b'512',^M
417815                        'server-sig-algs': b'x509v3-ecdsa-sha2-nistp256,x509v'^M
417816                                           b'3-ecdsa-sha2-nistp384,x509v3-ecd'^M
417817                                           b'sa-sha2-nistp521,x509v3-ssh-rsa,'^M
417818                                           b'x509v3-rsa2048-sha256,ssh-ed2551'^M
417819                                           b'9,ssh-dss,ecdsa-sha2-nistp256,ec'^M
417820                                           b'dsa-sha2-nistp384,ecdsa-sha2-nis'^M
417821                                           b'tp521,ssh-rsa,rsa-sha2-256,rsa-s'^M
417822                                           b'ha2-512'},^M
417823  'server_key_dict': {},^M
417824  'server_mode': False,^M
417825  'server_object': None,^M
417826  'server_sig_algs': True,^M
417827  'session_id': b'[\xde\\p\n~Mlt\xf7\xd5\xb2\x07b\x80\x08Pe\x12\x94'^M
417828                b'\xdc\t\xd5\x05#FJ\x1c7\xcb0\xed',^M
417829  'sock': <socket.socket fd=183, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.193.170.82', 34004), raddr=('10.193.169.147', 22)>,^M
417830  'subsystem_table': {},^M
417831  'sys': <module 'sys' (built-in)>,^M
417832  'use_gss_kex': False}^M

418662 Client object at 0x7f7df942a460> 21327^

418666 {'H': b'\xa6x\t\xabD\xc7L\xc1^\xc8N\xe3\x7f\xc9&f0w\x10\xef\x03\xe93\x8c'^M
418667       b'g\xcee\x9aB\xe2\x00\xd3',^M
418668  'K': None,^M
418669  '_agreed_pubkey_algorithm': 'rsa-sha2-512',^M
418670  '_args': (),^M
418671  '_channel_counter': 1,^M
418672  '_channels': <paramiko.transport.ChannelMap object at 0x7f7df942abe0>,^M
418673  '_daemonic': True,^M
418674  '_expected_packet': (),^M
418675  '_forward_agent_handler': None,^M
418676  '_handler_table': {7: <bound method Transport._parse_ext_info of <paramiko.Transport at 0xf942a790 (unconnected)>>,^M
418677                     20: <bound method Transport._negotiate_keys of <paramiko.Transport at 0xf942a790 (unconnected)>>,^M
418678                     21: <bound method Transport._parse_newkeys of <paramiko.Transport at 0xf942a790 (unconnected)>>,^M
418679                     80: <bound method Transport._parse_global_request of <paramiko.Transport at 0xf942a790 (unconnected)>>,^M
418680                     81: <bound method Transport._parse_request_success of <paramiko.Transport at 0xf942a790 (unconnected)>>,^M
418681                     82: <bound method Transport._parse_request_failure of <paramiko.Transport at 0xf942a790 (unconnected)>>,^M
418682                     90: <bound method Transport._parse_channel_open of <paramiko.Transport at 0xf942a790 (unconnected)>>,^M
418683                     91: <bound method Transport._parse_channel_open_success of <paramiko.Transport at 0xf942a790 (unconnected)>>,^M
418684                     92: <bound method Transport._parse_channel_open_failure of <paramiko.Transport at 0xf942a790 (unconnected)>>},^M
418685  '_ident': 140178924492544,^M
418686  '_initialized': True,^M
418687  '_invoke_excepthook': <function _make_invoke_excepthook.<locals>.invoke_excepthook at 0x7f7df9ed33a0>,^M
418688  '_is_stopped': True,^M
418689  '_kwargs': {},^M
418690  '_latest_kex_init': b"\x14\xa3\xb4\xb1:\xea:\x0f\x9f\xda\x0b'\xc2\xf8'^"^M
418691                      b'\x87\x00\x00\x01\x1bcurve25519-sha256@libssh.org,ecdh-s'^M
418692                      b'ha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffi'^M
418693                      b'e-hellman-group16-sha512,diffie-hellman-group-exchange-s'^M
418694                      b'ha256,diffie-hellman-group14-sha256,diffie-hellman-group'^M
418695                      b'-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellma'^M
418696                      b'n-group1-sha1,ext-info-c\x00\x00\x01\xe9rsa-sha2-256,rsa'^M
418697                      b'-sha2-512,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nis'^M
418698                      b'tp384,ecdsa-sha2-nistp521,rsa-sha2-512,rsa-sha2-256,ssh-'^M
418699                      b'rsa,ssh-dss,rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-5'^M
418700                      b'12-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com'^M
418701                      b',ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nis'^M
418702                      b'tp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@'^M
418703                      b'openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-2'^M
418704                      b'56-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh'^M
418705                      b'-dss-cert-v01@openssh.com\x00\x00\x00Jaes128-ctr,aes1'^M
418706                      b'92-ctr,aes256-ctr,aes128-cbc,aes192-cbc,aes256-cbc,3des-'^M
418707                      b'cbc\x00\x00\x00Jaes128-ctr,aes192-ctr,aes256-ctr,aes1'^M
418708                      b'28-cbc,aes192-cbc,aes256-cbc,3des-cbc\x00\x00\x00\x83hma'^M
418709                      b'c-sha2-256,hmac-sha2-512,hmac-sha2-256-etm@openssh.com,h'^M
418710                      b'mac-sha2-512-etm@openssh.com,hmac-sha1,hmac-md5,hmac-sha'^M
                                                                                                                                                                                                 418684,120    74%
418711                      b'1-96,hmac-md5-96\x00\x00\x00\x83hmac-sha2-256,hmac-sha2-'^M
418712                      b'512,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@open'^M
418713                      b'ssh.com,hmac-sha1,hmac-md5,hmac-sha1-96,hmac-md5-96\x00'^M
418714                      b'\x00\x00\x04none\x00\x00\x00\x04none\x00\x00\x00\x00\x00'^M
418715                      b'\x00\x00\x00\x00\x00\x00\x00\x00',^M
418716  '_logged_hash_selection': True,^M
418717  '_name': 'Thread-74',^M
418718  '_native_id': 20778,^M
418719  '_preferred_compression': ('none',),^M
418720  '_remote_ext_info': None,^M
418721  '_started': <threading.Event object at 0x7f7df942aa60>,^M
418722  '_stderr': <_io.TextIOWrapper name='<stderr>' mode='w' encoding='utf-8'>,^M
418723  '_target': None,^M
418724  '_tcp_handler': None,^M
418725  '_tstate_lock': None,^M
418726  '_x11_handler': None,^M
418727  'active': False,^M
418728  'auth_handler': <paramiko.auth_handler.AuthHandler object at 0x7f7df9ef7c40>,^M
418729  'auth_timeout': 30,^M
418730  'authenticated': True,^M
418731  'banner_timeout': 15,^M
418732  'channel_events': {},^M
418733  'channel_timeout': 3600,^M
418734  'channels_seen': {0: True},^M
418735  'clear_to_send': <threading.Event object at 0x7f7df942acd0>,^M
418736  'clear_to_send_lock': <unlocked _thread.lock object at 0x7f7df942adb0>,^M
418737  'clear_to_send_timeout': 30.0,^M
418738  'completion_event': <threading.Event object at 0x7f7df9444d90>,^M
418739  'default_max_packet_size': 32768,^M
418740  'default_window_size': 2097152,^M
418741  'disabled_algorithms': {},^M
418742  'global_response': None,^M
418743  'gss_host': None,^M
418744  'gss_kex_used': False,^M
418745  'handshake_timeout': 15,^M
418746  'host_key': PKey(alg=RSA, bits=2048, fp=SHA256:paKa8cWmnUU7P7Cv10NljlSdcfkYwakbiNY18WVnozw),^M
418747  'host_key_type': 'rsa-sha2-256',^M
418748  'hostname': None,^M
418749  'in_kex': False,^M
418750  'initial_kex_done': True,^M
418751  'kex_engine': None,^M
418752  'kexgss_ctxt': None,^M
418753  'local_cipher': 'aes128-ctr',^M
418754  'local_compression': 'none',^M
418755  'local_kex_init': None,^M
418756  'local_mac': 'hmac-sha2-256',^M
418757  'local_version': 'SSH-2.0-paramiko_3.3.1',^M
418758  'lock': <unlocked _thread.lock object at 0x7f7df942aba0>,^M
418759  'log_name': 'paramiko.transport',^M
418760  'logger': <RuntestsLogger paramiko.transport (INFO)>,^M
418761  'packetizer': <paramiko.packet.Packetizer object at 0x7f7df942ab50>,^M
418762  'remote_cipher': 'aes128-ctr',^M
418763  'remote_compression': 'none',^M
418764  'remote_kex_init': None,^M
418765  'remote_mac': 'hmac-sha2-256',^M
418766  'remote_version': 'SSH-2.0-OpenSSH_9.1 PKIX[13.5 FIPS]',^M
418767  'saved_exception': EOFError(),^M
418768  'server_accept_cv': <Condition(<unlocked _thread.lock object at 0x7f7df942aba0>, 0)>,^M
418769  'server_accepts': [],^M
418770  'server_extensions': {'publickey-algorithms@roumenpetrov.info': b'x509v3-ecdsa'^M
418771                                                                  b'-sha2-nistp2'^M
418772                                                                  b'56,x509v3-ec'^M
418773                                                                  b'dsa-sha2-nis'^M
418774                                                                  b'tp384,x509v3'^M
418775                                                                  b'-ecdsa-sha2-'^M
418776                                                                  b'nistp521,x50'^M
418777                                                                  b'9v3-ssh-rsa,'^M
418778                                                                  b'x509v3-rsa20'^M
418779                                                                  b'48-sha256,ss'^M
418780                                                                  b'h-ed25519,ss'^M
418781                                                                  b'h-dss,ecdsa-'^M
418782                                                                  b'sha2-nistp25'^M
418783                                                                  b'6,ecdsa-sha2'^M
418784                                                                  b'-nistp384,ec'^M
418785                                                                  b'dsa-sha2-nis'^M
418786                                                                  b'tp521,ssh-rs'^M
418787                                                                  b'a,rsa-sha2-2'^M
418788                                                                  b'56,rsa-sha2-'^M
418789                                                                  b'512',^M
418790                        'server-sig-algs': b'x509v3-ecdsa-sha2-nistp256,x509v'^M
418791                                           b'3-ecdsa-sha2-nistp384,x509v3-ecd'^M
418792                                           b'sa-sha2-nistp521,x509v3-ssh-rsa,'^M
418793                                           b'x509v3-rsa2048-sha256,ssh-ed2551'^M
418794                                           b'9,ssh-dss,ecdsa-sha2-nistp256,ec'^M
418795                                           b'dsa-sha2-nistp384,ecdsa-sha2-nis'^M
                                                                                                                                                                                                 418750,27     74%

418796                                           b'tp521,ssh-rsa,rsa-sha2-256,rsa-s'^M
418797                                           b'ha2-512'},^M
418798  'server_key_dict': {},^M
418799  'server_mode': False,^M
418800  'server_object': None,^M
418801  'server_sig_algs': True,^M
418802  'session_id': b'\xa6x\t\xabD\xc7L\xc1^\xc8N\xe3\x7f\xc9&f0w\x10\xef'^M
418803                b'\x03\xe93\x8cg\xcee\x9aB\xe2\x00\xd3',^M
418804  'sock': <socket.socket [closed] fd=-1, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0>,^M
418805  'subsystem_table': {},^M
418806  'sys': <module 'sys' (built-in)>,^M
418807  'use_gss_kex': False}^M

418861     self.stdin.write(data)^M
418862   File "/home/rajraghu/Envs/proj3/lib/python3.8/site-packages/paramiko/file.py", line 389, in write^M
418863     self._write_all(data)^M
418864   File "/home/rajraghu/Envs/proj3/lib/python3.8/site-packages/paramiko/file.py", line 507, in _write_all^M
418865     count = self._write(data)^M
418866   File "/home/rajraghu/Envs/proj3/lib/python3.8/site-packages/paramiko/channel.py", line 1362, in _write^M
418867     self.channel.sendall(data)^M
418868   File "/home/rajraghu/Envs/proj3/lib/python3.8/site-packages/paramiko/channel.py", line 844, in sendall^M
418869     sent = self.send(s)^M
418870   File "/home/rajraghu/Envs/proj3/lib/python3.8/site-packages/paramiko/channel.py", line 799, in send^M
418871     return self._send(s, m)^M
418872   File "/home/rajraghu/Envs/proj3/lib/python3.8/site-packages/paramiko/channel.py", line 1196, in _send^M
418873     raise socket.error("Socket is closed")^M
418874 OSError: Socket is closed^M

418878 Client object at 0x7f7df9444310> 21327 ----------####################^M
418879 paramiko.transport(1895): INFO     starting thread (client mode): 0xf9de3fd0^M
419438 paramiko.transport(1895): INFO     ############ DONE WITH self.packetizer.read_message 21327 ##############^M
419439 paramiko.transport(1895): INFO     === Key exchange possibilities ===^M
419440 paramiko.transport(1895): INFO     kex algos: ecdh-sha2-nistp256, ecdh-sha2-nistp384, ecdh-sha2-nistp521, diffie-hellman-group16-sha512, diffie-hellman-group14-sha1^M
419441 paramiko.transport(1895): INFO     server key: ssh-rsa, rsa-sha2-256, rsa-sha2-512^M
419442 paramiko.transport(1895): INFO     client encrypt: aes128-ctr, aes192-ctr, aes256-ctr, aes128-gcm@openssh.com, aes256-gcm@openssh.com^M
419443 paramiko.transport(1895): INFO     server encrypt: aes128-ctr, aes192-ctr, aes256-ctr, aes128-gcm@openssh.com, aes256-gcm@openssh.com^M
419444 paramiko.transport(1895): INFO     client mac: hmac-sha1, hmac-sha2-256, hmac-sha2-512^M
419445 paramiko.transport(1895): INFO     server mac: hmac-sha1, hmac-sha2-256, hmac-sha2-512^M
419446 paramiko.transport(1895): INFO     client compress: none, zlib@openssh.com^M
419447 paramiko.transport(1895): INFO     server compress: none, zlib@openssh.com^M
419448 paramiko.transport(1895): INFO     client lang: <none>^M
419449 paramiko.transport(1895): INFO     server lang: <none>^M
419450 paramiko.transport(1895): INFO     kex follows: False^M
419451 paramiko.transport(1895): INFO     === Key exchange agreements ===^M
419452 paramiko.transport(1895): INFO     Kex: ecdh-sha2-nistp256^M
419453 paramiko.transport(1895): INFO     HostKey: rsa-sha2-256^M
419454 paramiko.transport(1895): INFO     Cipher: aes128-ctr^M
419455 paramiko.transport(1895): INFO     MAC: hmac-sha2-256^M
419456 paramiko.transport(1895): INFO     Compression: none^M
419457 paramiko.transport(1895): INFO     === End of kex handshake ===^M
419458 paramiko.transport(1895): INFO     ############ KEX_INIT FUNCTION DONE 21327  ##############^M

-------------------------------- HANGS HERE -------------------------------------------------------------------  PROCESS 21327 

Can you let me know if this gives any insight. What i see is that while processing the connection for 21327 i do see lot of hex dumps happening for process 21326 connection as it is succesful and it keeps going on with the connection.

saiaprameya commented 1 year ago

Let me know if we can connect over ZOOM call. I can probably show and explain the problem if that works out in any way? I work in PST timezone.

jun66j5 commented 1 year ago
  1. Parent process 16636 creates a connection and after some time it run atfork() on that connection before Forking 2 processes (Process 21326 and Process 21327)

The step is incorrect. Don't invoke atfork() before invoking fork. Invoke atfork() on other processes except the transport in one process after invoking fork. If parent process will not use the ssh session, I'd suggest connecting to remote server after invoking fork.

jun66j5 commented 1 year ago

Let me know if we can connect over ZOOM call. I can probably show and explain the problem if that works out in any way? I work in PST timezone.

No. I absolutely will not contact via ZOOM call. Here is not free-support.

jun66j5 commented 1 year ago

When Python script hangs, you could use gdb and python3.X-dbg to investigate where the code is running/stopping. See also https://wiki.python.org/moin/DebuggingWithGdb.

saiaprameya commented 1 year ago

Hi Jun Omae, Can you elaborate on this "Invoke atfork() on other processes except the transport in one process after invoking fork." I could not understand what you meant here?

More over its hierarchical where we have the process invocation that can go 4 - 5 levels deep. (Parent -> Child -> Child -> Child) Parent process uses ssh connection for some activity and then it starts forking child processes. Its like Suite setup calling multiple Testcases which has setup, runtest and cleanup. And this can have multilevel where we can have suite1 calling suite2 calling suite3 and then multiple testcases under it.

So considering this how do you intend to use atfork and when?

jun66j5 commented 1 year ago

Can you elaborate on this "Invoke atfork() on other processes except the transport in one process after invoking fork." I could not understand what you meant here?

You said "Parent process 16636 creates a connection and after some time it run atfork() on that connection before Forking 2 processes "

I understand that your code is like this:

Please share your code without changes (I don't understand what you want to do and your code).

jun66j5 commented 1 year ago

So considering this how do you intend to use atfork and when?

Typically, the child process should use atfork() when 0 be returned from os.fork() if ssh connection in the parent process is not closed. As the result, the parent process continues to be able use the ssh connection and the child process doesn't wrongly use the ssh connection.

See also https://github.com/paramiko/paramiko/commit/6a145dea09e8781711adcfe1c718ee5fd6c4f3fd

saiaprameya commented 1 year ago

Hi Jun Omae, Thanks a lot for your suggestion. After moving the "atfork" inside the forked process the problem got resolved.

Curious to know how this works when moved inside the forked and there is no contention for the mutex any more?

Regards Sai

jun66j5 commented 1 year ago

Please close this issue if it is solved.

Curious to know how this works when moved inside the forked ...

The root cause is that you try to use the ssh session although the connection has been closed by atfork (lack of understanding fork, socket and atfork).