scp.put fails with large files

pwo commented 3 weeks ago

SUMMARY

Copying large file with scp.put results in the remote file becoming 0 bytes and pylibssh fails with an exception. The size limit seems to be 2g, truncate -s 2047m works, truncate -s 2048m does not for the source file.

ISSUE TYPE

Bug Report

PYLISSH and LIBSSH VERSION

<pylibsshext v1.2.2 with libssh v0.9.6>

OS / ENVIRONMENT

Controller: RHEL 9.4 Python 3.9 ansible 8.7.0 ansible-core 2.15.12 Ansible installed from pypi in venv.

Target device: Cisco Nexus N9K-C9336C-FX2, NX-OS 10.2(7)

Also reproducible against sshd on CentOS Stream 9. OpenSSH_8.7p1, OpenSSL 3.2.2 4 Jun 2024

STEPS TO REPRODUCE

$ ./scp-test.py
Enter password:
ssh.is_connected=1
Traceback (most recent call last):
  File "/opt/ansible/./scp-test.py", line 20, in <module>
    scp.put(local_file, remote_file)
  File "src/pylibsshext/scp.pyx", line 52, in pylibsshext.scp.SCP.put
  File "src/pylibsshext/scp.pyx", line 85, in pylibsshext.scp.SCP.put
pylibsshext.errors.LibsshSCPException: Can't write to remote file: b''

#!/usr/bin/env python

import getpass

from pylibsshext.session import Session

ssh = Session()

ssh_password = getpass.getpass(prompt='Enter password: ')

ssh.connect(host='cisco-n9k', user='admin', password=ssh_password, timeout=30, port=22, host_key_checking=False)

print(f'{ssh.is_connected=}')

local_file = '/var/lib/tftpboot/n9k/10.3.6/nxos64-cs.10.3.6.M.bin' # 2437178880 bytes, does not work
#local_file = '/var/lib/tftpboot/n9k/10.3.6/n9000-epld.10.3.6.M.img' # 289312032 bytes, works
remote_file = 'test.bin'

scp = ssh.scp()
scp.put(local_file, remote_file)

EXPECTED RESULTS

Would expect the scp to not fail.

ACTUAL RESULTS

The remote_file ends up being 0 bytes. The "end-user" failure of this is that the cisco.nxos.nxos_file_copy ansible module fails with: "msg": "Exception received : Error transferring file to nxos64-cs.10.3.6.M.bin: Can't write to remote file: b''",

Regular OpenSSH scp works fine for both files. OpenSSH_8.7p1, OpenSSL 3.0.7 1 Nov 2022

NilashishC commented 3 weeks ago

@pwo There are some known issue with libssh and SCP. Would you mind trying this with SFTP instead and let us know if this works for you? Thanks!

Jakuje commented 1 week ago

This sounds like an issue of the SCP implementation in pylibssh, where it tries to read the whole file into a single memory buffer in here:

https://github.com/ansible/pylibssh/blob/cc2ceff272d390f1b95cbe97dbd3770ec8db67c6/src/pylibsshext/scp.pyx#L83C48-L83C65

This is likely not going to work for large files. The SCP should read the file per smaller chunks and feed it to the libssh. I am not sure if it fails just from nowhere, or fills the whole RAM as the buffers might end up being copied to some other places.

ansible / pylibssh