Fix reading large files over SCP

Jakuje commented 3 months ago

SUMMARY

The current SCP code does not handle large reads. Instead just allocates the large buffer fills up to 64k bytes and then writes the whole to the file (most likely the rest filled with junk).

The libssh returns the number of read bytes from ssh_scp_read() and for practical reasons does not allocate the whole file in memory. The reads are capped to 64k B.

This PR fixes the read by both not allocating the entire file size in memory and reading it by chunks (that much max libssh chunk atm) and writing that to the local file.

ISSUE TYPE

Bugfix Pull Request

ADDITIONAL INFORMATION

The attached test case is failing without the change, with errors like this:

_______________________________________________________________________ test_get_large _______________________________________________________________________
[gw0] linux -- Python 3.12.3 /home/jjelen/devel/pylibssh/.tox/python/bin/python

dst_path = PosixPath('/tmp/pytest-of-jjelen/pytest-35/popen-gw0/test_get_large0/dst-file.txt')
src_path_large = PosixPath('/tmp/pytest-of-jjelen/pytest-35/popen-gw0/test_get_large0/large.txt'), ssh_scp = <pylibsshext.scp.SCP object at 0x7f3f83cffac0>
large_payload = b'&PE~~,!=1@F7RaUccg#)q-ru^~f8Zv/-{~tFE-\\>v2_b}i@`N/]_%*~{oD`"F`*4Ns]HK^lewILvC1O[]gFC9(27%MuX$s&24et<dRJ$"& zf_XIqm7...**<J_;?uI47vm\'/A"xi5a!kQXoL=M}ne;$E[B=EpasVbRO&\\zy7t,GK yu;l*$s<#P:sJLi@od#:&:m?8]9KFN4@QAF8hD, ~T@WNP;)%+>)TV\'ODi9'

    def test_get_large(dst_path, src_path_large, ssh_scp, large_payload):
        """Check that SCP file download works also for large files."""
>       ssh_scp.get(str(src_path_large), str(dst_path))

dst_path   = PosixPath('/tmp/pytest-of-jjelen/pytest-35/popen-gw0/test_get_large0/dst-file.txt')
large_payload = (b'&PE~~,!=1@F7RaUccg#)q-ru^~f8Zv/-{~tFE-\\>v2_b}i@`N/]_%*~{oD`"F`*4Ns]HK^le'
[...]
 b"?8]9KFN4@QAF8hD, ~T@WNP;)%+>)TV'ODi9")
src_path_large = PosixPath('/tmp/pytest-of-jjelen/pytest-35/popen-gw0/test_get_large0/large.txt')
ssh_scp    = <pylibsshext.scp.SCP object at 0x7f3f83cffac0>

tests/unit/scp_test.py:111: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   raise LibsshSCPException("Unexpected request: %s" % self._get_ssh_error_str())
E   pylibsshext.errors.LibsshSCPException: Unexpected request: b'ssh_scp_pull_request called under invalid state'

src/pylibsshext/scp.pyx:148: LibsshSCPException

(lines do not match)

Additionally, there is one more test case, that I suspected will be problematic, but it turned out it works just ok, so leaving for the coverage.

packit-as-a-service[bot] commented 3 months ago

Congratulations! One of the builds has completed. :champagne:

You can install the built RPMs by following these steps:

sudo yum install -y dnf-plugins-core on RHEL 8
sudo dnf install -y dnf-plugins-core on Fedora
dnf copr enable packit/ansible-pylibssh-621
And now you can install the packages.

Please note that the RPMs should be used only in a testing environment.