SFTP extremely slow with libssh

Andyjb8 commented 1 year ago

I did not see this specific issue reported yet. But this issue is similar https://github.com/ansible/pylibssh/issues/208

SUMMARY

I am using net_put module with protocol sftp to push software file to a Juniper SRX. The transfer is extremely slow (16min for 400MB file) when it uses the libssh but is very quick with paramiko. (30 seconds). I also verified that a manual sftp from the same ansible server to same destination was quick, so its not the network path, the issue is with the net_put module.

ISSUE TYPE

Bug Report

PYLISSH and LIBSSH VERSION

ansible-pylibssh 1.1.0

ansible --version ansible [core 2.14.5] config file = /home/andy/ansible.cfg configured module search path = ['/home/andy/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules'] ansible python module location = /usr/lib/python3/dist-packages/ansible ansible collection location = /home/andy/.ansible/collections:/usr/share/ansible/collections executable location = /usr/bin/ansible python version = 3.10.6 (main, Mar 10 2023, 10:55:28) [GCC 11.3.0] (/usr/bin/python3) jinja version = 3.0.3 libyaml = True

OS / ENVIRONMENT

Target OS: vSRX runing Junos: 22.2R2.10 Also have seen same issue with WTI ([Western Telematic) console servers.

STEPS TO REPRODUCE

I am able to reproduce this issue my running the below playbook:

hosts: FW-01 tasks:
- name: PUSH TEST FILE TO DEVICE ansible.netcommon.net_put: src: /home/software/juniper/junos-srxsme-22.1R3.9.tgz dest: /var/tmp/junos-srxsme-22.1R3.9.tgz protocol: sftp vars: ansible_connection: network_cli ansible_network_cli_ssh_type: paramiko ### FAST WHEN PARAMIKO IS ENABLED, SLOW WHEN I REMOVE THIS.

EXPECTED RESULTS

I expect pylibssh to not take way longer than it should to transfer file.

ACTUAL RESULTS

pylibssh takes about 16minutes longer than paramiko for 400Mb file

Andyjb8 commented 1 year ago

Also, in testing this paramiko fix with Juniper SRX300s, I found that I did not need to specify protocol sftp. Before with libssh, I had to specify "protocol: sftp" instead of using default protocol scp since scp did not work. With libssh and scp I got the message FAILED! => {"changed": false, "destination": /var/temo/", "msg": "Exception received: unable to connect to socket/username/.ansible/pc12eeb156f2. See the socket path issue category in Network Debug and Troubleshooting Guide"

When I added "ansible_network_cli_ssh_type: paramiko" I was able to use net_put with the default protocol scp and it worked without getting the above message.

Jakuje commented 3 months ago

The libssh will provide a new SFTP api, which can operate asynchronously and therefore get much better throughput, see https://gitlab.com/libssh/libssh-mirror/-/merge_requests/375 Some early testing would be welcomed.

NilashishC commented 1 month ago

@Jakuje looks like the merge request you shared is closed now. Has there been any updates since then?

Jakuje commented 1 month ago

The MR was merged, the API is ready for testing and feedback in master now. We expect next major release with this API in coming ~month.

webknjaz commented 3 days ago

@Jakuje we still bundle + test against 0.9.4. Bumping it in this lib would require some work. Not sure if we'll need to support both 0.9.x + 0.10.x or the latest is fine.

Jakuje commented 3 days ago

As an libssh upstream, we support the last two versions, which is now 0.9 and 0.10. But with the 0.11 release we will drop the 0.9 so porting to latest version and making sure it works will be needed anyway.

I am not aware of any breaking changes, but there might be things we missed.

ansible / pylibssh