extremely slow file transfer

Description

i'm facing a very slow file transfer, i don't know if i'm missing anything, but scp and my dart implementation using libssh manages to finish the transfer much faster.

Steps to reproduce

fn main() {
    //configure log
    CombinedLogger::init(
        vec![
            TermLogger::new(LevelFilter::Debug, Config::default(), TerminalMode::Mixed, ColorChoice::Auto),           
        ]
    ).unwrap();
    //configure host
    let localhost = Localhost::new(PathBuf::from(r"C:\MyRustProjects\fsbackup_engine")).unwrap();

    //connection setup
    let config = ProtocolParams::Generic(GenericProtocolParams {
        address: "192.168.133.13".to_string(),
        port: 22,
        username: Some(String::from("isaque.neves")),
        password: Some(String::from("Ins257257")),
    });

    let mut activity = FileTransferActivity::new(localhost, FileTransferProtocol::Scp, config);
    activity.connect();

    //let file_to_download = FsEntry::File(FsFile::from_str("/var/www/.profile")); //this work
    let dir_to_download = FsEntry::Directory(FsDirectory::from_str("/var/www/html/Alex"));
    let dest_dir_path = r"C:\MyRustProjects\fsbackup_engine\download";
    let destiny= PathBuf::from(dest_dir_path);
    //create directory if it doesn't exist
    std::fs::create_dir_all(dest_dir_path).expect(format!("Failed to create directory: {}", dest_dir_path).as_str());

    match activity.filetransfer_recv(TransferPayload::Any(dir_to_download), &destiny, None) {
        Ok(result) => {
            println!("result: {:?}", result);
        }
        Err(e) => {
            println!("error: {}", e);//eprintln!()
            process::exit(0);//1
        }
    }
}

Expected behaviour

the expected behavior should be copying the directory without any errors

Environment

OS: Linux Debian 10 to Windows 11
Architecture [x86_64]
Rust version rustc 1.56.1 (59eed8a2a 2021-11-01)
termscp version last
Protocol used SCP
Remote server version and name Linux Debian 10

Additional information

With SCP: 14 seconds

With rust + lib ssh2: 132.175213 seconds

in dart + libssh

PS C:\MyDartProjects\fsbackup\packages\libssh_binding> .\example\main.exe 
scpDownloadDirectory: total size: 48958110 in bytes | 46.69009208679199 megabytes of directory /var/www/html/portalPmro
scpDownloadDirectory: end of directories
scpDownloadDirectory: total size: 48958110 | copied: 48958110

0:00:10.203692

Not a big surprise for me.

When you call this function:

activity.filetransfer_recv(TransferPayload::Any(dir_to_download), &dest, None)

it's just not transferring files, it's mounting UI components, calculating progress, and polling for your IO to check if you've pressed some keys (and polling is taking some time at each cycle). Termscp is a UI application, not a library.

I cannot say what's going wrong, especially because this just doesn't look right.

I'm taking some days on the next week to finish the implementation of remotefs-rs, so please, wait for the release, before any comparison, because using termscp as a library is not good and I'm going to forbid it as soon as 0.8.0 will be out.

For those interested in the binary development

Anyway, just for the record, I've made at the time some benchmarks related to filetransfer speed and the results are basically:

SFTP is kinda slow, but is a problem related to libssh2 (I also reported the issue to the rust library maintainers, without much success)
SCP is slightly slower than the scp binary: the issue is always the same. We're comparing a pure application which justs writes progress on stdout once a second against a TUI which makes tons of operations; there is an entire UI framework which runs underneath. I've never tested HUGE amount of files and I won't anyway. I don't think my base user use case is transferring TB of files.

Moving the transfer client in a dedicated thread could be an idea, but I'm postponing this task at each update, because it's just too long to implement and I'm slowly drowning in the amount of job I have to do on the other projects I'm following. I just hope I will find the time to do this, considering that it would improve performance a lot, since it would make the transfer independent from the UI thread.

@veeso

it's just not transferring files, it's mounting UI components, calculating progress, and polling for your IO to check if you've pressed some keys (and polling is taking some time at each cycle). Termscp is a UI application, not a library.

It's not because of the UI, I also thought it might be because of that, but it's another problem because I commented on all the parts related to the UI, I commented on the line that calls total calculation, I haven't figured out the problem yet, but it's not related to the UI, I think there is a problem with reading the data, as I commented on the line that writes to disk and even so it is very slow, my implementation in dart is much faster, which is strange to me I am using libssh which many say it is slower than libssh2, I'm trying to assemble a simple code here to try to figure out the problem.

I did more testing and ended up porting the code I implemented in dart to rust, after testing I realized that libssh recursive file transfer is extremely faster than libssh2, how to speed up transfers with libssh2

Time elapsed in file transfer: 11.2026743s
download of "/var/www/html/portalPmro" complete!

use ssh::*;
use std::io::{Read, Write};
use std::path::{Path, PathBuf};
use test_ssh::{FsEntry, FsEntryType};

const BUFFER_SIZE: usize = 1024 * 128;// 100 * 1024;  128 *

pub fn run() {
    let mut ssh = Libssh1::new("192.168.133.13").unwrap();
    ssh.connect("isaque.neves", "Ins257257");

    let dir_to_download = Path::new("/var/www/html/portalPmro");
    let dest_dir_path = Path::new(r"C:/MyRustProjects/test_ssh/download");
    std::fs::create_dir_all(&dest_dir_path).unwrap();

    let start = std::time::Instant::now();
    /* let items_to_download = ssh.list_dir(&dir_to_download);
     println!("Time elapsed in list dir: {:?}", start.elapsed());

     let start = std::time::Instant::now();
     for item in items_to_download.iter() {
         //remove a parte inicial do caminho
         let mut dst_path = PathBuf::from(&item.path.strip_prefix(&dir_to_download).unwrap());
         dst_path = dest_dir_path.join(dst_path);
         ssh.download_item(&item, &dst_path);
         println!("item {}", item.path.display());
     }*/

    ssh.download_dir_recursive(dir_to_download, dest_dir_path);
    println!("Time elapsed in file transfer: {:?}", start.elapsed());
    println!("download of {:?} complete!", dir_to_download);

    //println!("out: {}", ssh.run_cmd("ls -la").unwrap())
}

pub struct Libssh1 {
    pub session: Session,
}

impl Libssh1 {
    pub fn new(host: &str) -> Result<Libssh1, ()> {
        let mut session = Session::new().unwrap();
        session.set_host(host).unwrap();
        session.parse_config(None).unwrap();
        Ok(Libssh1 {
            session
        })
    }
    pub fn connect(&mut self, user: &str, pass: &str) {
        self.session.set_username(user).unwrap();
        self.session.connect().unwrap();
        self.session.userauth_password(pass).unwrap();
    }

    pub fn download_item(&mut self, entry: &FsEntry, dst_path: &Path) {
        if entry.file_type == FsEntryType::File {
            let mut scp = self.session.scp_new(READ, &entry.path).unwrap();

            scp.init().unwrap();
            loop {
                match scp.pull_request().unwrap() {
                    Request::NEWFILE => {
                        //let mut buffer: [u8; BUFFER_SIZE] = [0; BUFFER_SIZE];
                        let mut buffer: Vec<u8> = vec!();
                        scp.accept_request().unwrap();
                        let src_file = scp.reader();//.read_to_end(&mut buf).unwrap();

                        let mut dest_file = std::fs::File::create(&dst_path).unwrap();

                        loop {
                            let num_read = src_file
                                .read_to_end(&mut buffer).expect("Could not read from remote");
                            if num_read == 0 {
                                break;
                            }
                            dest_file.write_all(&buffer).expect("Could not write");
                        }

                        break;
                    }
                    Request::WARNING => {
                        scp.deny_request().unwrap();
                        break;
                    }
                    _ => scp.deny_request().unwrap()
                }
            }
        } else if entry.file_type == FsEntryType::Directory {
            std::fs::create_dir_all(&dst_path)
                .expect(format!("Failed to create directory: {:?}", &dst_path).as_str());
        }
    }
    /// this is fast do download directory
    pub fn download_dir_recursive(&mut self, source_path: &Path, dst_path: &Path) {
        let mut scp = self.session.scp_new(READ | RECURSIVE, source_path).unwrap();

        match scp.init() {
            Ok(..) => {}
            Err(e) => { println!("error on init scp") }
        }

        let mut current_path = PathBuf::from(dst_path);

        loop {
            match scp.pull_request().unwrap() {
                Request::NEWFILE => {
                    let raw_filename = scp.request_get_filename().unwrap();
                    let temp_filename = String::from_utf8_lossy(raw_filename).to_owned();

                    let full_local_path_target = PathBuf::from(&current_path.join(&*temp_filename));
                    println!("full_local_path_target {:?}", full_local_path_target);

                    let mut dest_file = std::fs::File::create(full_local_path_target).unwrap();

                    scp.accept_request().unwrap();
                    let src_file = scp.reader();
                    let mut buffer: Vec<u8> = vec!();
                    src_file
                        .read_to_end(&mut buffer).expect("Could not read from remote");
                    dest_file.write_all(&buffer).expect("Could not write");

                    println!("NEWFILE");
                }
                Request::NEWDIR => {
                    //Um novo diretório será puxado
                    let raw_temp_dir_name = scp.request_get_filename().unwrap();
                    let temp_dir_name = String::from_utf8_lossy(raw_temp_dir_name).to_owned();
                    current_path = PathBuf::from(&current_path.join(&*temp_dir_name));
                    println!("current_path {:?}", current_path);
                    std::fs::create_dir_all(&current_path).unwrap();
                    scp.accept_request().unwrap();
                    println!("NEWDIR");
                }
                Request::EOF => {
                    println!("EOF");
                    break;
                }
                Request::ENDDIR => {
                    current_path = current_path.as_path().parent().unwrap().to_path_buf();
                    println!("ENDDIR {}", current_path.display());
                    //break;
                }
                Request::WARNING => {
                    println!("WARNING");
                    //scp.deny_request().unwrap();
                }
            }
        }
    }

    pub fn run_cmd(&mut self, command: &str) -> Result<String, i32> {
        let mut channel = self.session.channel_new().unwrap();
        channel.open_session().unwrap();
        channel.request_exec(command.as_bytes()).unwrap();
        channel.send_eof().unwrap();
        let mut buf = Vec::new();
        channel.stdout().read_to_end(&mut buf).unwrap();
        let out = String::from_utf8_lossy(&buf).into_owned();
        Ok(out)
    }

    pub fn list_dir(&mut self, path: &Path) -> Vec<FsEntry> {
        let cmd = format!("unset LANG; find \"$(cd '{}'; pwd)\" -printf '%M|||%u|||%g|||%s|||%Ts|||%p|||%f|||%l\n'", path.display());
        let output = self.run_cmd(&cmd).unwrap();
        // Split output by \0
        let lines: Vec<&str> = output.as_str().split("\n").collect();
        let mut entries: Vec<FsEntry> = Vec::with_capacity(lines.len());
        //println!("lines {:?}", lines);
        let mut index = 0;
        for line in lines.iter() {
            // First line must always be ignored
            if index > 0 {
                let columns: Vec<&str> = line.split("|||").collect();
                //println!("columns {:?}", columns);
                let path = columns.get(5);
                if let Some(&p) = path {
                    if let Some(&pex) = columns.get(0) {
                        let mut file_type = pex.get(0..1).unwrap();
                        let mut is_link = false;
                        if file_type == "l" {
                            is_link = true;
                            if let Some(_link_info) = columns.get(7) {
                                let cmd = format!("unset LANG; find -L \"$(readlink -f {})\" -printf '%y'", p);
                                let link_file_type = self.run_cmd(&cmd).unwrap();
                                if link_file_type.contains("find:") {
                                    println!("link out: {} | cmd: {}", link_file_type, cmd);
                                } else {
                                    file_type = "-";
                                }
                            }
                        }
                        let entry = FsEntry {
                            path: p.parse().unwrap(),
                            file_type: match file_type {
                                "d" => FsEntryType::Directory,
                                _ => FsEntryType::File
                            },
                            is_link,
                        };
                        entries.push(entry);
                    }
                }
            }
            index += 1;
        }
        return entries;
    }
}

https://github.com/insinfo/test_ssh

That's interesting actually. I'm going to make some further tests with libssh2 in rust, to see if there are actually some issues with it when dealing with a huge amount of transfers. The ssh2 crate should just be a wrapper around the C libssh2, so I may also make some tests with the C library.

By the way, remotefs has just been released, so you can use it as a dependency, instead of termscp now.

congratulations for the release of the lib "remotefs" , excellent work, researching a little I think the problem with the slow speed of libssh2 is because it does not implement the recursive directory copy of the SCP protocol, libssh implemented the recursive copy, I think it is that's the problem.

Perhaps an interesting solution would be to port an SCP implementation like C# (SSH.NET) to rust on top of crate thrussh or even on top of libssh2

https://github.com/sshnet/SSH.NET/blob/develop/src/Renci.SshNet/ScpClient.cs https://github.com/phpseclib/phpseclib-php5/blob/master/phpseclib/Net/SCP.php https://github.com/net-ssh/net-scp/blob/master/lib/net/scp.rb

implementation by directly calling libssh2 functions without using "ssh2_rs"

#![allow(dead_code)]
#![allow(unused_imports)]
#![cfg_attr(debug_assertions, allow(dead_code, unused_imports, deprecated))]

use std::ffi::CString;
#[cfg(unix)]
use std::os::unix::io::{AsRawFd, RawFd};
#[cfg(windows)]
use std::os::windows::io::{AsRawSocket, RawSocket};

use std::path::{Path, PathBuf};
use std::{io, slice};
use std::io::Write;
use std::str;
use std::net::TcpStream;
use std::ptr::null_mut;

extern crate libssh2_sys;

use libc::{self, c_char, c_int, c_long, c_uint, c_void, size_t};
use libssh2_sys::*;
use test_ssh::{FsEntry, FsEntryType};

use test_ssh::utils::{make_error_message, path2bytes, print_error};

const BUFFER_SIZE: usize = (1024 * 128) * 2;// 100 * 1024;  128 *

pub fn run() {
    let mut ssh = Libssh2::new().unwrap();
    let tcp = TcpStream::connect("192.168.133.13:22").unwrap();
    ssh.connect(tcp, "isaque.neves", "Ins257257");

    let dir_to_download = Path::new("/var/www/html/portalPmro");
    let dest_dir_path = Path::new(r"C:\MyRustProjects\test_ssh\download");
    std::fs::create_dir_all(&dest_dir_path).unwrap();

    let start = std::time::Instant::now();
    let items_to_download = ssh.list_dir(&dir_to_download);
    println!("Time elapsed in list dir: {:?}", start.elapsed());

    let start = std::time::Instant::now();
    for item in items_to_download.iter() {
        //remove a parte inicial do caminho
        let mut dst_path = PathBuf::from(&item.path.strip_prefix(&dir_to_download).unwrap());
        dst_path = dest_dir_path.join(dst_path);
        ssh.download_item(&item, &dst_path);
        println!("item {}", item.path.display());
    }
    println!("Time elapsed in file transfer: {:?}", start.elapsed());
    println!("download of {:?} complete!", dir_to_download);

    ssh.disconnect();
}

pub struct Libssh2 {
    pub session: *mut LIBSSH2_SESSION,
    #[cfg(unix)]
    tcp: Option<Box<dyn AsRawFd>>,
    #[cfg(windows)]
    tcp: Option<Box<dyn AsRawSocket>>,
    channel: *mut LIBSSH2_CHANNEL,
}

impl Libssh2 {
    pub fn new() -> Result<Libssh2, ()> {
        unsafe {
            let session = libssh2_session_init_ex(None, None, None, 0 as *mut _);
            if session.is_null() {
                println!("Error on init libssh2_session");
            }
            Ok(Libssh2 {
                session,
                tcp: None,
                channel: std::ptr::null_mut(),
            })
        }
    }

    pub fn connect<S: 'static + AsRawSocket>(&mut self, stream: S, user: &str, pass: &str) {
        unsafe {
            self.tcp = Some(Box::new(stream));
            let mut rc = libssh2_session_handshake(self.session, self.tcp.as_ref().unwrap().as_raw_socket());
            if rc != 0 {
                println!("Failure establishing SSH session: {}", rc);
                print_error(self.session);
            }

            rc = libssh2_userauth_password_ex(self.session, user.as_ptr() as *const _,
                                              user.len() as c_uint,
                                              pass.as_ptr() as *const _,
                                              pass.len() as c_uint,
                                              None, );
            if rc != 0 {
                println!("Authentication by password failed: {}", rc);
                print_error(self.session);
            }
        }
    }

    pub fn download_item(&mut self, entry: &FsEntry, dst_path: &Path) {
        if entry.file_type == FsEntryType::File {
            unsafe {
                let path = CString::new(path2bytes(&entry.path).unwrap()).unwrap();
                let mut fileinfo: libssh2_struct_stat = std::mem::uninitialized();

                let channel = libssh2_scp_recv2(self.session, path.as_ptr(), &mut fileinfo);

                if channel.is_null() {
                    println!("Failed to recv file: ");
                    print_error(self.session);
                    return;
                }
                let mut dest_file = std::fs::File::create(&dst_path).unwrap();

                let mut got = 0;
                let mut buffer: [u8; BUFFER_SIZE] = std::mem::uninitialized();
                let mut amount = BUFFER_SIZE as i64;

                while got < fileinfo.st_size {
                    if (fileinfo.st_size - got) < amount {
                        amount = (fileinfo.st_size - got) as i64;
                    }

                    let rc = libssh2_channel_read_ex(channel, 0, buffer.as_mut_ptr() as *mut _, amount as size_t) as i64;

                    if rc > 0 {
                        dest_file.write(&buffer[..rc as usize]).unwrap();
                    } else if rc < 0 {
                        println!("libssh2_channel_read() failed: {}", rc);
                        print_error(self.session);
                        break;
                    }
                    got += rc;
                }
                libssh2_channel_free(channel);
            }
        } else if entry.file_type == FsEntryType::Directory {
            std::fs::create_dir_all(&dst_path)
                .expect(format!("Failed to create directory: {:?}", &dst_path).as_str());
        }
    }

    pub fn run_cmd(&mut self, command: &str) -> Result<String, i32> {
        unsafe {
            let c_str = CString::new("session").unwrap();
            let channel_type = c_str.as_ptr() as *const c_char;
            let channel_type_len = "session".len() as c_uint;

            let channel = libssh2_channel_open_ex(
                self.session, channel_type,
                channel_type_len, LIBSSH2_CHANNEL_WINDOW_DEFAULT, LIBSSH2_CHANNEL_PACKET_DEFAULT,
                std::ptr::null_mut(), 0);
            if (channel as usize) == 0 {
                println!("erro on libssh2_channel_open_ex");
                print_error(self.session);
                return Err(-1);
            }

            let c_str = CString::new("exec").unwrap();
            let req_type = c_str.as_ptr() as *const c_char;
            let req_type_len = "exec".len() as c_uint;

            let c_str = CString::new(command).unwrap();
            let cmd = c_str.as_ptr() as *const c_char;
            let cmd_len = command.len() as c_uint;

            let rc = libssh2_channel_process_startup(channel, req_type, req_type_len, cmd, cmd_len);
            if rc != 0 {
                println!("Error on libssh2_channel_process_startup");
                print_error(self.session);
                return Err(-1);
            }

            let mut output = Vec::new();
            #[allow(deprecated)]
                let mut buffer: [u8; BUFFER_SIZE] = std::mem::uninitialized();//*mut c_char
            let buffer_len = buffer.len() as size_t;
            while
            {
                let rc = libssh2_channel_read_ex(channel, 0,
                                                 buffer.as_mut_ptr() as *mut _,
                                                 buffer_len) as usize;
                if rc > 0
                {
                    //println!("We read:");
                    output.append(&mut Vec::from(&buffer[0..rc]));
                } /*else {
                    if rc != LIBSSH2_ERROR_EAGAIN as usize {
                        println!("libssh2_channel_read returned  {}", rc);
                    }
                }*/
                rc > 0
            } {}
            let result = String::from_utf8_lossy(&output[..]).into_owned();
            libssh2_channel_free(channel);
            Ok(result)
        }
    }

    pub fn list_dir(&mut self, path: &Path) -> Vec<FsEntry> {
        let cmd = format!("unset LANG; find \"$(cd '{}'; pwd)\" -printf '%M|||%u|||%g|||%s|||%Ts|||%p|||%f|||%l\n'", path.display());
        let output = self.run_cmd(&cmd).unwrap();
        // Split output by \0
        let lines: Vec<&str> = output.as_str().split("\n").collect();
        let mut entries: Vec<FsEntry> = Vec::with_capacity(lines.len());
        //println!("lines {:?}", lines);
        let mut index = 0;
        for line in lines.iter() {
            // First line must always be ignored
            if index > 0 {
                let columns: Vec<&str> = line.split("|||").collect();
                //println!("columns {:?}", columns);
                let path = columns.get(5);
                if let Some(&p) = path {
                    if let Some(&pex) = columns.get(0) {
                        let mut file_type = pex.get(0..1).unwrap();
                        let mut is_link = false;
                        if file_type == "l" {
                            is_link = true;
                            if let Some(_link_info) = columns.get(7) {
                                let cmd = format!("unset LANG; find -L \"$(readlink -f {})\" -printf '%y'", p);
                                let link_file_type = self.run_cmd(&cmd).unwrap();
                                if link_file_type.contains("find:") {
                                    println!("link out: {} | cmd: {}", link_file_type, cmd);
                                } else {
                                    file_type = "-";
                                }
                            }
                        }
                        let entry = FsEntry {
                            path: p.parse().unwrap(),
                            file_type: match file_type {
                                "d" => FsEntryType::Directory,
                                _ => FsEntryType::File
                            },
                            is_link,
                        };
                        entries.push(entry);
                    }
                }
            }
            index += 1;
        }
        return entries;
    }

    pub fn disconnect(&mut self) {
        unsafe {
            let msg = CString::new("Normal Shutdown").unwrap();
            let lang = CString::new("").unwrap();
            libssh2_session_disconnect_ex(self.session, SSH_DISCONNECT_BY_APPLICATION, msg.as_ptr(), lang.as_ptr());
            libssh2_session_free(self.session);
        }
    }
}

https://github.com/insinfo/test_ssh

As stated in the ssh2-rs repository, it is an issue related to libssh2. If it will ever be fixed, it will be fixed in remotefs-ssh too, which is currently not part of the termscp code anymore. So this issue can be closed.

veeso / termscp