The ssh input is very slow

[lonnywong]

I'm making trzsz, which similar to lrzsz.

The trzsz wraps ssh in cmd. The download speed is fast as expected, but the upload speed is very slow.

Troubleshooting steps

1. Download trzsz.exe from trzsz_windows_amd64.zip, or make it yourself.

   git clone https://github.com/trzsz/trzsz-go.git
   cd trzsz-go
   GOOS=windows make

2. Add trzsz.exe before ssh, login to a server. e.g.:

   .\trzsz.exe ssh yourname@127.0.0.1

3. Install trzsz on the server. Require the test version 0.2.111:

   sudo python3 -m pip install --upgrade -i https://test.pypi.org/simple/ trzsz==0.2.111

4. command trz to upload a file.

5. The trzsz.exe will read the file and send the content to the ssh stdin. The trz will read the content from stdin and save it to the server.

"OpenSSH for Windows" version

V8.9.1.0p1-Beta

Actual output

The upload speed is about 24 KB/s.

Is the ssh process having a small buffer, and the input data from stdin is not processed in time?

Expected output

The upload speed should be above 2 MB/s, same as on ubuntu.

[mgkuhn]

Are you handing over to write individual bytes (or similarly short chunks) or do you only hand over large (i.e., multiple kilobytes long) blocks via buffered I/O? The former is usually the most common cause for inefficient I/O.

Can you link to the line in your source code where the data actually leaves your application towards ssh?

Or provide a minimal working example (MWE) that shows the problem?

[lonnywong]

@mgkuhn Thanks for your help.

I'm using ConPTY. The source code: https://github.com/UserExistsError/conpty/blob/master/conpty.go#L261

I'll make a minimal example in a few hours.

[lonnywong]

@mgkuhn

1. A C program run on a Linux server. Compile it with gcc. I name it test_server.

   
   #include <stdio.h>
   #include <stdlib.h>
   #include <sys/time.h>
   #include <termios.h>
   #include <unistd.h>

int ReadTestData() { unsigned int size = 0; if (scanf("%u", &size) != 1) { puts("invalid size"); return -1; }

char buf[100]; unsigned int last_count = 0, read_count = 0; struct timeval begin_time, last_time, current_time; gettimeofday(&begin_time, NULL); last_time = begin_time;

while (read_count < size) { ssize_t n = read(0, buf, sizeof(buf)); if (n <= 0) { puts("read error"); return -2; } read_count += n;

gettimeofday(&current_time, NULL);
if (current_time.tv_sec - last_time.tv_sec > 1) {
  long t = (current_time.tv_sec - last_time.tv_sec) * 1000 +
           (current_time.tv_usec - last_time.tv_usec) / 1000;
  printf("read %u bytes in %ldms, speed: %.2fKB/s\n",
         read_count - last_count, t,
         (read_count - last_count) / 1024.0 * 1000 / t);
  last_time = current_time;
  last_count = read_count;
}

}

gettimeofday(&current_time, NULL); long t = (current_time.tv_sec - begin_time.tv_sec) 1000 + (current_time.tv_usec - begin_time.tv_usec) / 1000; printf("read %u bytes in %ldms, speed: %.2fKB/s\n", read_count, t, read_count / 1024.0 1000 / t);

return 0; }

int main() { struct termios old, raw; tcgetattr(STDIN_FILENO, &old); raw = old; raw.c_lflag &= ~(ECHO); tcsetattr(STDIN_FILENO, TCSAFLUSH, &raw);

int ret = ReadTestData();

tcsetattr(STDIN_FILENO, TCSAFLUSH, &old);

return ret; }

// gcc -o test_server test_server.c

2. A `go` program run on Windows, which call `ssh -t test_server` to interact with the Linux server program. I name it `test_client.go`.
```go
package main

import (
    "context"
    "io"
    "log"
    "os"
    "strconv"

    "github.com/UserExistsError/conpty"
)

func writeTestData(cpty *conpty.ConPty) {
    const size int = 100
    const count int = 10000

    var data = make([]byte, size)
    for i := 0; i < size - 2; i++ {
        data[i] = 'A'
    }
    data[size - 2] = '\r'
    data[size - 1] = '\n'

    cpty.Write([]byte(strconv.Itoa(size*count) + "\r\n"))

    for i := 0; i < count; i++ {
        cpty.Write(data)
    }
}

func main() {
    // 1. `ssh dev` to login remote server, change it to your server.
    // 2. make `test_server` on your server, change the absolute path.
    commandLine := "ssh dev -t /tmp/test_server"

    cpty, err := conpty.Start(commandLine)
    if err != nil {
        log.Fatalf("Failed to spawn a pty:  %v", err)
    }
    defer cpty.Close()

    go func() {
        go io.Copy(os.Stdout, cpty)
        writeTestData(cpty)
    }()

    exitCode, err := cpty.Wait(context.Background())
    if err != nil {
        log.Fatalf("Error: %v", err)
    }
    log.Printf("ExitCode: %d", exitCode)
}

3. Run test_client on Windows.

   mkdir test_dir
   cd test_dir
   go mod init example.com/m/v2
   go get github.com/UserExistsError/conpty
   # save the source code to test_client.go
   # change one line `ssh dev -t /tmp/test_server` to your test environment.
   go run test_client.go

   You should get some output like:

   read 1800 bytes in 1140ms, speed: 1.54KB/s
   read 3000 bytes in 1991ms, speed: 1.47KB/s
   read 17100 bytes in 2001ms, speed: 8.35KB/s
   read 38400 bytes in 2000ms, speed: 18.75KB/s
   read 39600 bytes in 1999ms, speed: 19.35KB/s
   read 39200 bytes in 2001ms, speed: 19.13KB/s
   read 39600 bytes in 2001ms, speed: 19.33KB/s
   read 40600 bytes in 1999ms, speed: 19.83KB/s
   read 38100 bytes in 1998ms, speed: 18.62KB/s
   read 40600 bytes in 2000ms, speed: 19.82KB/s
   read 40600 bytes in 2002ms, speed: 19.80KB/s
   read 39600 bytes in 1997ms, speed: 19.36KB/s
   read 40500 bytes in 2001ms, speed: 19.77KB/s
   read 39000 bytes in 1999ms, speed: 19.05KB/s
   read 45500 bytes in 2000ms, speed: 22.22KB/s
   read 48100 bytes in 2002ms, speed: 23.46KB/s
   read 47800 bytes in 1999ms, speed: 23.35KB/s
   read 48500 bytes in 2000ms, speed: 23.68KB/s
   read 47300 bytes in 2000ms, speed: 23.10KB/s
   read 48200 bytes in 2000ms, speed: 23.54KB/s
   read 48000 bytes in 2001ms, speed: 23.43KB/s
   read 47800 bytes in 2000ms, speed: 23.34KB/s
   read 47100 bytes in 2000ms, speed: 23.00KB/s
   read 48200 bytes in 1999ms, speed: 23.55KB/s
   read 46800 bytes in 2000ms, speed: 22.85KB/s
   read 1000000 bytes in 49911ms, speed: 19.57KB/s

[lonnywong]

The go source code is at https://github.com/UserExistsError/conpty/blob/master/conpty.go

There is a C++ example for ConPTY. I can change the go program to C++ if necessary. https://github.com/microsoft/terminal/tree/main/samples/ConPTY/EchoCon

[lonnywong]

@mgkuhn It's a little faster without ssh, https://github.com/microsoft/terminal/issues/13594#issuecomment-1249960748

I'm not sure what is causing the issue. May be some issues with ssh, ConPTY and Windows stdin.

[mgkuhn]

If you change between

const size int = 100
const count int = 10000

and

const size int = 10000
const count int = 100

does that make a noticeable difference in runtime?

[lonnywong]

does that make a noticeable difference in runtime?

I switch the value of size and count, and change char buf[100]; to char buf[10000];.

It loses some data, and doesn't exit automatically.

The test result is slower.

read 4097 bytes in 2421ms, speed: 1.65KB/s
read 20485 bytes in 2001ms, speed: 10.00KB/s
read 20485 bytes in 2028ms, speed: 9.86KB/s
read 20485 bytes in 2184ms, speed: 9.16KB/s
read 20485 bytes in 2029ms, speed: 9.86KB/s
read 16388 bytes in 1726ms, speed: 9.27KB/s
read 20485 bytes in 2082ms, speed: 9.61KB/s
read 20485 bytes in 1982ms, speed: 10.09KB/s
read 20485 bytes in 2042ms, speed: 9.80KB/s
read 20485 bytes in 2067ms, speed: 9.68KB/s
read 20485 bytes in 2030ms, speed: 9.85KB/s
read 20485 bytes in 2087ms, speed: 9.59KB/s
read 20485 bytes in 2062ms, speed: 9.70KB/s
read 20485 bytes in 2064ms, speed: 9.69KB/s
read 16388 bytes in 1624ms, speed: 9.85KB/s
read 20485 bytes in 2036ms, speed: 9.83KB/s
read 20485 bytes in 2078ms, speed: 9.63KB/s
read 20485 bytes in 2044ms, speed: 9.79KB/s
read 20485 bytes in 2091ms, speed: 9.57KB/s
read 20485 bytes in 1986ms, speed: 10.07KB/s
read 20485 bytes in 2016ms, speed: 9.92KB/s