my scp'ed files are getting truncated at 204800 bytes after upgrading openssh

[maertendMSFT]

Discussed in https://github.com/PowerShell/Win32-OpenSSH/discussions/2096

^{Originally posted by **tomwlane** July 12, 2023} my scp'ed files are getting truncated at 204800 bytes after upgrading openssh to v9.2.2.0p1-Beta I'm using an openbsd client on windows (git/bash for windows), it works on other systems, all other combinations actually, but this one system to my one upgraded system just truncates the thing last minute like. heres the log from the client scp: debug3: Received data 256977920 -> 257009159 scp: debug3: Short data block, re-requesting 257009160 -> 257080319 ( 1) scp: debug3: Finish at 257080320 ( 1) scp: debug3: Received reply T:101 I:2522 R:1 nfhl_s_fld_haz_ar.shp 100% 245MB 19.0MB/s 00:12 **scp: debug1: truncating at 204800** scp: debug3: Sent message SSH2_FXP_CLOSE I:2523 scp: debug3: SSH2_FXP_STATUS 0 debug2: channel 0: read failed rfd 5 maxlen 32768: Broken pipe debug2: channel 0: read failed $ ls -l nfhl_s_fld_haz_ar.shp -rw-r--r-- 1 AzureAD+TomLane 4096 204800 Jul 12 08:41 nfhl_s_fld_haz_ar.shp and on the server $ sshd -v OpenSSH_9.3p1, OpenSSL 1.1.1u 30 May 2023

[tgauth]

@tomwlane - I am trying to setup a repro, have a few follow-up questions:

• can you provide the actual command that leads to this?
• what is the approximate file size that's being transferred?
• in the logs, do you see "Buffer value of X is too large for Win32-OpenSSH. Setting buffer to 204800"? It would be at the very beginning of the scp client log. I remember running into what I think is a related issue when the 9.2 changes got pulled from upstream because a -X parameter regarding buffer size was added to scp - 9.2 release notes
• does running the scp command with -O change the behavior at all? https://man.openbsd.org/scp.1

[tomwlane]

nothing unusual about the command scp @.**@.:c:/workspace/apache24/logs/error.log> . $ ls -l error.log -rw-r--r-- 1 AzureAD+TomLane 4096 204800 Jul 18 08:05 error.log I only got 204800 bytes

this file on the server is 20622247 bytes -rw-r--r-- 1 Administrator 197121 20622247 Jul 18 02:14 c:/workspace/apache24/logs/error.log

I don't see that message, the entire debug log is attached,

I get disconnected using that -O option, I get no file

I think its going to be hard to reproduce, I think its just happening from my own pc, other machines with what should be the same scp work, it may even be that I performed the upgrade incorrectly somewhere

unless you can tell anything from the logs, its may be hard to reproduce, Id send the debug log from the server, but it seems to b e hidden in even viewer with the new version

Tom Lane

scp: debug3: Received data 20331520 -> 20433919 scp: debug3: Finish at 20638720 ( 2) debug2: channel 0: window 1933312 sent adjust 163840 debug2: channel 0: window 1984166 sent adjust 65536 scp: debug3: Received reply T:103 I:205 R:1 scp: debug3: Received data 20433920 -> 20536319 scp: debug3: Finish at 20638720 ( 1) scp: debug3: Received reply T:103 I:206 R:1 scp: debug3: Received data 20536320 -> 20622246 scp: debug3: Short data block, re-requesting 20622247 -> 20638719 ( 1) scp: debug3: Finish at 20638720 ( 1) scp: debug3: Received reply T:101 I:207 R:1 scp: debug1: truncating at 204800 scp: debug3: Sent message SSH2_FXP_CLOSE I:208 scp: debug3: SSH2_FXP_STATUS 0 debug2: channel 0: read failed rfd 5 maxlen 32768: Broken pipe debug2: channel 0: read failed debug2: chan_shutdown_read: channel 0: (i0 o0 sock -1 wfd 5 efd 7 [write]) debug2: channel 0: input open -> drain debug2: channel 0: ibuf empty debug2: channel 0: send eof debug3: send packet: type 96 debug2: channel 0: input drain -> closed debug3: receive packet: type 96 debug2: channel 0: rcvd eof debug2: channel 0: output open -> drain debug2: channel 0: obuf empty debug2: chan_shutdown_write: channel 0: (i3 o1 sock -1 wfd 6 efd 7 [write]) debug2: channel 0: output drain -> closed debug3: receive packet: type 98 debug1: client_input_channel_req: channel 0 rtype exit-status reply 0 debug3: receive packet: type 97 debug2: channel 0: rcvd close debug3: channel 0: will not send data after close debug2: channel 0: almost dead debug2: channel 0: gc: notify user debug2: channel 0: gc: user detached debug2: channel 0: send close debug3: send packet: type 97 debug2: channel 0: is dead debug2: channel 0: garbage collecting debug1: channel 0: free: client-session, nchannels 1 debug3: channel 0: status: The following connections are open:

0 client-session (t4 [session] r0 i3/0 o3/0 e[write]/0 fd -1/-1/7 sock -1 cc -1 io 0x00/0x00)

debug3: send packet: type 1 Transferred: sent 15360, received 20641016 bytes, in 6.5 seconds Bytes per second: sent 2371.0, received 3186225.1 debug1: Exit status 0

[tomwlane]

this may actually an issue with my scp client git-for-windows 2.41.0.windows.3, I verified it works with the scp.exe from the windows.1 version, but not the latest .3 version.

from https://gitforwindows.org/

I'll open an issue there https://github.com/git-for-windows/git/issue

[huangqqss]

I have the same problem. The server is OpenSSH_for_Windows_9.2p1 running on Windows 7, LibreSSL 3.6.1, and the client is openssh-client Version: 1:9.3p2-1 running on Debian 12 (bookworm). I copy a file from Windows to Debian through WiFi at home. Although the copying takes quite a long time and it seems a large file got transfer, the saved file is only 204800 bytes.

I have also tried some other configuration, such as: 1) (different protocol) sftp instead of scp, 2) (reversed transfer) copy a file from Debian to Windows (by scp running on Debian), 3) (different source) scp files from other SSH servers, which are running some versions of Linux, 4) (different destination) running scp in a Debian within a VMWare box hosted by the same Windows. All of these are OK.

I hope these observation can provide some help.

[niknah]

I've tried this on the different binaries on this repository. v8.1.0.0p1-Beta works. v8.6.0.0p1-Beta truncates the file at 204k. 8.1.log 8.6.log

I am starting scp on Arch Linux, OpenSSH_9.3p2, OpenSSL 3.1.2 1 Aug 2023.

Copying a file from Linux(scp) to Windows(sshd.exe) works. Copying a file from Windows(sshd.exe) to Linux(scp) gives me only 204k of the file.

Copying using scp.exe back to linux works. This is the workaround. Login to windows with ssh -R 2200:127.0.0.1:22 .... then scp.exe -P 2200 <file> 127.0.0.1:<destination>

[reinux]

Having the same issue receiving to NixOS. Baffled.

[tgauth]

Thanks for the repro information - seems like the issue is within Windows SSHD. I'm curious if using -X with a value less that 204800 in the SCP command would mitigate the file truncation.

[niknah]

-X buffer=204800 worked.
-X buffer=204801 has the problem.

But the problem has gone away now, after I upgraded from openssh 9.3 to 9.4 on the Linux machine.

[vickkhera]

FWIW, I'm observing the same truncation (at seemingly arbitrary size even fetching the same file multiple times) with OpenSSH 9.3p2 on macOS 14. The hint above to upgrade to 9.4 worked around it for me on the client side. The remote server I'm using is an AWS managed SFTP server.

[reinux]

-X buffer=204800 worked. -X buffer=204801 has the problem.

But the problem has gone away now, after I upgraded from openssh 9.3 to 9.4 on the Linux machine.

Just gotta remember to do this until my distro updates openssh. Thanks!

[markkuleinio]

Windows updates in October 2024 for Windows Server 2022 caused this exact problem when using scp on Debian 12 (Bookworm, the latest stable version with updates installed):

debug1: Local version string SSH-2.0-OpenSSH_9.2p1 Debian-2+deb12u3 debug1: Remote protocol version 2.0, remote software version OpenSSH_for_Windows_9.5 debug1: compat_banner: match: OpenSSH_for_Windows_9.5 pat OpenSSH* compat 0x04000000 ... scp: debug1: truncating at 204800

I added -X buffer=204800 to all client-side commands, as suggested above (thanks!), to get file transfers to work again.

I don't have a record about the previous OpenSSH version on the Windows server.