git-for-windows / git

A fork of Git containing Windows-specific patches.
http://gitforwindows.org/
Other
8.37k stars 2.54k forks source link

scp'ed files truncated at 204800 bytes #4518

Open tomwlane opened 1 year ago

tomwlane commented 1 year ago

Setup

$ git --version --build-options

git version 2.41.0.windows.3
cpu: x86_64
built from commit: 4a1821dfb0c2160a4fcc85d2845f095c54339578
sizeof-long: 4
sizeof-size_t: 8
shell-path: /bin/sh
feature: fsmonitor--daemon

it works if i copy scp.exe from my old windows.1 version.
$ cmd.exe /c ver

Microsoft Windows [Version 10.0.19045.3086]
64 bit

default installation

$ cat /etc/install-options.txt

Editor Option: VIM
Custom Editor Path:
Default Branch Option:
Path Option: Cmd
SSH Option: OpenSSH
Tortoise Option: false
CURL Option: OpenSSL
CRLF Option: CRLFAlways
Bash Terminal Option: MinTTY
Git Pull Behavior Option: Merge
Use Credential Manager: Enabled
Performance Tweaks FSCache: Enabled
Enable Symlinks: Disabled
Enable Pseudo Console Support: Disabled
Enable FSMonitor: Disabled

nothing

Details

bash


normal simple scp command

scp  ***@***.******@***.***:c:/workspace/apache24/logs/error.log .

expected a complete file

got truncated file

$ ls -l error.log -rw-r--r-- 1 AzureAD+* 4096 204800** Jul 18 08:05 error.log I only got 204800 bytes

this file on the server is 20622247 bytes -rw-r--r-- 1 Administrator * 20622247** Jul 18 02:14 c:/workspace/apache24/logs/error.log

it works if i copy scp.exe from my old windows.1 version.

heres some of the debug log showing it truncates it at the last minute

scp: debug3: Received data 256977920 -> 257009159 scp: debug3: Short data block, re-requesting 257009160 -> 257080319 ( 1) scp: debug3: Finish at 257080320 ( 1) scp: debug3: Received reply T:101 I:2522 R:1 nfhl_s_fld_haz_ar.shp 100% 245MB 19.0MB/s 00:12 scp: debug1: truncating at 204800 scp: debug3: Sent message SSH2_FXP_CLOSE I:2523 scp: debug3: SSH2_FXP_STATUS 0 debug2: channel 0: read failed rfd 5 maxlen 32768: Broken pipe debug2: channel 0: read failed

the servers running openssh for windows, I think it works on older versions

$ sshd -v OpenSSH_9.3p1, OpenSSL 1.1.1u 30 May 2023

from https://github.com/PowerShell/Win32-OpenSSH/releases/tag/v9.2.2.0p1-Beta

ldennington commented 1 year ago

While the team is investigating your issue, a potential workaround you could try is re-installing Git for Windows and indicating that you would like to use the Windows version of OpenSSH like so:

image

Note, however, that this may have implications on existing setups - SSH setups that rely on the Git unix-like sockets won't work with the Windows exe.

dscho commented 1 year ago

Between v2.41.0.windows.1 and .3, msys2-runtime was upgraded, as was OpenSSL, and openssh was also rebuilt to adapt for the OpenSSL upgrade.

@tomwlane to pinpoint the culprit better, could you download PortableGit variants of both versions from https://github.com/git-for-windows/git/releases/, extract the .1 version and replace usr\bin\msys-2.0.dll with the one from the .3 version and retry the experiment? If the problematic change was in the MSYS2 runtime, that would "introduce the problem into .1" and we would then be able to pursue that angle.

tomwlane commented 1 year ago

I have verified that using the scp.exe from a git version 2.39.2.windows.1 installation solves the problem, thats actually been my workaround, Does that answer your question ?

also, the openssh group is looking too, since that was upgraded too, and another user reports the same problem using openssh-client Version: 1:9.3p2-1 running on Debian 12.

https://github.com/PowerShell/Win32-OpenSSH/issues/2098

dscho commented 1 year ago

I have verified that using the scp.exe from a git version 2.39.2.windows.1 installation solves the problem, thats actually been my workaround, Does that answer your question ?

No. You basically downgraded Git for Windows, and unless we cannot figure out how to fix the issue in the current Git for Windows you won't be able to upgrade once the next security fix release comes out, which is a bad idea.

The only way forward is to pinpoint the bug in a much finer detail, we're still way too coarse over here, and that won't help us fix the bug.

tomwlane commented 1 year ago

@dscho I verified I have the same truncation problem on an old 2.41.0.windows.1 installation, using the 2.41.0.windows.3 version of usr\bin\msys-2.0.dll, I hope thats the test you wanted, if I'm reading it right.

dscho commented 1 year ago

I verified I have the same truncation problem on an old 2.41.0.windows.1 installation, using the 2.41.0.windows.3 version of usr\bin\msys-2.0.dll

Yes, that confirms that the MSYS2 runtime contains the bug.

Unfortunately the upgrade contains many patches, as it upgraded msys2-runtime from 3.4.6-4 to 3.4.7-2.

The most promising way that I can think of to diagnose this better is unfortunately quite tedious: use git bisect in a clone of msys2-runtime, applying the MSYS2/Git for Windows changes on top of the Cygwin commits and then re-building the MSYS2 runtime. The last time I had to do this, it took me three days to pinpoint the problem.

A less tedious approach might be this: Ifff vanilla Cygwin also exposes the problem, we can ask the good Cygwin people for help. They might know off the top of their head what changes/code might be responsible.

dscho commented 1 year ago

Potentially related: #4550. The release notes say this:

  • scp(1), sftp(1): adjust ftruncate() logic to handle servers that reorder requests. Previously, if the server reordered requests then the resultant file would be erroneously truncated.