Windows Server 2019 OpenSSH SFTP Server Won't Authenticate Users Anymore (Connection Reset)

[ShieldOfSalvation]

I am running Microsoft Windows Server 2019 Datacenter Version 10.0.17763 Build 17763 on Azure and I had SFTP working just fine until EITHER a recent update and reboot on Windows OR an SFTP username (the "vendor1" user) password change on ActiveDirectory clobbered this working install of OpenSSH.

Now when attempting to SFTP from a client machine, all I get is,

Connection reset by xxx.xxx.xxx.xxx port 22 Connection closed

What could be wrong? Has anyone else experienced this and solved it?

Here's my sshd_config file, which was working:

# This is the sshd server system-wide configuration file.  See
# sshd_config(5) for more information.

# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented.  Uncommented options override the
# default value.

#Port 22
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::

#HostKey __PROGRAMDATA__/ssh/ssh_host_rsa_key
#HostKey __PROGRAMDATA__/ssh/ssh_host_dsa_key
#HostKey __PROGRAMDATA__/ssh/ssh_host_ecdsa_key
#HostKey __PROGRAMDATA__/ssh/ssh_host_ed25519_key

# Ciphers and keying
#RekeyLimit default none

# Logging
#SyslogFacility AUTH
#LogLevel INFO

# Authentication:

#LoginGraceTime 2m
#PermitRootLogin prohibit-password
#StrictModes yes
#MaxAuthTries 6
#MaxSessions 10

#PubkeyAuthentication yes

# The default is to check both .ssh/authorized_keys and .ssh/authorized_keys2
# but this is overridden so installations will only check .ssh/authorized_keys
AuthorizedKeysFile  .ssh/authorized_keys

#AuthorizedPrincipalsFile none

# For this to work you will also need host keys in %programData%/ssh/ssh_known_hosts
#HostbasedAuthentication no
# Change to yes if you don't trust ~/.ssh/known_hosts for
# HostbasedAuthentication
#IgnoreUserKnownHosts no
# Don't read the user's ~/.rhosts and ~/.shosts files
#IgnoreRhosts yes

# To disable tunneled clear text passwords, change to no here!
PasswordAuthentication yes
#PermitEmptyPasswords no

#AllowAgentForwarding yes
#AllowTcpForwarding yes
#GatewayPorts no
#PermitTTY yes
#PrintMotd yes
#PrintLastLog yes
#TCPKeepAlive yes
#UseLogin no
#PermitUserEnvironment no
#ClientAliveInterval 0
#ClientAliveCountMax 3
#UseDNS no
#PidFile /var/run/sshd.pid
#MaxStartups 10:30:100
#PermitTunnel no
#ChrootDirectory none
#VersionAddendum none

# no default banner path
Banner F:\SFTP-Welcome.txt
#Banner /SFTP-Welcome.txt

# override default of no subsystems
Subsystem   sftp    sftp-server.exe

# Example of overriding settings on a per-user basis
#Match User anoncvs
#   AllowTcpForwarding no
#   PermitTTY no
#   ForceCommand cvs server

#Match Group administrators
#       AuthorizedKeysFile __PROGRAMDATA__/ssh/administrators_authorized_keys

#DenyGroups company\vendors company\auditors
#AllowGroups company\administrators

Match Group vendors
   ChrootDirectory F:\Vendors
   #ChrootDirectory F:\Vendors\%u
   #ChrootDirectory %h
   ForceCommand internal-sftp
   X11Forwarding no
   AllowTcpForwarding no

# no default banner path
#Banner F:\Vendors\SFTP-Welcome.txt
#Banner /SFTP-Welcome.txt

Using the -v (verbose) option in my SFTP command (sftp -v vendor1@its.my.ip.addr) yields:

debug1: Authentications that can continue: publickey,password,keyboard-interactive debug1: Next authentication method: publickey debug1: Trying private key: C:\Users\mylocalusername/.ssh/id_rsa debug1: Trying private key: C:\Users\mylocalusername/.ssh/id_dsa debug1: Trying private key: C:\Users\mylocalusername/.ssh/id_ecdsa debug1: Trying private key: C:\Users\mylocalusername/.ssh/id_ed25519 debug1: Trying private key: C:\Users\mylocalusername/.ssh/id_xmss debug1: Next authentication method: keyboard-interactive debug1: Authentications that can continue: publickey,password,keyboard-interactive debug1: Next authentication method: password debug1: read_passphrase: can't open /dev/tty: No such file or directory vendor1@its.my.ip.addr's password: debug1: Authentication succeeded (password). Authenticated to its.my.ip.addr ([its.my.ip.addrr]:22). debug1: channel 0: new [client-session] debug1: Requesting no-more-sessions@openssh.com debug1: Entering interactive session. debug1: pledge: network Connection reset by its.my.ip.addr port 22 Connection closed

That user "mylocalusername" is my local client PC Windows login name. I've attached both server and client debug logs. For the server logging, I edited my ssd_config to have:

   # Logging
   SyslogFacility LOCAL0
   LogLevel DEBUG3

and got the output of my C:\ProgramData\ssh\logs\sshd.log in the attached, renamed server logfile.

SSHD Debug 4 Client Side.txt SSHD Debug 4 Server Side.txt

Troubleshooting steps Just try to login with any SFTP client (this used to work just fine a few days ago).

Terminal issue? please go through wiki Nope.

Please answer the following

"OpenSSH for Windows" version 7.7.2.2

Server OperatingSystem Windows Server 2019 Datacenter

Client OperatingSystem Windows 10 Pro

What is failing SFTP logins

Expected output Successful login

Actual output Connection reset by server.public.ip.addr port 22 Connection closed

[ShieldOfSalvation]

Any potential immediate solutions available for my problem? This SFTP server going down has the potential of hurting my credibility and the company's.

[NoMoreFood]

This might be the multiple group match issue which I fixed in my fork:

https://github.com/NoMoreFood/openssh-portable/releases/tag/v7.9-merge-3

May want to give it a try.

[shorbachuk]

Hi @NoMoreFood, will your fork be merged into the official release? Waiting on a few fixes you have incorporated….

[NoMoreFood]

@shorbachuk No idea. I believe the Windows OpenSSH at Microsoft has been preoccupied with other efforts.

[bugz8unny69]

Why not? Is there PR?

[bugz8unny69]

It's taken to https://github.com/PowerShell/openssh-portable

[maertendMSFT]

Can you provide the server side logs with Debug 3 enabled?

[theamazingsamguy]

I'm also experiencing this issue. sshd.txt log.txt

SSHD:

AllowGroups sftpusers

DenyGroups administrators

ForceCommand internal-sftp

Match User wns ChrootDirectory C:\FTPRoot\WNS

Logs 5740 2020-06-11 11:34:13.023 Server listening on :: port 22. 5740 2020-06-11 11:34:13.023 Server listening on 0.0.0.0 port 22. 6128 2020-06-11 11:34:16.570 Connection from [Client_IP] port 63062 on [Server_IP] port 22 6128 2020-06-11 11:34:17.164 Accepted password for WNS from [Client_IP] port 63062 ssh2 6128 2020-06-11 11:34:17.429 User child is on pid 5068 7080 2020-06-11 11:39:48.222 Server listening on :: port 22. 7080 2020-06-11 11:39:48.222 Server listening on 0.0.0.0 port 22. 6436 2020-06-11 11:39:59.878 Connection from [Client_IP] port 63279 on [Server_IP] port 22 6436 2020-06-11 11:40:04.378 Accepted password for WNS from [Client_IP] port 63279 ssh2 6436 2020-06-11 11:40:06.003 User child is on pid 6532

...

6652 2020-06-11 11:40:54.473 debug1: attempt 2 failures 1 [preauth] 6652 2020-06-11 11:40:54.473 debug2: input_userauth_request: try method password [preauth] 6652 2020-06-11 11:40:54.473 debug3: mm_auth_password entering [preauth] 6652 2020-06-11 11:40:54.473 debug3: mm_request_send entering: type 12 [preauth] 6652 2020-06-11 11:40:54.473 debug3: mm_request_receive entering 6652 2020-06-11 11:40:54.473 debug3: monitor_read: checking request 12 6652 2020-06-11 11:40:54.489 debug3: mm_answer_authpassword: sending result 1 6652 2020-06-11 11:40:54.489 debug3: mm_request_send entering: type 13 6652 2020-06-11 11:40:54.489 Accepted password for WNS from [Client_IP] port 63306 ssh2 6652 2020-06-11 11:40:54.489 debug1: monitor_child_preauth: WNS has been authenticated by privileged process 6652 2020-06-11 11:40:54.489 debug3: mm_get_keystate: Waiting for new keys 6652 2020-06-11 11:40:54.489 debug3: mm_request_receive_expect entering: type 26 6652 2020-06-11 11:40:54.489 debug3: mm_request_receive entering 6652 2020-06-11 11:40:54.489 debug3: mm_get_keystate: GOT new keys 6652 2020-06-11 11:40:54.489 debug3: mm_auth_password: waiting for MONITOR_ANS_AUTHPASSWORD [preauth] 6652 2020-06-11 11:40:54.489 debug3: mm_request_receive_expect entering: type 13 [preauth] 6652 2020-06-11 11:40:54.489 debug3: mm_request_receive entering [preauth] 6652 2020-06-11 11:40:54.489 debug3: mm_auth_password: user authenticated [preauth] 6652 2020-06-11 11:40:54.489 debug3: send packet: type 52 [preauth] 6652 2020-06-11 11:40:54.489 debug3: mm_request_send entering: type 26 [preauth] 6652 2020-06-11 11:40:54.504 debug3: mm_send_keystate: Finished sending state [preauth] 6652 2020-06-11 11:40:54.504 debug1: monitor_read_log: child log fd closed 6652 2020-06-11 11:40:54.864 debug3: spawning "C:\Windows\System32\OpenSSH\sshd.exe" "-z" 6652 2020-06-11 11:40:54.911 User child is on pid 6204 6652 2020-06-11 11:40:54.911 debug3: send_rexec_state: entering fd = 5 config len 318 6652 2020-06-11 11:40:54.911 debug3: ssh_msg_send: type 0 6652 2020-06-11 11:40:54.973 debug3: send_rexec_state: done 6652 2020-06-11 11:40:54.973 debug3: ssh_msg_send: type 0 6652 2020-06-11 11:40:55.066 debug3: ssh_msg_send: type 0 6652 2020-06-11 11:40:55.113 debug3: ssh_msg_send: type 0 6652 2020-06-11 11:40:55.160 debug3: ssh_msg_send: type 0 6652 2020-06-11 11:40:55.207 debug3: ssh_msg_send: type 0 6652 2020-06-11 11:40:55.238 debug3: ReadFileEx() ERROR:109, io:0000023C145314A0 6652 2020-06-11 11:40:55.254 debug3: mm_request_receive entering 6652 2020-06-11 11:40:55.254 debug1: do_cleanup

[theamazingsamguy]

Mine appears to be tied to chroot - as soon as that's commented out it starts working again.

[bagajjal]

@theamazingsamguy - Did you debug why it's causing issue with CHROOT? Is the user didn't have sufficient permissions?

To debug further, you need to look at sftp-server logs.. Add the below config to the $env:programdata\ssh\sshd_config file and then restart the sshd service (net stop sshd; net start sshd).. Subsystem sftp sftp-server.exe -f LOCAL0 -l DEBUG3

Please note this will enable the file based logging. After you are done with the debugging revert your change otherwise you might run out of disk space depending on your traffic.

[theamazingsamguy]

I had a colleague look at it and my folder structure didn't match what I had in the config file. All on me.