search

neutrinolabs / xrdp

xrdp: an open source RDP server
http://www.xrdp.org/
Apache License 2.0
5.64k stars 1.73k forks source link

Authentication error has occurred. The function requested is not supported #3208

Closed tumatanquang closed 1 month ago

tumatanquang commented 1 month ago

xrdp version

0.10.1

Detailed xrdp version, build options

xrdp 0.10.1
  A Remote Desktop Protocol Server.
  Copyright (C) 2004-2024 Jay Sorg, Neutrino Labs, and all contributors.
  See https://github.com/neutrinolabs/xrdp for more information.

  Configure options:
      --build=aarch64-redhat-linux-gnu
      --host=aarch64-redhat-linux-gnu
      --program-prefix=
      --disable-dependency-tracking
      --prefix=/usr
      --exec-prefix=/usr
      --bindir=/usr/bin
      --sbindir=/usr/sbin
      --sysconfdir=/etc
      --datadir=/usr/share
      --includedir=/usr/include
      --libdir=/usr/lib64
      --libexecdir=/usr/libexec
      --localstatedir=/var
      --sharedstatedir=/var/lib
      --mandir=/usr/share/man
      --infodir=/usr/share/info
      --enable-fuse
      --enable-pixman
      --enable-painter
      --enable-vsock
      --enable-ipv6
      --with-socketdir=/run/xrdp
      --with-imlib2
      build_alias=aarch64-redhat-linux-gnu
      host_alias=aarch64-redhat-linux-gnu
      CFLAGS=-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -fasynchronous-unwind-tables -fstack-clash-protection
      LDFLAGS=-Wl,-z,relro  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld
      CXXFLAGS=-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -fasynchronous-unwind-tables -fstack-clash-protection
      PKG_CONFIG_PATH=:/usr/lib64/pkgconfig:/usr/share/pkgconfig

  Compiled with OpenSSL 1.1.1k  FIPS 25 Mar 2021

Operating system & version

AlmaLinux 8.10 (Cerulean Leopard)

Installation method

dnf / apt / zypper / pkg / etc

Which backend do you use?

tigervnc-server-minimal.aarch64-1.13.1-12.el8_10

What desktop environment do you use?

Xfce

Environment xrdp running on

Oracle Standard.A1.Flex VM

What's your client?

Remote Desktop Connection 6.1.7601

Area(s) with issue?

Authentication

Steps to reproduce

  1. Install epel-release and update: 
    dnf -y install epel-release
dnf -y -y update
  2. Install Xfce4: 
    dnf -y groupinstall "Xfce"
echo "startxfce4" > ~/.Xclients
chmod +x ~/.Xclients
  3. Install xrdp (will automatically install tigervnc-license.noarch and tigervnc-server-minimal.aarch64): 
    dnf -y install xrdp
systemctl start xrdp.service
systemctl enable xrdp.service
  4. Configure firewall and SELinux: 
    firewall-cmd --zone=public --add-port=3389/tcp --permanent
firewall-cmd --reload
semanage fcontext -a -t bin_t /usr/sbin/xrdp
semanage fcontext -a -t bin_t /usr/sbin/xrdp-sesman
chcon --type=bin_t /usr/sbin/xrdp
chcon --type=bin_t /usr/sbin/xrdp-sesman
restorecon -v /usr/sbin/xrdp*

    I then tried to connect via Remote Desktop Connection and got the error Authentication error has occurred. The function requested is not supported. Then I tried adding TLSv1, TLSv1.1 to ssl_protocols in the xrdp.ini file:

    vi /etc/xrdp/xrdp.ini
...
ssl_protocols=TLSv1, TLSv1.1, TLSv1.2, TLSv1.3
...
systemctl restart xrdp.service

    However, this error still occurs. Here is the output of systemctl status xrdp.service:

Before adding TLSv1, TLSv1.1 to ssl_protocols:

xrdp[1448]: [INFO ] Using default X.509 certificate: /etc/xrdp/cert.pem xrdp[1448]: [INFO ] Using default X.509 key file: /etc/xrdp/key.pem xrdp[1448]: [INFO ] Security protocol: configured [SSL|RDP], requested [SSL|HYBRID|RDP], selected [SSL] xrdp[1448]: [ERROR] SSL_accept: I/O error xrdp[1448]: [ERROR] trans_set_tls_mode: ssl_tls_accept failed xrdp[1448]: [ERROR] xrdp_sec_incoming: trans_set_tls_mode failed xrdp[1448]: [ERROR] xrdp_rdp_incoming: xrdp_sec_incoming failed xrdp[1448]: [ERROR] xrdp_process_main_loop: libxrdp_process_incoming failed xrdp[1448]: [ERROR] xrdp_iso_send: trans_write_copy_s failed xrdp[1448]: [ERROR] Sending [ITU T.125] DisconnectProviderUltimatum failed

After adding TLSv1, TLSv1.1 to ssl_protocols:

xrdp[1510]: [INFO ] Socket 14: connection accepted from [::ffff:]:15899 xrdp[1514]: [INFO ] Using default X.509 certificate: /etc/xrdp/cert.pem xrdp[1514]: [INFO ] Using default X.509 key file: /etc/xrdp/key.pem xrdp[1514]: [INFO ] Security protocol: configured [SSL|RDP], requested [SSL|HYBRID|RDP], selected [SSL] xrdp[1514]: [ERROR] SSL_accept: Failure in SSL library (protocol error?) xrdp[1514]: [ERROR] SSL: error:14209102:SSL routines:tls_early_post_process_client_hello:unsupported protocol xrdp[1514]: [ERROR] trans_set_tls_mode: ssl_tls_accept failed xrdp[1514]: [ERROR] xrdp_sec_incoming: trans_set_tls_mode failed xrdp[1514]: [ERROR] xrdp_rdp_incoming: xrdp_sec_incoming failed xrdp[1514]: [ERROR] xrdp_process_main_loop: libxrdp_process_incoming failed

However, when I change security_layer from negotiate to rdp it will connect normally, without even adding TLSv1, TLSv1.1 to ssl_protocols.

Is there a way for me to use tls or negotiate?

✔️ Expected Behavior

No response

❌ Actual Behavior

No response

Anything else?

No response

matt335672 commented 1 month ago

Your first log file looks fine. Generally a client connects, we send the TLS certificate, and then the client disconnects when it prompts the user what to do. The client will reconnect again if the user says to proceed.

However, the error "the function requested is not supported" suggests to me that your Windows client is insisting on the RDP server providing NLA. We don't support NLA. You'll need to stop the client insisting on using NLA.

Check the Advanced tab on mstsc.exe looks like this:-

image

Then put your xrdp.ini file back:- 1) Get rid of the TLS versions before 1.2. These should no longer be used. 2) set security_layer to tls or negotiate (up to you) 3) restart xrdp.service

Try to connect again, and report back exactly what you see on the client.

tumatanquang commented 1 month ago

@matt335672 I have configured the Advanced tab like this:

image

Then I removed selinux (including tigervnc-selinux) and firewall. Output of rpm -q firewalld:

package firewalld is not installed

Output of getenforce:

Disabled

All xrdp configuration (like security_layer and ssl_protocols) is default. Then I try to connect via Remote Desktop Connection. When the window below appeared, I clicked Don't ask me again... then Connect:

image

Error Authentication error has occurred. The function requested is not supported appears again:

image

When I check systemctl status xrdp.service:

xrdp[1293]: [INFO ] Socket 14: connection accepted from [::ffff:<MY_IP_ADDRESS>]:1241
xrdp[1385]: [INFO ] Using default X.509 certificate: /etc/xrdp/cert.pem
xrdp[1385]: [INFO ] Using default X.509 key file: /etc/xrdp/key.pem
xrdp[1385]: [INFO ] Security protocol: configured [SSL|RDP], requested [SSL|HYBRID|RDP], selected [SSL]
xrdp[1385]: [ERROR] SSL_accept: Failure in SSL library (protocol error?)
xrdp[1385]: [ERROR] SSL: error:14209102:SSL routines:tls_early_post_process_client_hello:unsupported protocol
xrdp[1385]: [ERROR] trans_set_tls_mode: ssl_tls_accept failed
xrdp[1385]: [ERROR] xrdp_sec_incoming: trans_set_tls_mode failed
xrdp[1385]: [ERROR] xrdp_rdp_incoming: xrdp_sec_incoming failed
xrdp[1385]: [ERROR] xrdp_process_main_loop: libxrdp_process_incoming failed
matt335672 commented 1 month ago

The server and client are failing to agree on a TLS version, or a cipher suite. There's an overview of the TLS handshake here:-

https://www.cloudflare.com/en-gb/learning/ssl/what-happens-in-a-tls-handshake/

That's an old version of the RDP client. I assume you're on Windows 7 SP1 or thereabouts. Assuming you're on x86_64, is KB3080079 installed?

tumatanquang commented 1 month ago

I am using Windows 7 build 7601.24214 updated to August 1, 2018. And when I check in Internet options I saw and enabled TLS 1.0 to TLS 1.2:

image

I think this is related to the latest version of xrdp or tigervnc because when I install xrdp on Ubuntu 20.04, it installs xrdp 0.9.12-1ubuntu0.1 along with OpenSSL 1.1.1f and xorgxrdp (instead of tigervnc) and I can connect directly right after installation without having to reconfigure/edit xrdp.ini. And xrdp on CentOS/AlmaLinux/Oracle Linux will be installed by default with tigervnc.

Additional information: I also tried installing xorgxrdp and enabling [Xorg] in xrdp.ini (similar to Ubuntu) but still got the same error.

matt335672 commented 1 month ago

From the logs, xrdp is getting a client hello, and doesn't like it. Things stop at that point. We're not getting as far as using xorgxrdp or tigervnc.

We've not changed a lot in this area - we just make library calls. So it still seems most likely to me that your client is not offering TLS 1.2. We can tell for sure if you can get a wireshark trace of the connection. We can extract the client hello and look at it.

Other than that:-

  1. I can't see from here whether Windows version 7601.24214 includes KB3080079. Are you able to check the updates list at your end?
  2. Also, it seems that AlmaLinux 8 disables TLS 1.0 and 1.1 by default.

Do you have a more recent client you can try connecting with? That will log the TLS parameters that have been successfully negotiated.

tumatanquang commented 1 month ago

I followed the instructions in the link to enable TLS 1.0/1.1 that you provided. After running the command update-crypto-policies --set LEGACY then restarting the VM, after the restart was complete I tried to connect via RDP and got this error:

image

When I check the status of xrdp via systemctl status xrdp.service command:

# systemctl status xrdp.service
● xrdp.service - xrdp daemon
   Loaded: loaded (/usr/lib/systemd/system/xrdp.service; disabled; vendor preset: disabled)
   Active: inactive (dead)
     Docs: man:xrdp(8)
           man:xrdp.ini(5)

I tried restarting it via systemctl restart xrdp.service command :

● xrdp.service - xrdp daemon
   Loaded: loaded (/usr/lib/systemd/system/xrdp.service; disabled; vendor preset: disabled)
   Active: active (running) since Tue 2024-08-27 17:03:34 UTC; 47s ago
     Docs: man:xrdp(8)
           man:xrdp.ini(5)
 Main PID: 1391 (xrdp)
    Tasks: 1 (limit: 19346)
   Memory: 7.5M
   CGroup: /system.slice/xrdp.service
           └─1391 /usr/sbin/xrdp --nodaemon

Aug 27 17:03:34 systemd[1]: Starting xrdp daemon...
Aug 27 17:03:34 systemd[1]: Started xrdp daemon.
Aug 27 17:03:34 xrdp[1391]: [INFO ] starting xrdp with pid 1391
Aug 27 17:03:34 xrdp[1391]: [INFO ] address [0.0.0.0] port [3389] mode 1
Aug 27 17:03:34 xrdp[1391]: [INFO ] listening to port 3389 on 0.0.0.0
Aug 27 17:03:34 xrdp[1391]: [INFO ] xrdp_listen_pp done

And tried connecting again via RDP and still got the SSL_accept: Failure in SSL library (protocol error?) error.

Then I tried editing ssl_protocols in xrdp.ini file to ssl_protocols=TLSv1, TLSv1.1, TLSv1.2, TLSv1.3 then restarted xrdp. Now it worked, I was able to connect using RDP!

But a new problem occurs, that is every time I restart the VM, xrdp will be inactive (dead)! I tried uninstalling xrdp along with tigervnc, then reinstalling xrdp (tigervnc was automatically installed). But after successful installation, I checked the status of xrdp, it is inactive (dead). I also tried deleting the /etc/xrdp directory after uninstalling xrdp and then reinstalling xrdp, but nothing changed either.

Do you have any idea how to fix this?

P/s: Firewall is not installed, SELinux is disabled.

matt335672 commented 1 month ago

Reboot and post the output of these commands so we can see what's going on at startup:-

sudo systemctl status xrdp
sudo journalctl -u xrdp -S "$(who -b | sed -e 's/^.*boot//')"

For the record, when you get a successful connection from your Win7 machine, what does this command say?

sudo grep -i 'tls connection' /var/log/xrdp.log | tail -4
tumatanquang commented 1 month ago

Here is the output of the first 2 commands you said:

# sudo systemctl status xrdp
● xrdp.service - xrdp daemon
   Loaded: loaded (/usr/lib/systemd/system/xrdp.service; disabled; vendor preset: disabled)
   Active: inactive (dead)
     Docs: man:xrdp(8)
           man:xrdp.ini(5)

# sudo journalctl -u xrdp -S "$(who -b | sed -e 's/^.*boot//')"
-- Logs begin at Wed 2024-08-28 09:50:59 UTC, end at Wed 2024-08-28 09:53:40 UTC. --
-- No entries --

After running update-crypto-policies command, xrdp will always be in inactive (dead) state even if I don't edit ssl_protocols in xrdp.ini file.

But in order to be able to connect I will edit ssl_protocols in xrdp.ini file and restart xrdp.

Here is the command line output after a successful connection from my Win7 machine:

# sudo grep -i 'tls connection' /var/log/xrdp.log | tail -4
[2024-08-28T09:59:18.611+0000] [INFO ] TLS connection established from [::ffff:<MY_IPV4_ADDRESS>]:9103 TLSv1 with cipher ECDHE-RSA-AES256-SHA
matt335672 commented 1 month ago

xrdp isn't enabled on boot:=

sudo systemctl enable xrdp

Also, your Win7 machine is definitely connecting with TLS 1.0. You're not getting a great deal of security from that, and future updates to the OS would be a concern.

tumatanquang commented 1 month ago

Oh, xrdp is running after reboot.

By the way, I also checked xrdp.log on Ubuntu, and got the result:

# sudo grep -i 'tls connection' /var/log/xrdp.log | tail -4
[20240829-02:31:02] [INFO ] Non-TLS connection established from ::ffff:<MY_IPV4_ADDRESS> port 13753: encrypted with standard RDP security

It is using RDP protocol so I did not get any error when connecting. But when I checked xrdp.ini, the values ​​of ssl_protocols and security_layer are both default values, i.e.:

...
security_layer=negotiate
...
ssl_protocols=TLSv1.2, TLSv1.3

It seems xrdp on Ubuntu automatically switched to RDP protocol when I didn't support TLS 1.2+ instead of giving an error like on AlmaLinux?

I also tried running update-crypto-policies --set LEGACY and editing ssl_protocols in xrdp.ini. It also switched to using TLS protocol instead of RDP.

matt335672 commented 1 month ago

It's more likely your Ubuntu installation is ignoring TLS as it can't read the certificate and/or key file. See Debian BTS #860890. Downstream probably won't change their stance on this.

tumatanquang commented 1 month ago

So the condition for xrdp to automatically switch from TLS to RDP is that the client is using an unsupported protocol and cannot read the certificate/key? If both of these conditions are not met, xrdp will not automatically switch from TLS to RDP and the client using the unsupported protocol will receive the error like me?

matt335672 commented 1 month ago

The decision as to whether RDP or TLS is used is made before the TLS handshake.

xrdp performs a sanity check on the certificate and key files. If these are unusable, TLS is not offered as an option.

If TLS is negotiated, it has to be used. In the protocol, there is no provision for a fallback to RDP if the TLS handshake fails.

tumatanquang commented 1 month ago

Oh, I see. This problem has been solved too, thank you very much.