search

neutrinolabs / xrdp

xrdp: an open source RDP server
http://www.xrdp.org/
Apache License 2.0
5.51k stars 1.72k forks source link

Unable to reconnect - reconnectwm.sh defunct #3156

Open sousaaguilherme opened 2 weeks ago

sousaaguilherme commented 2 weeks ago

xrdp version

0.9.17-2

Detailed xrdp version, build options

xrdp 0.9.17
  A Remote Desktop Protocol Server.
  Copyright (C) 2004-2020 Jay Sorg, Neutrino Labs, and all contributors.
  See https://github.com/neutrinolabs/xrdp for more information.

  Configure options:
      --build=x86_64-redhat-linux-gnu
      --host=x86_64-redhat-linux-gnu
      --program-prefix=
      --disable-dependency-tracking
      --prefix=/usr
      --exec-prefix=/usr
      --bindir=/usr/bin
      --sbindir=/usr/sbin
      --sysconfdir=/etc
      --datadir=/usr/share
      --includedir=/usr/include
      --libdir=/usr/lib64
      --libexecdir=/usr/libexec
      --localstatedir=/var
      --sharedstatedir=/var/lib
      --mandir=/usr/share/man
      --infodir=/usr/share/info
      --enable-fuse
      --enable-pixman
      --enable-painter
      --enable-vsock
      --with-socketdir=/run/xrdp
      build_alias=x86_64-redhat-linux-gnu
      host_alias=x86_64-redhat-linux-gnu
      CFLAGS=-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection
      LDFLAGS=-Wl,-z,relro  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld
      PKG_CONFIG_PATH=:/usr/lib64/pkgconfig:/usr/share/pkgconfig

  Compiled with OpenSSL 1.1.1k  FIPS 25 Mar 2021

Operating system & version

CentOS Stream release 8

Installation method

dnf / apt / zypper / pkg / etc

Which backend do you use?

Xvnc

What desktop environment do you use?

Xfce

Environment xrdp running on

VM

What's your client?

Microsoft's official client

Area(s) with issue?

Session manager (sesman)

Steps to reproduce

Sometimes users try to reconnect to a session and they are unable to.

I can see that reconnectwm.sh is defunct.

✔️ Expected Behavior

User is able to reconnect

❌ Actual Behavior

User is unable to reconnect and prompted with a message like:

connecting to sesman ip 127.0.0.1 port 3350
sesman connect ok
sending login info to session manager, please wait...
login successful for display 120
VNC started connecting
VNC connecting to 127.0.0.1 6020
VNC error - problem connecting
some problem

After a long time, like a day, tipically users are able to connect again.

I can't restart xrdp or xrdp-sesman to solve the problem, since we have dozens of users connected and that would make them loose their work and it's very hard to accommodate such a maintenance window.

Anything else?

In xrdp-sesman.log I get logs like this:

[20240709-14:33:46] [INFO ] Socket 5: AF_INET connection received from 127.0.0.1 port 60408
[20240709-14:33:46] [DEBUG] session_get_bydata: search policy 0 U user1 W 1920 H 1080 bpp 16 T 2 IP 172.27.6.151:51825 - socket: 12
[20240709-14:33:46] [DEBUG] session_get_bydata: try 0x55808882ed40 U user2 W 1920 H 1080 bpp 16 T 2 IP 172.27.6.138:61845 - socket: 12
[20240709-14:33:46] [DEBUG] session_get_bydata: try 0x55808881c880 U user3 W 1648 H 916 bpp 16 T 2 IP 172.27.10.20:60571 - socket: 12
[20240709-14:33:46] [DEBUG] session_get_bydata: try 0x55808881e300 U user1 W 1920 H 1080 bpp 16 T 2 IP 172.27.6.151:58819 - socket: 12
[20240709-14:33:46] [INFO ] ++ reconnected session: username user1, display :23.0, session_pid 3446468, ip 172.27.6.151:51825 - socket: 12
[20240709-14:33:46] [ERROR] sesman_data_in: scp_process_msg failed
[20240709-14:33:46] [ERROR] sesman_main_loop: trans_check_wait_objs failed, removing trans
[20240709-14:33:46] [DEBUG] Closed socket 5 (AF_INET 127.0.0.1:3350)
[20240709-14:33:46] [INFO ] Starting session reconnection script on display 23: /usr/libexec/xrdp/reconnectwm.sh
[20240709-14:33:46] [DEBUG] Calling exec (excutable: /usr/libexec/xrdp/reconnectwm.sh, arguments: /usr/libexec/xrdp/reconnectwm.sh )
matt335672 commented 2 weeks ago

That log looks fine- the errors are generated when the xrdp connection to sesman is terminated.

This is interesting and may well be the cause of your problem:-

login successful for display 120
VNC started connecting
VNC connecting to 127.0.0.1 6020

By default VNC allocates ports for a display :$D.0 as follows:-

VNC client TCP port : 5900 + $D VNC X11 TCP port : 6000 + $D

So for display :20.0, VNC will use TCP ports 5920 and 6020. For display 120, VNC will use ports 6020 and 6120. You can see there's a collision here. If you've got a user on :20.0, a user on :120.0 won't be able to connect, as xrdp will connect to TCP port 6020 and send it VNC commands. However port 6020 will be expecting X11 commands.

Later versions of xrdp place a limit on the display number to stop this happening.

A display number of 120 suggests you've got at least 111 sessions active (allocations 1 uses display 10, allocation 2 uses display 11 and so-on). This is pretty enormous, and I think needs some investigation. Do you have that many users on one machine?

sousaaguilherme commented 2 weeks ago

Thanks for the feedback @matt335672

We do have a lot of users. I just did a count of Xvnc sessions and currently I see 76 processes.

What i suspect is happening is that some of the sessions get defunct, but xrdp still keeps record of them somehow and still tries to reconnect the user to an old connection that no longer exists. But I really don't understand the inner works of xrdp, this is just a wild guess from reading the logs and seeing a lot of defunct reconnectwm.sh processes.

Is there a way to like make xrdp forget about the old session and just try and create a new one? I've found that the sockets are created under /run/xrdp/* , maybe deleting the respective sockets of those old connections would make xrdp create a new one?

Best regards

matt335672 commented 2 weeks ago

You can use the xrdp-sesadmin command to list the sessions that xrdp knows about. On your version of xrdp however it's very clunky, and you'll need to authenticate with a username and password. It should however, let you tie up usernames with sessions.

There's no way at the moment to get xrdp to forget about sessions. The state information is held in memory by xrdp-sesadmin.

What do you mean by 'defunct reconnectwm.sh' processes?

sousaaguilherme commented 2 weeks ago

Do you think an upgrade to 0.9.22.1-2 could solve this problems? It's the version I have available at the EPEL repo.

Are there breaking changes and configurations I need to change from 0.9.17?

reconnectwm.sh is the script configured in ReconnectScript in sesman, I have the default one. Currently, I have 65 defunct (zombie) reconnectwm.sh processes, with 10 months 23 days of xrdp service running.

Thanks!

matt335672 commented 2 weeks ago

You shouldn't have any zombie reconnectwm.sh processes, let alone 65 of them. That suggests sesman is missing the SIGCHLD signals from these processes.

I've done a bit of archeology, and this was fixed for v0.9.19 - see #2146, #2151, #2168. So an upgrade to 0.9.22.1 should help.

As far as breaking changes go, here are the relevant release notes:-

I can't see anything in there which I'd describe as a breaking change. However - you're the systems admin, so if it does break you'll be the one explaining why! Please look through the above yourself. I've been a systems admin myself, and I can see you've got a huge system there with no scope for down-time. Take all the usual precautions (including a rollback strategy, tested on a VM or desktop machine) and you should be fine. Please ask if you need any clarification from the release notes above.

sousaaguilherme commented 2 weeks ago

Thank you so much @matt335672 !! This host is for accessing a very costumized htcondor cluster for our users, so it's very crowded 😄

I'll try to find a maintenance window to do the update during a weekend. It's a VM, so a snapshot should be fine in case we need to rollback.

Best regards!