XRDP and VNC inconsistencies

[SensorAnalyticsAus]

I have RPI Imager installs of Bookworm on a PI5 and a PI3B. Both also have xrdp installed, VNC enabled, and W1 X11 Openbox window manager with X11 backend option selected within raspi-config. PI5 can be connected over xrdp while VNC comes up with Cannot currently show the desktop after successfully authenticating to VNC.

OTH PI3B connection over xrdp fails with MS Remote Desktop client opening up a black screen which after a few minutes timesout to a grey popup ending with a message error connecting to user session with an OK button. Clicking on it leads to another popup screen for re-typing in the password, and this also fails after a long wait. Last few entries in .xorgxrdp-11.log:

[867399.538] (WW) xf86CloseConsole: KDSETMODE failed: Bad file descriptor
[867399.538] (WW) xf86CloseConsole: VT_GETMODE failed: Bad file descriptor
[867399.541] (EE) Server terminated with error (1). Closing log file.

paradoxically, VNC connection works on PI3B.

RPIs OS and HW info

PI5 uname -a Linux rpi5 6.6.31+rpt-rpi-2712 #1 SMP PREEMPT Debian 1:6.6.31-1+rpt1 (2024-05-29) aarch64 GNU/Linux

cat /proc/cpuinfo
...
Revision    : d04170
Serial      : c4c6119e310d22ba
Model       : Raspberry Pi 5 Model B Rev 1.0

PI3B uname -a Linux rpi3bw 6.6.31+rpt-rpi-v8 #1 SMP PREEMPT Debian 1:6.6.31-1+rpt1 (2024-05-29) aarch64 GNU/Linux

cat /proc/cpuinfo
...
Revision    : a22082
Serial      : 00000000a37635af
Model       : Raspberry Pi 3 Model B Rev 1.2

I can get to gui desktop on both these headless PI's (one way or the other) but these inconsistencies are not re-assuring.

[lurch]

Using the X11 backend is deprecated on Raspberry Pi 5. Are you able to use the Wayfire backend and WayVNC instead?

[SensorAnalyticsAus]

I have switched to Wayfire backend on PI5, no change i.e. xrdp works and vnc shows 'cannot currently show the desktop' message. How does one use WayVNC? Can't see it in raspi-config only option is to enable VNC (which is done).

[lurch]

https://www.raspberrypi.com/documentation/computers/remote-access.html#vnc

I believe that if you use raspi-config to enable VNC, it'll automatically use WayVNC if you're using Wayfire or vncserver-x11 if you're using X11. So if you switched to X11 and then enabled VNC, and then switched back to Wayfire, perhaps it's still trying to use vncserver-x11 with Wayfire? (which obviously won't work). Does it work if (after you've switched to Wayfire) you use raspi-config to disable VNC (i.e. disable vncserver-x11), reboot, and then use raspi-config to enable VNC again (i.e. enable WayVNC)?

Note that when using WayVNC you might also need to use a different VNC client to what you were using with vncserver-x11 - the documentation link I provided above recommends TigerVNC.

[SensorAnalyticsAus]

I was able to switch to WayVNC with your docu link and information, thanks. However tigerVNC client (1.3.1 for MacOS) connects to a blank grey screen.