search

neutrinolabs / xrdp

xrdp: an open source RDP server
http://www.xrdp.org/
Apache License 2.0
5.65k stars 1.73k forks source link

x264 not working #3141

Open thexperiments opened 3 months ago

thexperiments commented 3 months ago

xrdp version

devel

Detailed xrdp version, build options

xrdp 0.10.80
  A Remote Desktop Protocol Server.
  Copyright (C) 2004-2024 Jay Sorg, Neutrino Labs, and all contributors.
  See https://github.com/neutrinolabs/xrdp for more information.

  Configure options:
      --prefix=/usr
      --sysconfdir=/etc
      --localstatedir=/var
      --sbindir=/usr/bin
      --libexecdir=/usr/lib
      --with-systemdsystemdunitdir=/usr/lib/systemd/system
      --enable-jpeg
      --enable-tjpeg
      --enable-fuse
      --enable-opus
      --enable-rfxcodec
      --enable-mp3lame
      --enable-pixman
      --enable-vsock
      --enable-x264
      CFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions         -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security         -fstack-clash-protection -fcf-protection         -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -g -ffile-prefix-map=/home/thex/xrdp-devel-git/src=/usr/src/debug/xrdp-devel-git -flto=auto
      LDFLAGS=-Wl,-O1 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now          -Wl,-z,pack-relative-relocs -flto=auto
      CXXFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions         -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security         -fstack-clash-protection -fcf-protection         -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -Wp,-D_GLIBCXX_ASSERTIONS -g -ffile-prefix-map=/home/thex/xrdp-devel-git/src=/usr/src/debug/xrdp-devel-git -flto=auto

  Compiled with OpenSSL 3.3.1 4 Jun 2024

Operating system & version

arch

Installation method

other

Which backend do you use?

xorgxrdp

What desktop environment do you use?

KDE

Environment xrdp running on

VM with Nvidia 1080 TI via HyperV Partitioning

What's your client?

HyperV Enhanced session / Windows 11 RDP client

Area(s) with issue?

Performance

Steps to reproduce

✔️ Expected Behavior

Session uses h264

❌ Actual Behavior

session is using rfx pro instead of h264

Anything else?

Using AUR

Can't find any references to XRDP_USE_HELPER in the inis or code. Are there any other magic things I need to configure or is there simply some code missing still?

Wondering if there is something wrong with the flags to not fall into this: https://github.com/neutrinolabs/xrdp/blob/1d927add296ddc64511f48e20cc326b74a19d6ce/xrdp/xrdp_mm.c#L1412

[2024-06-29T23:53:54.674+0200] [INFO ] xrdp_egfx_create: error 0 channel_id 1
[2024-06-29T23:53:54.679+0200] [INFO ] xrdp_mm_egfx_caps_advertise:
[2024-06-29T23:53:54.683+0200] [INFO ]   version 0x00080004 flags 0x00000000 (index: 0)
[2024-06-29T23:53:54.687+0200] [INFO ]   version 0x00080105 flags 0x00000000 (index: 1)
[2024-06-29T23:53:54.692+0200] [INFO ]   version 0x000a0002 flags 0x00000000 (index: 2)
[2024-06-29T23:53:54.696+0200] [INFO ]   version 0x000a0200 flags 0x00000000 (index: 3)
[2024-06-29T23:53:54.701+0200] [INFO ]   version 0x000a0301 flags 0x00000000 (index: 4)
[2024-06-29T23:53:54.705+0200] [INFO ]   version 0x000a0400 flags 0x00000000 (index: 5)
[2024-06-29T23:53:54.710+0200] [INFO ]   version 0x000a0502 flags 0x00000000 (index: 6)
[2024-06-29T23:53:54.714+0200] [INFO ]   version 0x000a0600 flags 0x00000000 (index: 7)
[2024-06-29T23:53:54.718+0200] [INFO ]   version 0x000a0701 flags 0x00000000 (index: 8)
[2024-06-29T23:53:54.723+0200] [INFO ]   replying version 0x000a0701 flags 0x00000000
[2024-06-29T23:53:54.727+0200] [INFO ] xrdp_mm_egfx_caps_advertise: xrdp_egfx_send_capsconfirm error 0 best_index 8
[2024-06-29T23:53:54.731+0200] [INFO ] xrdp_egfx_reset_graphics:
[2024-06-29T23:53:54.736+0200] [INFO ] xrdp_egfx_reset_graphics: width 3840 height 2160 monitorcount 1
[2024-06-29T23:53:54.740+0200] [INFO ] xrdp_mm_egfx_caps_advertise: xrdp_egfx_send_reset_graphics error 0 monitorCount 0
[2024-06-29T23:53:54.744+0200] [INFO ] xrdp_mm_egfx_create_surfaces: map surface_id 0 left 0 top 0 width 3840 height 2160
[2024-06-29T23:53:54.749+0200] [INFO ] xrdp_encoder_create: starting gfx rfx pro codec session
matt335672 commented 3 months ago

@thexperiments - H.264 isn't yet supported in v0.10.x. From the release notes

GFX H.264/AVC 444 mode and hardware-accelerated encoding are not supported in this version yet.

thexperiments commented 3 months ago

I know, but arch is building from the devel branch sources thus I was hoping it is in there already. Is there a branch I can use to test it?

matt335672 commented 3 months ago

There's nothing for this in devel yet I'm afraid.

What will become the next version of xrdp is documented on this wiki page. v0.10.x was a big release graphics-wise, and we're still working on the issues that are being reported against it.

thexperiments commented 3 months ago

Ok thanks, understand that. I don't have to much time but if there is something I can do to push this forward let me know. Is there a PR for the missing code? As I understand there was some fork where this was already working and to me it looked like most of the changes from there already made it to the devel branch.

matt335672 commented 3 months ago

@Nexarian - is there any way you can make use of @thexperiments offer above? I'll admit I'm not fully up to speed on where you and Jay have gotten to with this.

Nexarian commented 3 months ago

x264 was merged into devel awhile ago? I'll take a look

Hiero32 commented 3 months ago

I built and installed devel on Ubuntu 24.04/VirtualBox.

$ xrdp -v
xrdp 0.10.80
  A Remote Desktop Protocol Server.
  Copyright (C) 2004-2024 Jay Sorg, Neutrino Labs, and all contributors.
  See https://github.com/neutrinolabs/xrdp for more information.

  Configure options:
      --enable-fuse
      --enable-jpeg
      --enable-rfxcodec
      --enable-mp3lame
      --enable-vsock
      --enable-pixman
      --enable-x264

  Compiled with OpenSSL 3.0.13 30 Jan 2024

Accessed from both Windows 10 (mstsc.exe) and Windows 11.

$ sudo cat /var/log/xrdp.log | grep xrdp_encoder
[2024-07-04T21:02:37.109+0900] [INFO ] xrdp_encoder_create: starting h264 codec session gfx
[2024-07-04T21:02:38.994+0900] [INFO ] xrdp_encoder_x264_encode: x264_encoder_open rv 0x71d404306e70 for 2560 height 1600

It looks working. I played back youtube. Bandwidth is less than 50% compared to no h264.

Nexarian commented 3 months ago

@thexperiments: @Hiero32's testing matches mine. Can you try again on a clean VM?

thexperiments commented 3 months ago

Interesting, what config are you using? As said I could not find the flags/config mentioned anywhere in the sources. Is it necessary to set the flags? Is there anything else to configure?

(damn always posting from wrong account)

Nexarian commented 3 months ago

You need to build with --enable-x264. I'm not sure if you need --enable-rfxcodec as well.

Try this gist to install it.

thexperiments commented 3 months ago

I have both as can be seen in the initial issue. I was thinking about any config for xorg or similar. I can't see anything in the gist that I currently don't have... can't easily apply it as I'm on arch with KDE

thexperiments commented 3 months ago

Looks like both clients I was using do not support h264... investigating...

[2024-07-04T19:03:27.831+0200] [INFO ] xrdp_caps_process_codecs: NSCodec(CA8D1BB9-000F-154F-589F-AE2D1A87E2D6), codec id [1], properties len [3]
[2024-07-04T19:03:27.836+0200] [INFO ] xrdp_caps_process_codecs: Image RemoteFX(2744CCD4-9D8A-4E74-803C-0ECBEEA19C54), codec id [5], properties len [49]
thexperiments commented 3 months ago

it becomes more mysterious, debugging with GDB now

Thread 6 "xrdp" hit Breakpoint 2, xrdp_mm_egfx_caps_advertise (user=0x77cb4e6ca110, caps_count=9, versions=<optimized out>, flagss=<optimized out>)
    at /usr/src/debug/xrdp-devel-git/xrdp-devel-git/xrdp/xrdp_mm.c:1380
1380            version = ver_flags[index].version;
(gdb) print best_pro_index
$1 = <optimized out>
(gdb) print best_h264_index
$2 = <optimized out>
Nexarian commented 3 months ago

You have to recompile XRDP with --enable-all-debug

thexperiments commented 3 months ago

Really wired, build log tells me it is compiled with the right configure resulting in right defines...

cat build.log |grep -i x264
checking for XRDP_X264... yes
configure: running /bin/sh ./configure --disable-option-checking '--prefix=/usr'  '--sysconfdir=/etc' '--localstatedir=/var' '--sbindir=/usr/bin' '--libexecdir=/usr/lib' '--with-systemdsystemdunitdir=/usr/lib/systemd/system' '--enable-jpeg' '--enable-tjpeg' '--enable-fuse' '--enable-opus' '--enable-rfxcodec' '--enable-mp3lame' '--enable-pixman' '--enable-vsock' '--enable-x264' 'CFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions         -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security         -fstack-clash-protection -fcf-protection         -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -g -ffile-prefix-map=/home/thex/xrdp-devel-git/src=/usr/src/debug/xrdp-devel-git -flto=auto' 'LDFLAGS=-Wl,-O1 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now          -Wl,-z,pack-relative-relocs -flto=auto' 'CXXFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions         -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security         -fstack-clash-protection -fcf-protection         -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -Wp,-D_GLIBCXX_ASSERTIONS -g -ffile-prefix-map=/home/thex/xrdp-devel-git/src=/usr/src/debug/xrdp-devel-git -flto=auto' --cache-file=/dev/null --srcdir=.
configure: running /bin/sh ./configure --disable-option-checking '--prefix=/usr'  '--sysconfdir=/etc' '--localstatedir=/var' '--sbindir=/usr/bin' '--libexecdir=/usr/lib' '--with-systemdsystemdunitdir=/usr/lib/systemd/system' '--enable-jpeg' '--enable-tjpeg' '--enable-fuse' '--enable-opus' '--enable-rfxcodec' '--enable-mp3lame' '--enable-pixman' '--enable-vsock' '--enable-x264' 'CFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions         -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security         -fstack-clash-protection -fcf-protection         -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -g -ffile-prefix-map=/home/thex/xrdp-devel-git/src=/usr/src/debug/xrdp-devel-git -flto=auto' 'LDFLAGS=-Wl,-O1 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now          -Wl,-z,pack-relative-relocs -flto=auto' 'CXXFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions         -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security         -fstack-clash-protection -fcf-protection         -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -Wp,-D_GLIBCXX_ASSERTIONS -g -ffile-prefix-map=/home/thex/xrdp-devel-git/src=/usr/src/debug/xrdp-devel-git -flto=auto' --cache-file=/dev/null --srcdir=.
  x264                    yes
gcc -DHAVE_CONFIG_H -I. -I..  -DXRDP_CFG_PATH=\"/etc/xrdp\" -DXRDP_SBIN_PATH=\"/usr/bin\" -DXRDP_SHARE_PATH=\"/usr/share/xrdp\" -DXRDP_PID_PATH=\"/var/run\" -DXRDP_MODULE_PATH=\"/usr/lib/xrdp\" -DXRDP_SOCKET_ROOT_PATH=\"/var/run/xrdp\" -I.. -I../common -I../libipm -I../libxrdp -I../third_party -I../third_party/tomlc99 -DXRDP_RFXCODEC -I../librfxcodec/include -DXRDP_X264 -DX264_API_IMPORTS -DXRDP_PIXMAN -I/usr/include/pixman-1 -DXRDP_PAINTER -I../libpainter/include   -march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions         -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security         -fstack-clash-protection -fcf-protection         -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -g -ffile-prefix-map=/home/thex/xrdp-devel-git/src=/usr/src/debug/xrdp-devel-git -flto=auto -Wall -Wwrite-strings -Wmissing-prototypes -Werror -MT xrdp.o -MD -MP -MF .deps/xrdp.Tpo -c -o xrdp.o xrdp.c
/bin/sh ../libtool  --tag=CC   --mode=link gcc  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions         -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security         -fstack-clash-protection -fcf-protection         -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -g -ffile-prefix-map=/home/thex/xrdp-devel-git/src=/usr/src/debug/xrdp-devel-git -flto=auto -Wall -Wwrite-strings -Wmissing-prototypes -Werror  -Wl,-O1 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now          -Wl,-z,pack-relative-relocs -flto=auto -o xrdp funcs.o lang.o xrdp.o xrdp_bitmap.o xrdp_bitmap_load.o xrdp_bitmap_common.o xrdp_cache.o xrdp_encoder.o xrdp_font.o xrdp_listen.o xrdp_login_wnd.o xrdp_mm.o xrdp_painter.o xrdp_process.o xrdp_region.o xrdp_egfx.o xrdp_wm.o xrdp_main_utils.o xrdp_encoder_x264.o ../common/libcommon.la ../libipm/libipm.la ../libxrdp/libxrdp.la ../third_party/tomlc99/libtoml.la  ../librfxcodec/src/.libs/librfxencode.a -lx264 -lpixman-1 ../libpainter/src/.libs/libpainter.a
libtool: link: gcc -march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -g -ffile-prefix-map=/home/thex/xrdp-devel-git/src=/usr/src/debug/xrdp-devel-git -flto=auto -Wall -Wwrite-strings -Wmissing-prototypes -Werror -Wl,-O1 -Wl,--sort-common -Wl,--as-needed -Wl,-z -Wl,relro -Wl,-z -Wl,now -Wl,-z -Wl,pack-relative-relocs -flto=auto -o .libs/xrdp funcs.o lang.o xrdp.o xrdp_bitmap.o xrdp_bitmap_load.o xrdp_bitmap_common.o xrdp_cache.o xrdp_encoder.o xrdp_font.o xrdp_listen.o xrdp_login_wnd.o xrdp_mm.o xrdp_painter.o xrdp_process.o xrdp_region.o xrdp_egfx.o xrdp_wm.o xrdp_main_utils.o xrdp_encoder_x264.o  ../common/.libs/libcommon.so ../libipm/.libs/libipm.so ../libxrdp/.libs/libxrdp.so /home/thex/xrdp-devel-git/src/xrdp-devel-git/common/.libs/libcommon.so -lpthread -lssl -lcrypto -lturbojpeg ../third_party/tomlc99/.libs/libtoml.so ../librfxcodec/src/.libs/librfxencode.a -lx264 -lpixman-1 ../libpainter/src/.libs/libpainter.a -Wl,-rpath -Wl,/usr/lib/xrdp
thexperiments commented 3 months ago

also after building with --enable--all-debug I still can't set a breakpoint to xrdp_mm.c:1414 it jumps to 1418 (next thing that is not optimized away)

Nexarian commented 3 months ago

You may have to change the -O2 to -O0, this might be a bug in the compile flags.

matt335672 commented 2 months ago

If you're on Ubuntu, you might want to add -fvar-tracking as well. I found without that, it was challenging to examine variables.

Hiero32 commented 2 months ago

I built and installed xrdp-devel-git and xorgxrdp-devel-git on arch linux. desktop is gnome.

h264 is working. Bandwidth for youtube is roughly same as on Ubuntu 24.04.

$ sudo cat /var/log/xrdp.log |grep xrdp_encoder
[2024-07-06T10:51:49.419+0900] [INFO ] xrdp_encoder_create: starting h264 codec session gfx
[2024-07-06T10:51:52.412+0900] [INFO ] xrdp_encoder_x264_encode: x264_encoder_open rv 0x73fafc000be0 for width 2560 height 1600

I just added --enable-vsock and --enable-x264 to PKGBUILD of xrdp-devel-git. build() in PKGBUILD is as follows.

build() {
  cd $pkgname
  ./configure --prefix=/usr \
              --sysconfdir=/etc \
              --localstatedir=/var \
              --sbindir=/usr/bin \
          --libexecdir=/usr/lib \
              --with-systemdsystemdunitdir=/usr/lib/systemd/system \
              --enable-jpeg \
              --enable-tjpeg \
              --enable-fuse \
          --enable-opus \
          --enable-rfxcodec \
          --enable-mp3lame \
          --enable-pixman \
              --enable-vsock \
              --enable-x264

  # Fight unused direct deps
  sed -i -e 's/ -shared / -Wl,-O1,--as-needed\0 /g' -e 's/    if test "$export_dynamic" = yes && test -n "$export_dynamic_flag_spec"; then/      func_append compile_command " -Wl,-O1,--as-needed"\n      func_append finalize_command " -Wl,-O1,--as-needed"\n\0/' libtool
  make V=0
}
thexperiments commented 2 months ago

It started working after changing the optimization yesterday night. However i need to investigate more as there was something broken and I hat to completely reinstall xrdp and xorgrdp. (thus not sure if it was only the optimization)

I see GPU use (although I think h264 is still software right) but the performance is still meh maybe 5-10 FPS with quite some latency, any hints at what performance to expect. Connection is local to a VM

thexperiments commented 2 months ago

Switched back to vsock and performance is a bit better. Had switched to tcp earlier to try clients from differnt machines