xfreerdp-cli does not fall back to software rendering on some VAAPI errors

[jasom]

My video card (Radeon 6350, Linux radeon kernel driver) only supports up to 2048x1152 for h264 VAAPI decoding, so when I connect to a 1920x1200 display, the VAAPI frame setup fails

To Reproduce Steps to reproduce the behavior:

1. With a Radeon 6350, connect to a remote (I used freerdp-shadow-cli on the remonte) with more than 1152 horizontal lines
2. The initial image renders, but screen never updates (clicking/typing has an effect on the remote computer)
3. console shows repeated error messages like:

   [h264 @ 0x7fe7c805e1c0] Failed setup for format vaapi_vld: hwaccel initialisation returned error.
   [h264 @ 0x7fe7c805e1c0] decode_slice_header error
   [h264 @ 0x7fe7c805e1c0] Hardware does not support image size 1920x1200 (constraints: width 0-2048 height 0-1152).

4. Setting LIBVA_DRIVER_NAME to something bogus is a workaround; it causes VAAPI to fail to initialize and xfreerdp falls back on software rendering

Expected behavior xfreerdp-cli should fall back on software rendering

Application details

• This is FreeRDP version 2.7.0 (2.7.0)
• `xfreerdp /v:host -grab-keyboard
• Output of xfreerdp /buildconfig:

  This is FreeRDP version 2.7.0 (2.7.0)
  Build configuration: BUILD_TESTING=ON BUILTIN_CHANNELS=ON HAVE_AIO_H=1 HAVE_EXECINFO_H=1 HAVE_FCNTL_H=1 HAVE_GETLOGIN_R=1 HAVE_GETPWUID_R=1 HAVE_INTTYPES_H=1 HAVE_JOURNALD_H=TRUE HAVE_MATH_C99_LONG_DOUBLE=1 HAVE_POLL_H=1 HAVE_PTHREAD_MUTEX_TIMEDLOCK=ON HAVE_PTHREAD_MUTEX_TIMEDLOCK_LIBS= HAVE_PTHREAD_MUTEX_TIMEDLOCK_SYMBOL=1 HAVE_SYSLOG_H=1 HAVE_SYS_EVENTFD_H=1 HAVE_SYS_FILIO_H= HAVE_SYS_MODEM_H= HAVE_SYS_SELECT_H=1 HAVE_SYS_SOCKIO_H= HAVE_SYS_STRTIO_H= HAVE_SYS_TIMERFD_H=1 HAVE_TM_GMTOFF=1 HAVE_UNISTD_H=1 HAVE_XI_TOUCH_CLASS=1 WITH_ALSA=ON WITH_CAIRO=ON WITH_CCACHE=ON WITH_CHANNELS=ON WITH_CLANG_FORMAT=ON WITH_CLIENT=ON WITH_CLIENT_AVAILABLE=1 WITH_CLIENT_CHANNELS=ON WITH_CLIENT_CHANNELS_AVAILABLE=1 WITH_CLIENT_COMMON=ON WITH_CLIENT_INTERFACE=OFF WITH_CUNIT=ON WITH_CUPS=ON WITH_DEBUG_ALL=OFF WITH_DEBUG_CAPABILITIES=OFF WITH_DEBUG_CERTIFICATE=OFF WITH_DEBUG_CHANNELS=OFF WITH_DEBUG_CLIPRDR=OFF WITH_DEBUG_DVC=OFF WITH_DEBUG_KBD=OFF WITH_DEBUG_LICENSE=OFF WITH_DEBUG_MUTEX=OFF WITH_DEBUG_NEGO=OFF WITH_DEBUG_NLA=OFF WITH_DEBUG_NTLM=OFF WITH_DEBUG_RAIL=OFF WITH_DEBUG_RDP=OFF WITH_DEBUG_RDPDR=OFF WITH_DEBUG_RDPEI=OFF WITH_DEBUG_RDPGFX=OFF WITH_DEBUG_REDIR=OFF WITH_DEBUG_RFX=OFF WITH_DEBUG_RINGBUFFER=OFF WITH_DEBUG_SCARD=OFF WITH_DEBUG_SND=OFF WITH_DEBUG_SVC=OFF WITH_DEBUG_SYMBOLS=OFF WITH_DEBUG_THREADS=OFF WITH_DEBUG_TIMEZONE=OFF WITH_DEBUG_TRANSPORT=OFF WITH_DEBUG_TSG=OFF WITH_DEBUG_TSMF=OFF WITH_DEBUG_TSMF=OFF WITH_DEBUG_TSMF_AVAILABLE=0 WITH_DEBUG_URBDRC=OFF WITH_DEBUG_WND=OFF WITH_DEBUG_X11=OFF WITH_DEBUG_X11_CLIPRDR=OFF WITH_DEBUG_X11_LOCAL_MOVESIZE=OFF WITH_DEBUG_XV=OFF WITH_DSP_EXPERIMENTAL=OFF WITH_DSP_FFMPEG=OFF WITH_EVENTFD_READ_WRITE=1 WITH_FAAC=OFF WITH_FAAD2=OFF WITH_FFMPEG=TRUE WITH_FFMPEG=TRUE WITH_GFX_H264=ON WITH_GPROF=OFF WITH_GSM=OFF WITH_GSSAPI=OFF WITH_ICU=OFF WITH_IPP=OFF WITH_JPEG=ON WITH_LAME=OFF WITH_LIBRARY_VERSIONING=ON WITH_LIBSYSTEMD=ON WITH_MACAUDIO=OFF WITH_MACAUDIO=OFF WITH_MACAUDIO_AVAILABLE=0 WITH_MANPAGES=ON WITH_MBEDTLS=OFF WITH_OPENCL=OFF WITH_OPENH264=OFF WITH_OPENSLES=OFF WITH_OPENSSL=ON WITH_OSS=OFF WITH_PAM=ON WITH_PCSC=ON WITH_PROFILER=OFF WITH_PROXY=ON WITH_PROXY_MODULES=OFF WITH_PULSE=ON WITH_SAMPLE=OFF WITH_SANITIZE_ADDRESS=OFF WITH_SANITIZE_ADDRESS_AVAILABLE=1 WITH_SANITIZE_MEMORY=OFF WITH_SANITIZE_MEMORY_AVAILABLE=1 WITH_SANITIZE_THREAD=OFF WITH_SANITIZE_THREAD_AVAILABLE=1 WITH_SERVER=ON WITH_SERVER_CHANNELS=ON WITH_SERVER_INTERFACE=ON WITH_SHADOW=ON WITH_SMARTCARD_INSPECT=OFF WITH_SOXR=OFF WITH_SSE2=ON WITH_SWSCALE=OFF WITH_THIRD_PARTY=OFF WITH_VAAPI=ON WITH_VALGRIND_MEMCHECK=OFF WITH_VALGRIND_MEMCHECK_AVAILABLE=1 WITH_VERBOSE_WINPR_ASSERT=ON WITH_WAYLAND=ON WITH_WINPR_TOOLS=ON WITH_X11=ON WITH_XCURSOR=ON WITH_XDAMAGE=ON WITH_XEXT=ON WITH_XFIXES=ON WITH_XI=ON WITH_XINERAMA=ON WITH_XKBFILE=ON WITH_XRANDR=ON WITH_XRENDER=ON WITH_XSHM=ON WITH_XTEST=ON WITH_XV=ON WITH_ZLIB=ON
  Build type:          Release
  CFLAGS:               -fPIC -Wall -Wno-unused-result -Wno-unused-but-set-variable -Wno-deprecated-declarations -Wimplicit-function-declaration -Wredundant-decls -fno-omit-frame-pointer -DWINPR_DLL
  Compiler:            GNU, 11.3.0
  Target architecture: x64

• Connecting to xfreerdp on remote side
• log output

Environment (please complete the following information):

• OS: Linux
• Version/Distribution: NixOS 22.05
• Architecture: amd64

[hardening]

Some time ago I did some changes that were allowing the window size to be bigger than the actual physical screen and I hit the same bug. We shall definitely test the capabilities of the VAAPI driver before using it...

[akallabeth]

all for adding these checks, so who is willing to do it? (e.g. pull requests welcome) for the time being the feature is still marked experimental so better not use it in production