Hardware Encoding not working (VAAPI)

Expected Behavior

Hardware accelerated transcoding

Current Behavior

Playback won't transcode unless I untick hardware encoding. Hardware DEcoding alone works fine. If I copy and paste the ffmpeg command from the log into my host server then it runs and transcodes the file. Maybe the issue is with the version of ffmpeg included with jellyfin?

Client displays message "Playback Error This client isn't compatible with the media and the server isn't sending a compatible media format."

Steps to Reproduce

Play a file that requires transcoding with VAAPI on and hardware encoding on. Hardware is a 5600G cpu.

Environment

OS: Ubuntu 20.04 LTS (docker host) CPU architecture: x86_64 How docker service was installed: apt

Command used to create docker container (run/create/compose/screenshot)

docker run -d --name=jellyfin -e PUID=1000 -e PGID=1000 -e TZ=Europe/London -p 8096:8096 -p 8920:8920 #optional -p 7359:7359/udp #optional --volume jellyfin-config:/config --volume /transcodes:/transcodes --volume /storage/video:/media --device /dev/dri:/dev/dri --restart unless-stopped ghcr.io/linuxserver/jellyfin

Docker logs

/videos/c4b93074-5153-6f1c-179e-a4a4a0d92fc6/hls1/main/0.ts

{"Protocol":0,"Id":"c4b9307451536f1c179ea4a4a0d92fc6","Path":"/media/x.mp4","EncoderPath":null,"EncoderProtocol":null,"Type":0,"Container":"mov,mp4,m4a,3gp,3g2,mj2","Size":289023806,"Name":"x","IsRemote":false,"ETag":"f081eeba53fb771720e9c0ee1bc00756","RunTimeTicks":17620700000,"ReadAtNativeFramerate":false,"IgnoreDts":false,"IgnoreIndex":false,"GenPtsInput":false,"SupportsTranscoding":true,"SupportsDirectStream":true,"SupportsDirectPlay":true,"IsInfiniteStream":false,"RequiresOpening":false,"OpenToken":null,"RequiresClosing":false,"LiveStreamId":null,"BufferMs":null,"RequiresLooping":false,"SupportsProbing":true,"VideoType":0,"IsoType":null,"Video3DFormat":null,"MediaStreams":[{"Codec":"hevc","CodecTag":"hvc1","Language":"und","ColorRange":null,"ColorSpace":"bt709","ColorTransfer":"bt709","ColorPrimaries":"bt709","Comment":null,"TimeBase":"1/90000","CodecTimeBase":null,"Title":null,"VideoRange":"SDR","localizedUndefined":null,"localizedDefault":null,"localizedForced":null,"DisplayTitle":"720p HEVC SDR","NalLengthSize":null,"IsInterlaced":false,"IsAVC":null,"ChannelLayout":null,"BitRate":1173985,"BitDepth":null,"RefFrames":1,"PacketLength":null,"Channels":null,"SampleRate":null,"IsDefault":true,"IsForced":false,"Height":720,"Width":1280,"AverageFrameRate":49.896996,"RealFrameRate":200,"Profile":"Main","Type":1,"AspectRatio":"16:9","Index":0,"Score":null,"IsExternal":false,"DeliveryMethod":null,"DeliveryUrl":null,"IsExternalUrl":null,"IsTextSubtitleStream":false,"SupportsExternalStream":false,"Path":null,"PixelFormat":"yuv420p","Level":120,"IsAnamorphic":null},{"Codec":"aac","CodecTag":"mp4a","Language":"eng","ColorRange":null,"ColorSpace":null,"ColorTransfer":null,"ColorPrimaries":null,"Comment":null,"TimeBase":"1/48000","CodecTimeBase":null,"Title":null,"VideoRange":null,"localizedUndefined":null,"localizedDefault":null,"localizedForced":null,"DisplayTitle":"English - AAC - Stereo - Default","NalLengthSize":null,"IsInterlaced":false,"IsAVC":null,"ChannelLayout":"stereo","BitRate":128001,"BitDepth":null,"RefFrames":null,"PacketLength":null,"Channels":2,"SampleRate":48000,"IsDefault":true,"IsForced":false,"Height":null,"Width":null,"AverageFrameRate":null,"RealFrameRate":null,"Profile":"LC","Type":0,"AspectRatio":null,"Index":1,"Score":null,"IsExternal":false,"DeliveryMethod":null,"DeliveryUrl":null,"IsExternalUrl":null,"IsTextSubtitleStream":false,"SupportsExternalStream":false,"Path":null,"PixelFormat":null,"Level":0,"IsAnamorphic":null},{"Codec":"mov_text","CodecTag":"tx3g","Language":"und","ColorRange":null,"ColorSpace":null,"ColorTransfer":null,"ColorPrimaries":null,"Comment":null,"TimeBase":"1/90000","CodecTimeBase":null,"Title":null,"VideoRange":null,"localizedUndefined":"Undefined","localizedDefault":"Default","localizedForced":"Forced","DisplayTitle":"Und - Default","NalLengthSize":null,"IsInterlaced":false,"IsAVC":null,"ChannelLayout":null,"BitRate":136,"BitDepth":null,"RefFrames":null,"PacketLength":null,"Channels":null,"SampleRate":null,"IsDefault":true,"IsForced":false,"Height":null,"Width":null,"AverageFrameRate":null,"RealFrameRate":null,"Profile":null,"Type":2,"AspectRatio":null,"Index":2,"Score":null,"IsExternal":false,"DeliveryMethod":null,"DeliveryUrl":null,"IsExternalUrl":null,"IsTextSubtitleStream":true,"SupportsExternalStream":true,"Path":null,"PixelFormat":null,"Level":0,"IsAnamorphic":null}],"MediaAttachments":[],"Formats":[],"Bitrate":1312201,"Timestamp":null,"RequiredHttpHeaders":{},"TranscodingUrl":null,"TranscodingSubProtocol":null,"TranscodingContainer":null,"AnalyzeDurationMs":null,"DefaultAudioStreamIndex":null,"DefaultSubtitleStreamIndex":null}

/usr/lib/jellyfin-ffmpeg/ffmpeg -hwaccel vaapi -hwaccel_output_format vaapi -vaapi_device /dev/dri/renderD128 -i file:"/media/x.mp4" -map_metadata -1 -map_chapters -1 -threads 0 -map 0:0 -map 0:1 -map -0:s -codec:v:0 h264_vaapi -b:v 5869924 -maxrate 5869924 -bufsize 11739848 -profile:v:0 high -level 41  -force_key_frames:0 "expr:gte(t,0+n_forced*3)" -vf "format=nv12|vaapi,hwupload,scale_vaapi=format=nv12" -start_at_zero -vsync -1 -codec:a:0 aac -ac 2 -ab 128001 -copyts -avoid_negative_ts disabled -max_muxing_queue_size 2048 -f hls -max_delay 5000000 -hls_time 3 -hls_segment_type mpegts -start_number 0 -hls_segment_filename "/transcodes/35c1760f1a88e81d5842c2d05ab52939%d.ts" -hls_playlist_type vod -hls_list_size 0 -y "/transcodes/35c1760f1a88e81d5842c2d05ab52939.m3u8"

ffmpeg version 4.4.1-Jellyfin Copyright (c) 2000-2021 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-17ubuntu1~20.04)
  configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-shared --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-static --enable-gmp --enable-gnutls --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libdav1d --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --arch=amd64 --enable-opencl --enable-vaapi --enable-amf --enable-libmfx --enable-vdpau --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
  libavutil      56. 70.100 / 56. 70.100
  libavcodec     58.134.100 / 58.134.100
  libavformat    58. 76.100 / 58. 76.100
  libavdevice    58. 13.100 / 58. 13.100
  libavfilter     7.110.100 /  7.110.100
  libswscale      5.  9.100 /  5.  9.100
  libswresample   3.  9.100 /  3.  9.100
  libpostproc    55.  9.100 / 55.  9.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'file:/media/x.mp4':
  Metadata:
x
  Duration: 00:29:22.07, start: 0.000000, bitrate: 1312 kb/s
  Stream #0:0(und): Video: hevc (Main) (hvc1 / 0x31637668), yuv420p(tv, bt709), 1280x720 [SAR 1:1 DAR 16:9], 1173 kb/s, 49.90 fps, 200 tbr, 90k tbn, 50 tbc (default)
    Metadata:
      creation_time   : 2021-02-06T19:51:11.000000Z
      handler_name    : VideoHandler
      vendor_id       : [0][0][0][0]
  Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 128 kb/s (default)
    Metadata:
      creation_time   : 2021-02-06T19:51:11.000000Z
      handler_name    : Stereo
      vendor_id       : [0][0][0][0]
  Stream #0:2(und): Subtitle: mov_text (tx3g / 0x67337874), 1280x108, 0 kb/s (default)
    Metadata:
      creation_time   : 2021-02-06T19:51:11.000000Z
      handler_name    : SubtitleHandler
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (native) -> h264 (h264_vaapi))
  Stream #0:1 -> #0:1 (aac (native) -> aac (native))
Press [q] to stop, [?] for help
[h264_vaapi @ 0x559737a9e200] Driver does not support some wanted packed headers (wanted 0xd, found 0x1).
Output #0, hls, to '/transcodes/35c1760f1a88e81d5842c2d05ab52939.m3u8':
  Metadata:
    encoder         : Lavf58.76.100
  Stream #0:0: Video: h264 (High), vaapi_vld(tv, bt709, progressive), 1280x720 [SAR 1:1 DAR 16:9], q=2-31, 5869 kb/s, 200 fps, 90k tbn (default)
    Metadata:
      encoder         : Lavc58.134.100 h264_vaapi
  Stream #0:1: Audio: aac (LC), 48000 Hz, stereo, fltp, 128 kb/s (default)
    Metadata:
      encoder         : Lavc58.134.100 aac
frame=    1 fps=0.0 q=0.0 size=N/A time=00:00:00.00 bitrate=N/A speed=   0x    
[mpegts @ 0x5597383d1540] H.264 bitstream malformed, no startcode found, use the video bitstream filter 'h264_mp4toannexb' to fix it ('-bsf:v h264_mp4toannexb' option with ffmpeg)
av_interleaved_write_frame(): Invalid data found when processing input
[mpegts @ 0x5597383d1540] H.264 bitstream malformed, no startcode found, use the video bitstream filter 'h264_mp4toannexb' to fix it ('-bsf:v h264_mp4toannexb' option with ffmpeg)
av_interleaved_write_frame(): Invalid data found when processing input
[mpegts @ 0x5597383d1540] H.264 bitstream malformed, no startcode found, use the video bitstream filter 'h264_mp4toannexb' to fix it ('-bsf:v h264_mp4toannexb' option with ffmpeg)
av_interleaved_write_frame(): Invalid data found when processing input
[hls @ 0x559737acfcc0] Opening '/transcodes/35c1760f1a88e81d5842c2d05ab529390.ts' for writing
frame=    8 fps=0.0 q=-0.0 Lsize=N/A time=00:00:00.06 bitrate=N/A dup=6 drop=0 speed=1.22x    
video:0kB audio:6kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
[aac @ 0x559737a7c040] Qavg: 8367.418
Conversion failed!

linuxserver / docker-jellyfin