search

voc / voctomix

Full-HD Software Live-Video-Mixer in python
https://c3voc.de/
MIT License
580 stars 110 forks source link

[2.0-rc1] VAAPI preview rendering is broken #309

Closed Kunsi closed 6 months ago

Kunsi commented 10 months ago

voctomix starts up without any issues, the pipeline is showing vaapi as intended, just the preview outputs don't play anything. Non-VAAPI outputs (rawvideo) work fine. If i switch back to cpu encoding, everything works fine as well.

VAAPI itself works fine, i'm using this for encoding the streaming output. This issue is not related to that, because it occurs even with no other vaapi clients running.

voctocore config for previews:

[previews]
enabled=true
live=true
videodecoder=vaapi
videoencoder=vaapi
videocodec=jpeg
videocaps=image/jpeg

voctocore log:

Sat 2024-01-06 11:40:16 CET encoder96.lan.c3voc.de voctocore.py[5676]:     INFO AVPreviewOutput[cam1]: Incoming Connection from [::ffff:192.168.0.233]:45942 (fd=61)
Sat 2024-01-06 11:40:16 CET encoder96.lan.c3voc.de voctocore.py[5676]:     INFO AVPreviewOutput[cam1]: Now 1 Receiver(s) connected
Sat 2024-01-06 11:40:22 CET encoder96.lan.c3voc.de voctocore.py[5676]:     INFO AVPreviewOutput[cam1]: Now 0 Receiver connected

gstreamer packages:

ii  gir1.2-gst-plugins-base-1.0:amd64    1.22.0-3+deb12u1                                     amd64        GObject introspection data for the GStreamer Plugins Base library
ii  gir1.2-gstreamer-1.0:amd64           1.22.0-2                                             amd64        GObject introspection data for the GStreamer library
ii  gstreamer1.0-libav:amd64             1.22.0-2                                             amd64        ffmpeg plugin for GStreamer
ii  gstreamer1.0-plugins-bad:amd64       1.22.0-4+deb12u4                                     amd64        GStreamer plugins from the "bad" set
ii  gstreamer1.0-plugins-base:amd64      1.22.0-3+deb12u1                                     amd64        GStreamer plugins from the "base" set
ii  gstreamer1.0-plugins-good:amd64      1.22.0-5+deb12u1                                     amd64        GStreamer plugins from the "good" set
ii  gstreamer1.0-plugins-ugly:amd64      1.22.0-2+deb12u1                                     amd64        GStreamer plugins from the "ugly" set
ii  gstreamer1.0-tools                   1.22.0-2                                             amd64        Tools for use with GStreamer
ii  gstreamer1.0-vaapi:amd64             1.22.0-2                                             amd64        VA-API plugins for GStreamer
ii  gstreamer1.0-x:amd64                 1.22.0-3+deb12u1                                     amd64        GStreamer plugins for X11 and Pango
ii  libgstreamer-gl1.0-0:amd64           1.22.0-3+deb12u1                                     amd64        GStreamer GL libraries
ii  libgstreamer-plugins-bad1.0-0:amd64  1.22.0-4+deb12u4                                     amd64        GStreamer libraries from the "bad" set
ii  libgstreamer-plugins-base1.0-0:amd64 1.22.0-3+deb12u1                                     amd64        GStreamer libraries from the "base" set
ii  libgstreamer1.0-0:amd64              1.22.0-2                                             amd64        Core GStreamer libraries and elements

vainfo:

error: XDG_RUNTIME_DIR is invalid or not set in the environment.
error: can't connect to X server!
libva info: VA-API version 1.17.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_17
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.17 (libva 2.12.0)
vainfo: Driver version: Intel iHD driver for Intel(R) Gen Graphics - 23.1.1 ()
vainfo: Supported profile and entrypoints
      VAProfileNone                   : VAEntrypointVideoProc
      VAProfileNone                   : VAEntrypointStats
      VAProfileMPEG2Simple            : VAEntrypointVLD
      VAProfileMPEG2Simple            : VAEntrypointEncSlice
      VAProfileMPEG2Main              : VAEntrypointVLD
      VAProfileMPEG2Main              : VAEntrypointEncSlice
      VAProfileH264Main               : VAEntrypointVLD
      VAProfileH264Main               : VAEntrypointEncSlice
      VAProfileH264Main               : VAEntrypointFEI
      VAProfileH264Main               : VAEntrypointEncSliceLP
      VAProfileH264High               : VAEntrypointVLD
      VAProfileH264High               : VAEntrypointEncSlice
      VAProfileH264High               : VAEntrypointFEI
      VAProfileH264High               : VAEntrypointEncSliceLP
      VAProfileVC1Simple              : VAEntrypointVLD
      VAProfileVC1Main                : VAEntrypointVLD
      VAProfileVC1Advanced            : VAEntrypointVLD
      VAProfileJPEGBaseline           : VAEntrypointVLD
      VAProfileJPEGBaseline           : VAEntrypointEncPicture
      VAProfileH264ConstrainedBaseline: VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
      VAProfileH264ConstrainedBaseline: VAEntrypointFEI
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSliceLP
      VAProfileVP8Version0_3          : VAEntrypointVLD
      VAProfileVP8Version0_3          : VAEntrypointEncSlice
      VAProfileHEVCMain               : VAEntrypointVLD
      VAProfileHEVCMain               : VAEntrypointEncSlice
      VAProfileHEVCMain               : VAEntrypointFEI
      VAProfileHEVCMain10             : VAEntrypointVLD
      VAProfileHEVCMain10             : VAEntrypointEncSlice
      VAProfileVP9Profile0            : VAEntrypointVLD
      VAProfileVP9Profile2            : VAEntrypointVLD
sophieschi commented 6 months ago

Do previews have to be jpeg?

The following config snippet with h264 seems to be working correctly:

[previews]
enabled=true
live=true
videodecoder=vaapi
videoencoder=vaapi
videocodec=h264
videocaps=video/x-h264,width=1024,height=576,framerate=25/1
Kunsi commented 6 months ago

I don't think they have to.

voctogui is throwing some weird errors for me on my arch system, gonna test and report back once i have a proper test system.

Kunsi commented 6 months ago 
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de systemd[1]: Started voctomix2-voctogui.service - voctomix2 gui.
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de voctogui.py[9026]:     INFO root: Python Version: sys.version_info(major=3, minor=11, micro=2, releaselevel='final', serial=0)
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de voctogui.py[9026]:     INFO root: GStreamer Version: (major=1, minor=22, micro=0, nano=0)
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de voctogui.py[9026]:     INFO Connection: establishing Connection to 10.73.96.3
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de voctogui.py[9026]:     INFO Connection: Connection to host 10.73.96.3 at port 9999 successful
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de voctogui.py[9026]:     INFO VoctoguiConfigParser: reading server-config
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de voctogui.py[9026]:     INFO Connection: reading server-config
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de voctogui.py[9026]:     INFO VoctoguiConfigParser: merging server-config {'DEFAULT': {}, 'mix': {'videocaps': 'video/x-raw,format=I420,width=1920,height=1080,framerate=25/1,pixel-aspect-ratio=1/1,interlace-mode=progressive,colorimetry=bt709', 'audiocaps': 'audio/x-raw,format=S16LE,channels=8,layout=interleaved,rate=48000', 'sources': 'cam1,slides', 'livesources': 'slides'}, 'source.cam1': {'kind': 'decklink', 'devicenumber': '0', 'video_connection': 'SDI', 'video_mode': '1080p25', 'volume': '1.0', 'audio.original': '0+1'}, 'source.slides': {'kind': 'test', 'volume': '0.0'}, 'source.background': {'kind': 'img', 'file': '/opt/voc/share/bg.png'}, 'source.break': {'kind': 'file', 'location': '/opt/voc/share/pause.ts'}, 'previews': {'enabled': 'true', 'live': 'true', 'videodecoder': 'vaapi', 'videoencoder': 'vaapi', 'videocodec': 'h264', 'videocaps': 'video/x-h264,width=1024,height=576,framerate=25/1'}, 'blinder': {'enabled': 'true', 'videos': 'break'}, 'source.blinder': {'audio.original': '0+1', 'kind': 'file', 'location': '/opt/voc/share/pause-music.mp3'}, 'overlay': {'path': '/opt/voc/overlays', 'files': '', 'user-auto-off': 'true', 'auto-off': 'true', 'blend-time': '300'}, 'mirrors': {'enabled': 'true'}, 'composites': {'fs.a': '*', 'fs.b': '*', 'fs.alpha-b': '0', 'fs.noswap': 'true', 'fs-pip.a': '*', 'fs-pip.b': '0.86/0.85 0.0', 'fs-pip.alpha-b': '0', 'fs-pip.inter': 'true', 'fs-pip.mirror': 'true', 'fs-sbs.a': '*', 'fs-sbs.b': '1.0/0.5 0.0', 'fs-sbs.alpha-b': '0', 'fs-sbs.inter': 'true', 'fs-lec.a': '*', 'fs-lec.b': '1.0 0.0', 'fs-lec.alpha-b': '0', 'fs-lec.crop-b': '0.31/0', 'fs-lec.inter': 'true', 'fs-lec.mirror': 'true', 'pip.a': '*', 'pip.b': '0.73/0.72 0.26', 'pip.noswap': 'true', 'pip.mirror': 'true', 'sbs.a': '0.008/0.08 0.49', 'sbs.b': '0.503/0.42 0.49', 'lec.a': '0.006/0.01 0.75', 'lec.b': '0.60/0.42 0.56', 'lec.crop-b': '0.31/0', 'lec.mirror': 'true', 'lec_43.a': '-0.125/0.0 1.0', 'lec_43.b': '0.60/0.42 0.56', 'lec_43.crop-a': '0.125/0', 'lec_43.crop-b': '0.31/0', 'lec_43.mirror': 'true', 'fs-b.a': '*', 'fs-b.b': '*', 'fs-b.noswap': 'true', 'fs-b-sbs.a': '0.0/0.5 0.0', 'fs-b-sbs.alpha-a': '0.0', 'fs-b-sbs.b': '*', 'fs-b-sbs.inter': 'true', 'fs-b-lec.a': '0.0/1.0 0.0', 'fs-b-lec.b': '*', 'fs-b-lec.inter': 'true', 'fs-b-lec.mirror': 'true', 'oao.a': '0.3/0.2 0.4', 'oao.alpha-a': '0.5', 'oao.b': '0.2/0.3 0.6', 'oao.inter': 'true', 'oao.noswap': 'true'}, 'transitions': {'fs-fs': '750, fs / fs-b', 'fs-pip': '750, fs-pip / pip', 'fs-sbs': '750, fs-sbs / sbs', 'fs-b-pip': '750, fs-b / pip', 'fs-b-sbs': '750, fs-b-sbs / sbs', 'fs-lec': '750, fs-lec / lec', 'fs-b-lec': '750, fs-b-lec / lec', 'fs-lec_43': '750, fs-lec / lec_43', 'fs-b-lec_43': '750, fs-b-lec / lec_43', 'pip-pip': '750, pip / sbs / pip', 'sbs-sbs': '750, sbs / oao / sbs', '_sbs-sbs': '750, ^sbs / ^oao / sbs', 'fs-pip_': '750, |fs-pip / |pip', 'fs-b-pip_': '750, fs-b / |pip', 'fs-lec_': '750, fs-lec / |lec', 'fs-lec_43_': '750, fs-lec / |lec_43', 'fs-b-lec_': '750, fs-b-lec / |lec', 'fs-b-lec_43_': '750, fs-b-lec / |lec_43', 'pip-pip_': '750, |pip / sbs / |pip', 'def': '750, * / *'}, 'toolbar.sources.a': {'buttons': 'cam1,slides', 'cam1.name': 'cam1', 'cam1.key': 'F1', 'cam1.tip': 'Select source cam1 on channel A', 'slides.name': 'slides', 'slides.key': 'F4', 'slides.tip': 'Select source slides on channel A'}, 'toolbar.sources.b': {'buttons': 'cam1,slides', 'cam1.name': 'cam1', 'cam1.key': '1', 'cam1.tip': 'Select source cam1 on channel B', 'slides.name': 'slides', 'slides.key': '4', 'slides.tip': 'Select source slides on channel B'}, 'toolbar.composites': {'buttons': 'fs,sbs,lec', 'fs.name': 'FULL SCREEN', 'fs.key': 'F5', 'fs.tip': 'Show channel A on full screen', 'sbs.name': 'SIDE BY SIDE', 'sbs.key': 'F6', 'sbs.tip': 'Put channel A beside channel B', 'lec.name': 'LECTURE', 'lec.key': 'F7', 'lec.tip': 'Put cropped channel B beside large channel A'}, 'toolbar.mods': {'buttons': 'mirror,ratio', 'mirror.name': 'MIRROR', 'mirror.key': 'F9', 'mirror.replace': 'lec->|lec', 'mirror.tip': 'Horizontally mirror composite\\n(e.g. when speaker moves to the other side)', 'ratio.name': '4:3', 'ratio.replace': 'lec->lec_43', 'ratio.key': 'F10', 'ratio.tip': 'Crop channel A to 4:3 ratio'}, 'toolbar.mix': {'buttons': 'retake,cut,trans', 'retake.name': 'RETAKE', 'retake.key': 'BackSpace', 'retake.tip': 'Copy output composite to preview for modification\\n(output remains untouched)', 'cut.name': 'CUT', 'cut.key': 'Return', 'cut.tip': 'Hard cut preview composite to output.', 'cut.expand': 'True', 'trans.name': 'TRANS', 'trans.key': 'space', 'trans.tip': 'Use transition to cut preview composite to output', 'trans.expand': 'True'}, 'toolbar.insert': {'auto-off.name': 'AUTO-OFF', 'auto-off.key': 'o', 'auto-off.tip': 'automatically turn off insertion before every mix', 'update.name': 'UPDATE', 'update.key': 'u', 'update.tip': 'Update current event', 'insert.name': 'INSERT', 'insert.key': 'i', 'insert.tip': 'Show or hide current insertion'}}
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de voctogui.py[9026]:     INFO Clock: successfully synced NetClientClock to host
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de voctogui.py[9026]:     INFO Voctogui: Loading ui-file from file /opt/voctomix2/release/voctogui/ui/voctogui.ui
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de python3[9026]: Could not load image 'logo.svg': Couldn’t recognize the image file format for file “/opt/voctomix2/release/voctogui/ui/logo.svg”
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de python3[9026]: Could not load image 'ports.svg': Couldn’t recognize the image file format for file “/opt/voctomix2/release/voctogui/ui/ports.svg”
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de python3[9026]: Could not load image 'queues.svg': Couldn’t recognize the image file format for file “/opt/voctomix2/release/voctogui/ui/queues.svg”
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de python3[9026]: Could not load image 'voc2.svg': Couldn’t recognize the image file format for file “/opt/voctomix2/release/voctogui/ui/voc2.svg”
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de python3[9026]: Could not load image 'voc.svg': Couldn’t recognize the image file format for file “/opt/voctomix2/release/voctogui/ui/voc.svg”
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de voctogui.py[9026]:     INFO Voctogui: Loading css-file from file /opt/voctomix2/release/voctogui/ui/voctogui.css
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de voctogui.py[9026]:     INFO Ui: Initializing Ui
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de voctogui.py[9026]:     INFO VideoPreviewsController: Initializing video preview cam1 at port 13100
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de python3[9026]: Could not load image 'headphones.svg': Couldn’t recognize the image file format for file “/opt/voctomix2/release/voctogui/ui/headphones.svg”
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de python3[9026]: Could not load image 'mute.svg': Couldn’t recognize the image file format for file “/opt/voctomix2/release/voctogui/ui/mute.svg”
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de voctogui.py[9026]:     INFO VideoDisplay:CAM1: using encoded previews instead of raw-video
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de voctogui.py[9026]:     INFO VideoDisplay:CAM1: Creating Display-Pipeline:
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de voctogui.py[9026]:     
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de voctogui.py[9026]:     tcpclientsrc
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de voctogui.py[9026]:         name=tcpsrc-CAM1
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de voctogui.py[9026]:         host=10.73.96.3
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de voctogui.py[9026]:         port=13100
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de voctogui.py[9026]:         blocksize=1048576
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de voctogui.py[9026]:     ! matroskademux
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de voctogui.py[9026]:         name=demux-CAM1
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de voctogui.py[9026]:     
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de voctogui.py[9026]:     demux-CAM1.
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de voctogui.py[9026]:     ! queue
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de voctogui.py[9026]:         name=queue-video-CAM1
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de voctogui.py[9026]:     ! vaapih264dec
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de voctogui.py[9026]:     ! videoconvert
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de voctogui.py[9026]:     ! videoscale
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de voctogui.py[9026]:     ! textoverlay
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de voctogui.py[9026]:         name=title-CAM1
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de voctogui.py[9026]:         text="CAM1"
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de voctogui.py[9026]:         valignment=bottom
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de voctogui.py[9026]:         halignment=center
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de voctogui.py[9026]:         shaded-background=yes
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de voctogui.py[9026]:         font-desc="Roboto, 22"
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de voctogui.py[9026]:     ! xvimagesink
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de voctogui.py[9026]:         name=imagesink-CAM1
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de voctogui.py[9026]:     
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de voctogui.py[9026]:     demux-CAM1.
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de voctogui.py[9026]:     ! queue
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de voctogui.py[9026]:         name=queue-audio-CAM1
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de voctogui.py[9026]:     ! level
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de voctogui.py[9026]:         name=lvl
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de voctogui.py[9026]:         interval=50000000
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de voctogui.py[9026]:     ! audioconvert
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de voctogui.py[9026]:     ! fakesink
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de voctogui.py[9026]:     
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de voctogui.py[9026]:    ERROR Gst: : no element "vaapih264dec" (in function gst_parse_element_make() in file gst/parse/grammar.y:570)
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de voctogui.py[9026]:    ERROR Gst: : link has no sink [source=@0x1dcc020] (in function priv_gst_parse_yyparse() in file gst/parse/grammar.y:1262)
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de voctogui.py[9026]:    ERROR Gst: : link has no source [sink=@0x1dd7830] (in function priv_gst_parse_yyparse() in file gst/parse/grammar.y:1262)
Fri 2024-04-12 09:06:40 CEST mixer96.lan.c3voc.de voctogui.py[9026]:    ERROR VideoDisplay:CAM1: Can not launch pipeline
Fri 2024-04-12 09:06:41 CEST mixer96.lan.c3voc.de systemd[1]: voctomix2-voctogui.service: Main process exited, code=exited, status=255/EXCEPTION
Fri 2024-04-12 09:06:41 CEST mixer96.lan.c3voc.de systemd[1]: voctomix2-voctogui.service: Failed with result 'exit-code'.

Seems the code assumes that the client is vaapi capable, if the server is vaapi capable?

Kunsi commented 6 months ago

... yes. If i install the vaapi packages on the mixer laptop, everything works as intended.