Closed DemiMarie closed 3 years ago
My webcam doesn't support I420
either. What happens is in the background (when my webcam is configured to use MJPG
with set-webcam-format.sh
) is that GStreamer (or maybe V4L2?) automatically converts the MJPG
format to I420
which is a YUV raw video format.
Here's an official piece of documentation from FOURCC (the group that creates and helps standardize a lot of these video formats) about all the different YUV formats: https://www.fourcc.org/yuv.php
I already plan to create an issue on switching to MJPEG for streaming video from the webcam. There are many reasons for this (you have to understand a bit about how webcams work) and based on a bit of experimenting I've done, I think it can be done in a reasonably secure way with GStreamer. More on this later.
Your webcam almost certainly supports a YUV raw video format of some type though (as does mine).
By the way, could I please have the output of v4l2-ctl --device /dev/videoX --list-formats-ext
for your webcam as well as the make and model? Thank you.
The output of v4l2-ctl --device 0 --list-formats-ext
in sys-usb
:
ioctl: VIDIOC_ENUM_FMT
Type: Video Capture
[0]: 'YUYV' (YUYV 4:2:2)
Size: Discrete 640x480
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Size: Discrete 320x180
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Size: Discrete 320x240
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Size: Discrete 352x288
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Size: Discrete 424x240
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Size: Discrete 640x360
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Size: Discrete 848x480
Interval: Discrete 0.050s (20.000 fps)
Size: Discrete 960x540
Interval: Discrete 0.067s (15.000 fps)
Size: Discrete 1280x720
Interval: Discrete 0.100s (10.000 fps)
[1]: 'MJPG' (Motion-JPEG, compressed)
Size: Discrete 640x480
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Size: Discrete 320x180
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Size: Discrete 320x240
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Size: Discrete 352x288
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Size: Discrete 424x240
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Size: Discrete 640x360
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Size: Discrete 848x480
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Size: Discrete 960x540
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Size: Discrete 1280x720
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.067s (15.000 fps)
The webcam seems to be a SunplusIT Inc Integrated Camera
on a Lenovo P51.
The following qvc.Webcam
service in sys-usb
works when combined with a modified receiver script:
#!/usr/bin/python3 --
"""exec" /usr/bin/python3 -- "$0" "$@"
exit 127
"""
import struct
import os
import sys
import subprocess
def main(argv):
width, height, frame_rate = 1280, 720, 30
sys.stdout.buffer.write(struct.pack('=HHH', width, height, frame_rate))
sys.stdout.buffer.flush()
print('Starting webcam stream at {}x{} {} FPS...'.format(
width, height, frame_rate), file=sys.stderr)
sys.stderr.flush()
subprocess.run(('sudo', 'modprobe', 'uvcvideo'), check=True)
res = subprocess.run(executable='/usr/bin/gst-launch-1.0', args=(
'gst-launch-1.0',
'--quiet',
'v4l2src',
'!',
'queue',
'!',
'image/jpeg,'
'width={},'
'height={},'
'framerate={}/1'.format(width, height, frame_rate),
'!',
'jpegdec',
'!',
'fdsink',
), stdin=subprocess.DEVNULL)
sys.exit(res.returncode)
if __name__ == '__main__':
main(sys.argv)
And the receiver script:
#!/usr/bin/python3 --
# Copyright (C) 2021 Elliot Killick <elliotkillick@zohomail.eu>
# Copyright (C) 2021 Demi Marie Obenour <demi@invisiblethingslab.com>
# Licensed under the MIT License. See LICENSE file for details.
import struct
import os
import sys
def main(argv):
if len(argv) != 1:
raise RuntimeError('should not have any arguments')
s = struct.Struct('=HHH')
if s.size != 6:
raise AssertionError('bug')
untrusted_input = os.read(0, 6)
if len(untrusted_input) != 6:
raise RuntimeError('wrong number of bytes read')
untrusted_width, untrusted_height, untrusted_fps = s.unpack(untrusted_input)
del untrusted_input
if untrusted_width > 4096 or untrusted_height > 4096 or untrusted_fps > 4096:
raise RuntimeError('excessive width, height, and/or fps')
width, height, fps = untrusted_width, untrusted_height, untrusted_fps
del untrusted_width, untrusted_height, untrusted_fps
print('Receiving video stream at {}x{} {} FPS…'.format(width, height, fps),
file=sys.stderr)
os.execv('/usr/bin/gst-launch-1.0', (
'gst-launch-1.0',
'fdsrc',
'!',
'capsfilter',
'caps=video/x-raw,'
'width={},'
'height={},'
'framerate={}/1,'
'format=I420,'
'colorimetry=2:4:7:1,'
'chroma-site=none,'
'interlace-mode=progressive,'
'pixel-aspect-ratio=1/1,'
'max-framerate={}/1,'
'views=1'.format(width, height, fps, fps),
'!',
'rawvideoparse',
'use-sink-caps=true',
'!',
'v4l2sink',
'device=/dev/video0',
'sync=false',
))
if __name__ == '__main__':
main(sys.argv)
For me (with the GStreamer in dom0
) it seems to automatically convert the webcam input from MJPEG to YUV without ajpegdec
. If I remove format=I420
from the qvc.Webcam
sender it still works for me just as it did with format=I420
.
For me (with the GStreamer in
dom0
) it seems to automatically convert the webcam input from MJPEG to YUV without ajpegdec
. If I removeformat=I420
from theqvc.Webcam
sender it still works for me just as it did withformat=I420
.
It does not work for me, perhaps because my webcam does not support I420 video, only YUYV and MJPEG. Trying to convert from YUYV to I420 caused errors and assertion failures in GStreamer. Furthermore, video quality is lower in YUYV mode than in MJPEG mode, so I would prefer to use MJPEG mode anyway.
Below is the sender I use currently. It is exposed as the qvc.Webcam2
service directly, with no intervening shell scripts. It should work with the receiver script above. The sudo modprobe uvcvideo
line is because I have my system configured to prevent that module from autoloading. You will almost certainly not need it.
#!/usr/bin/python3 --
"""exec" /usr/bin/python3 -- "$0" "$@"
exit 127
"""
import struct
import os
os.environ['G_DEBUG'] = 'fatal-criticals'
import sys
import subprocess
import gi
gi.require_version('Gtk', '3.0')
gi.require_version('Gst', '1.0')
from gi.repository import Gtk, Gst
def main(argv):
width, height, frame_rate = 1280, 720, 30
sys.stdout.buffer.write(struct.pack('=HHH', width, height, frame_rate))
sys.stdout.buffer.flush()
print('Starting webcam stream at {}x{} {} FPS...'.format(
width, height, frame_rate), file=sys.stderr)
sys.stderr.flush()
subprocess.run(('sudo', 'modprobe', 'uvcvideo'), check=True)
Gst.init()
element = Gst.parse_launchv((
'v4l2src',
'!',
'queue',
'!',
'capsfilter',
'caps=image/jpeg,'
'width={},'
'height={},'
'framerate={}/1,'
'format=I420,'
'colorimetry=2:4:7:1,'
'chroma-site=none,'
'interlace-mode=progressive,'
'pixel-aspect-ratio=1/1,'
'max-framerate={}/1,'
'views=1'.format(width, height, frame_rate, frame_rate),
'!',
'jpegdec',
'!',
'capsfilter',
'caps=video/x-raw,'
'width={},'
'height={},'
'framerate={}/1,'
'format=I420,'
'interlace-mode=progressive,'
'pixel-aspect-ratio=1/1,'
'max-framerate={}/1,'
'views=1'.format(width, height, frame_rate, frame_rate),
'!',
'fdsink',
))
def msg_handler(bus, msg):
if msg.type == Gst.MessageType.EOS:
print('<6>End of stream, exiting', file=sys.stderr)
elif msg.type == Gst.MessageType.ERROR:
print('<3>Fatal error:', msg.parse_error(), file=sys.stderr)
elif msg.type == Gst.MessageType.CLOCK_LOST:
print('<6>Clock lost, resetting', file=sys.stderr)
element.set_state(Gst.State.PAUSED)
element.set_state(Gst.State.PLAYING)
return
else:
return # FIXME!
element.set_state(Gst.State.NULL)
Gtk.main_quit()
bus = element.get_bus()
bus.add_signal_watch()
bus.connect('message', msg_handler)
element.set_state(Gst.State.PLAYING)
Gtk.main()
if __name__ == '__main__':
main(sys.argv)
The second capsfilter
element is redundant, but I included it to ensure that if jpegdec
produced its output in the wrong format, I would get a useful error and not silently corrupted video. Since GStreamer is run in-process, the UI could also be part of the above script.
It is also worth noting that since the frontend qube cannot attack the webcam, the webcam’s own indicator light also serves as an unforgable and unpreventable indication that recording is taking place, even without the UI.
Hmm, alright well your way (with jpegdec
explicitly specified) works just as well for me too so I guess we can make that change.
It is also worth noting that since the frontend qube cannot attack the webcam, the webcam’s own indicator light also serves as an unforgable and unpreventable indication that recording is taking place, even without the UI.
Yes, as noted in the README
, the UI is more for the screen sharing component because there is no visual light indicator there. Also, I don't think all webcams have an indicator light (although I think the vast majority do).
perhaps because my webcam does not support I420 video
As mentioned prior, my webcam also doesn't support I420
but it is converted to I420
from MJPG
. Here are the formats supported on my Logitech C922 Pro Stream Webcam:
ioctl: VIDIOC_ENUM_FMT
Index : 0
Type : Video Capture
Pixel Format: 'YUYV'
Name : YUYV 4:2:2
Size: Discrete 640x480
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.042s (24.000 fps)
Interval: Discrete 0.050s (20.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.133s (7.500 fps)
Interval: Discrete 0.200s (5.000 fps)
Size: Discrete 160x90
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.042s (24.000 fps)
Interval: Discrete 0.050s (20.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.133s (7.500 fps)
Interval: Discrete 0.200s (5.000 fps)
Size: Discrete 160x120
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.042s (24.000 fps)
Interval: Discrete 0.050s (20.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.133s (7.500 fps)
Interval: Discrete 0.200s (5.000 fps)
Size: Discrete 176x144
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.042s (24.000 fps)
Interval: Discrete 0.050s (20.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.133s (7.500 fps)
Interval: Discrete 0.200s (5.000 fps)
Size: Discrete 320x180
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.042s (24.000 fps)
Interval: Discrete 0.050s (20.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.133s (7.500 fps)
Interval: Discrete 0.200s (5.000 fps)
Size: Discrete 320x240
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.042s (24.000 fps)
Interval: Discrete 0.050s (20.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.133s (7.500 fps)
Interval: Discrete 0.200s (5.000 fps)
Size: Discrete 352x288
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.042s (24.000 fps)
Interval: Discrete 0.050s (20.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.133s (7.500 fps)
Interval: Discrete 0.200s (5.000 fps)
Size: Discrete 432x240
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.042s (24.000 fps)
Interval: Discrete 0.050s (20.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.133s (7.500 fps)
Interval: Discrete 0.200s (5.000 fps)
Size: Discrete 640x360
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.042s (24.000 fps)
Interval: Discrete 0.050s (20.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.133s (7.500 fps)
Interval: Discrete 0.200s (5.000 fps)
Size: Discrete 800x448
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.042s (24.000 fps)
Interval: Discrete 0.050s (20.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.133s (7.500 fps)
Interval: Discrete 0.200s (5.000 fps)
Size: Discrete 800x600
Interval: Discrete 0.042s (24.000 fps)
Interval: Discrete 0.050s (20.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.133s (7.500 fps)
Interval: Discrete 0.200s (5.000 fps)
Size: Discrete 864x480
Interval: Discrete 0.042s (24.000 fps)
Interval: Discrete 0.050s (20.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.133s (7.500 fps)
Interval: Discrete 0.200s (5.000 fps)
Size: Discrete 960x720
Interval: Discrete 0.067s (15.000 fps)
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.133s (7.500 fps)
Interval: Discrete 0.200s (5.000 fps)
Size: Discrete 1024x576
Interval: Discrete 0.067s (15.000 fps)
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.133s (7.500 fps)
Interval: Discrete 0.200s (5.000 fps)
Size: Discrete 1280x720
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.133s (7.500 fps)
Interval: Discrete 0.200s (5.000 fps)
Size: Discrete 1600x896
Interval: Discrete 0.133s (7.500 fps)
Interval: Discrete 0.200s (5.000 fps)
Size: Discrete 1920x1080
Interval: Discrete 0.200s (5.000 fps)
Size: Discrete 2304x1296
Interval: Discrete 0.500s (2.000 fps)
Size: Discrete 2304x1536
Interval: Discrete 0.500s (2.000 fps)
Index : 1
Type : Video Capture
Pixel Format: 'MJPG' (compressed)
Name : Motion-JPEG
Size: Discrete 640x480
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.042s (24.000 fps)
Interval: Discrete 0.050s (20.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.133s (7.500 fps)
Interval: Discrete 0.200s (5.000 fps)
Size: Discrete 160x90
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.042s (24.000 fps)
Interval: Discrete 0.050s (20.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.133s (7.500 fps)
Interval: Discrete 0.200s (5.000 fps)
Size: Discrete 160x120
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.042s (24.000 fps)
Interval: Discrete 0.050s (20.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.133s (7.500 fps)
Interval: Discrete 0.200s (5.000 fps)
Size: Discrete 176x144
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.042s (24.000 fps)
Interval: Discrete 0.050s (20.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.133s (7.500 fps)
Interval: Discrete 0.200s (5.000 fps)
Size: Discrete 320x180
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.042s (24.000 fps)
Interval: Discrete 0.050s (20.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.133s (7.500 fps)
Interval: Discrete 0.200s (5.000 fps)
Size: Discrete 320x240
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.042s (24.000 fps)
Interval: Discrete 0.050s (20.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.133s (7.500 fps)
Interval: Discrete 0.200s (5.000 fps)
Size: Discrete 352x288
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.042s (24.000 fps)
Interval: Discrete 0.050s (20.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.133s (7.500 fps)
Interval: Discrete 0.200s (5.000 fps)
Size: Discrete 432x240
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.042s (24.000 fps)
Interval: Discrete 0.050s (20.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.133s (7.500 fps)
Interval: Discrete 0.200s (5.000 fps)
Size: Discrete 640x360
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.042s (24.000 fps)
Interval: Discrete 0.050s (20.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.133s (7.500 fps)
Interval: Discrete 0.200s (5.000 fps)
Size: Discrete 800x448
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.042s (24.000 fps)
Interval: Discrete 0.050s (20.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.133s (7.500 fps)
Interval: Discrete 0.200s (5.000 fps)
Size: Discrete 800x600
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.042s (24.000 fps)
Interval: Discrete 0.050s (20.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.133s (7.500 fps)
Interval: Discrete 0.200s (5.000 fps)
Size: Discrete 864x480
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.042s (24.000 fps)
Interval: Discrete 0.050s (20.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.133s (7.500 fps)
Interval: Discrete 0.200s (5.000 fps)
Size: Discrete 960x720
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.042s (24.000 fps)
Interval: Discrete 0.050s (20.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.133s (7.500 fps)
Interval: Discrete 0.200s (5.000 fps)
Size: Discrete 1024x576
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.042s (24.000 fps)
Interval: Discrete 0.050s (20.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.133s (7.500 fps)
Interval: Discrete 0.200s (5.000 fps)
Size: Discrete 1280x720
Interval: Discrete 0.017s (60.000 fps)
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.042s (24.000 fps)
Interval: Discrete 0.050s (20.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.133s (7.500 fps)
Interval: Discrete 0.200s (5.000 fps)
Size: Discrete 1600x896
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.042s (24.000 fps)
Interval: Discrete 0.050s (20.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.133s (7.500 fps)
Interval: Discrete 0.200s (5.000 fps)
Size: Discrete 1920x1080
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.042s (24.000 fps)
Interval: Discrete 0.050s (20.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.133s (7.500 fps)
Interval: Discrete 0.200s (5.000 fps)
The reason MJPG
is capable of outputting higher resolutions and FPS than YUV formats is because raw YUV video frames are so gigantic (as they would be with any raw format) that a standard USB 2.0 interface (for example) wouldn't have enough throughput (in terms of speed) to handle them. For that reason, the vast majority of webcams also support MJPG
as a compressed way of streaming frames from the webcam.
I've been seeing if it would be possible just to stream the JPEGs directly and have had success in that. Originally, I wanted to only do raw video because I figured that would allow for the smallest attack surface. However, the compressed MJPG/MJPEG is much more widely used by IP cameras so because of that it may be more "battle tested" and therefore still potentially reasonably secure. Because of it's widespread use, it also has better support from applications (e.g. #1) and better performance.
More info about MJPEG: https://en.wikipedia.org/wiki/Motion_JPEG
I have a very simple working pipeline for passing JPEGs here:
Pipeline in qvc.Webcam
:
#!/bin/bash
gst-launch-1.0 -q v4l2src ! \
queue ! \
"image/jpeg" ! \
fdsink
Pipeline in receiver.sh
#!/bin/bash
gst-launch-1.0 -v fdsrc ! \
"image/jpeg,width=1920,height=1080,framerate=30/1,colorimetry=sRGB" ! \
jpegdec ! \
glimagesink
Make sure you have your webcam format is set to MJPG
or this will not work.
For testing in one VM (wihout passing between VMs): ./qvc.Webcam | ./receiver.sh
This will open a preview of the video feed in a OpenGL viewer. Note that the OpenGL viewer doesn't have the best performance but that's nothing to do with the MJPEG format. I tested this in dom0 (where my webcam device is) where and xvimagesink
can be used and the performance is fantastic. Note, xvimagesink
can only be used in dom0
because it requires an X graphic adapter. Check for yours with the xvinfo
command. (Source: http://gstreamer-devel.966125.n4.nabble.com/xvimagesink-Could-not-initialise-Xv-output-and-No-port-available-td3092246.html)
Note the jpegdec
does decode the JPEG to a YUV format on the receiver (so it can be inputted into glimagesink
for testing purposes) but what I want to do is get rid of the jpegdec
and have the JPEG stream go straight into v4l2sink
.
My only problem now is getting that into a v4l2sink
for v4l2loopback
. However, v4l2loopback
doesn't seem to want to accept the JPEG stream and I'm not sure why. I've put MJPEG directly into v4l2sink
before in #1 with avenc_mjpeg
(I found this solution in the v4l2loopback
GitHub Issues). The avenc_mjpeg
element converts YUV formats into MJPG
. I verified this with the --verbose
output on the receiver and just putting all of the output into a log file on receiver.sh
. I may open an issue with v4l2loopback
on this.
By the way, here is my v4l2-ctl -d 0 --all
output as well (when my webcam is configured to use MJPEG):
Driver Info (not using libv4l2):
Driver name : uvcvideo
Card type : C922 Pro Stream Webcam
Bus info : usb-0000:00:14.0-12
Driver version: 5.4.88
Capabilities : 0x84A00001
Video Capture
Streaming
Extended Pix Format
Device Capabilities
Device Caps : 0x04200001
Video Capture
Streaming
Extended Pix Format
Priority: 2
Video input : 0 (Camera 1: ok)
Format Video Capture:
Width/Height : 1920/1080
Pixel Format : 'MJPG'
Field : None
Bytes per Line : 0
Size Image : 4147200
Colorspace : sRGB
Transfer Function : Default
YCbCr Encoding : Default
Quantization : Default
Flags :
Crop Capability Video Capture:
Bounds : Left 0, Top 0, Width 1920, Height 1080
Default : Left 0, Top 0, Width 1920, Height 1080
Pixel Aspect: 1/1
Selection: crop_default, Left 0, Top 0, Width 1920, Height 1080
Selection: crop_bounds, Left 0, Top 0, Width 1920, Height 1080
Streaming Parameters Video Capture:
Capabilities : timeperframe
Frames per second: 30.000 (30/1)
Read buffers : 0
brightness (int) : min=0 max=255 step=1 default=128 value=128
contrast (int) : min=0 max=255 step=1 default=128 value=128
saturation (int) : min=0 max=255 step=1 default=128 value=128
white_balance_temperature_auto (bool) : default=1 value=1
gain (int) : min=0 max=255 step=1 default=0 value=0
power_line_frequency (menu) : min=0 max=2 default=2 value=2
white_balance_temperature (int) : min=2000 max=6500 step=1 default=4000 value=4000 flags=inactive
sharpness (int) : min=0 max=255 step=1 default=128 value=128
backlight_compensation (int) : min=0 max=1 step=1 default=0 value=0
exposure_auto (menu) : min=0 max=3 default=3 value=3
exposure_absolute (int) : min=3 max=2047 step=1 default=250 value=250 flags=inactive
exposure_auto_priority (bool) : default=0 value=1
pan_absolute (int) : min=-36000 max=36000 step=3600 default=0 value=0
tilt_absolute (int) : min=-36000 max=36000 step=3600 default=0 value=0
focus_absolute (int) : min=0 max=250 step=5 default=0 value=0 flags=inactive
focus_auto (bool) : default=1 value=1
zoom_absolute (int) : min=100 max=500 step=1 default=100 value=100
Thank you so much for all the help so far! Porting the pipeline into a Python programs with GI bindings is a big help.
I don't have ample time on my hands right this very moment but it would seem that you do. I'm going to add you you as a contributor to the repo so you can just make any additions you want to without having to go through me. Thanks again!
For the time being, I'm going to work on the C program for getting webcam formats with ioctl
s and selecting the best format from for the webcam from that.
It does not work for me, perhaps because my webcam does not support I420
That implicit conversion may be working for me because I'm using an older version (still a 1.x version though) of GStreamer that comes packaged with dom0 (Fedora 25). Remember, my webcam USB device is located in dom0
not sys-usb
.
I noticed you are doing modprobe uvcvideo
in your script and I'm wonder why is that. My webcam has a UVC driver model too but I see that kernel module automatically gets loaded as soon as I plug in the webcam. Also, I'm sure you're aware of this but while UVC is by far the most popular driver model for webcams, it is certainly not the only one.
Lastly, I noticed you're hard-coding the dimensions and FPS to your webcam values whereas the current webcam sender script doesn't do that. This will of course have to be changed in an actual release.
Thanks again for the help @DemiMarie!
I noticed you are doing
modprobe uvcvideo
in your script and I'm wonder why is that. My webcam has a UVC driver model too but I see that kernel module automatically gets loaded as soon as I plug in the webcam. Also, I'm sure you're aware of this but while UVC is by far the most popular driver model for webcams, it is certainly not the only one.
To reduce attack surface, I disallow autoloading of many drivers, including most USB drivers. Therefore, the UVC driver needs to be loaded manually. This is specific to my system and should not be part of a released version, at least unless Qubes adds its own blocklist.
Lastly, I noticed you're hard-coding the dimensions and FPS to your webcam values whereas the current webcam sender script doesn't do that. This will of course have to be changed in an actual release.
Indeed it will. I didn’t want to try to parse the output of v4l2-ctl
, and I didn’t want to duplicate the C program you are working on either. There is a C library (libv4l
on Debian) that might be useful, although I have not tried it and GStreamer seems to not use it.
Thanks again for the help @DemiMarie!
You’re welcome!
I wonder if a custom GStreamer element could help. The errors I am getting seem to indicate that GStreamer does not know what to do with the loopback device, perhaps because of a bug in the driver.
Looks like the problem is https://github.com/umlaeute/v4l2loopback/issues/137, which is really a GStreamer bug. GStreamer tries to query the supported formats of /dev/video0
, but v4l2loopback
supports basically every format there is, so it can’t give a sensible answer. GStreamer then proceeds to error out rather than picking a format that works.
Hm, yes I came to a similar conclusion but based on this issue where umlaeute talked about setting the "output-format" of the pipeline: https://github.com/umlaeute/v4l2loopback/issues/198#issuecomment-638008230
Maybe we could even just take code from the avenc_mjpeg
element to set the output format?
v4l2loopback
supports basically every format there is
You got that right! Here's a list of all the formats I found: In the docs: https://github.com/umlaeute/v4l2loopback/blob/main/doc/v4l2_formats.txt As defined in the header file: https://github.com/umlaeute/v4l2loopback/blob/main/v4l2loopback_formats.h
perhaps because my webcam does not support I420 video
Just coming back to this, I think the reason for this issue was in the older version of GStreamer (In R4.0 dom0) I'm using v4l2src
automatically converts from MJPG to a YUV format behind the scenes where as in the newer version they took that magic out. This makes sense too because as can be seen by looking at the pipeline PDFs in doc/visualizations
at no point is MJPG even mentioned, straight out of v4l2src
it's a YUV format (I420) even though my webcam is configured for MJPG. So, I'm glad we got to the bottom of that!
Fantastic news! I just found this issue where someone seems to be having the same problem we're having and has solved the issue (with a "janky fix") just 20 days ago for a project called mjpg-streamer
:
https://github.com/jacksonliam/mjpg-streamer/issues/298#issuecomment-821990963
This confirms that the issue is what we both thought it to be.
As proven by my PoC right here:
https://github.com/elliotkillick/qubes-video-companion/issues/13#issuecomment-830872159
We already are fully capable of streaming MJPEG across VMs through file descriptors using just GStreamer (no mjpg-streamer
necessary). This mjpg-streamer
project just seems to add other ways of sending the video like over HTTP which we of course don't need or want for security reasons. And of course an all GStreamer solution would also be best because it's packaged in all the distros and we still need it for the screen sharing component. Also, mjpg-streamer
is label as experimental.
So, my understanding is then is if we just implement a similar fix with a GStreamer element of our own forcefully setting V4L2_PIX_FMT_JPEG
as the pixel format (as done in the the mjpg-streamer
patch instead of in our case making GStreamer choose between the wide array of possible formats v4l2loopback
provides thus causing the bug) leading up the v4l2sink
element then we should be golden!
My webcam does not support
I420
output, so GStreamer insys-usb
fails. Removing this constraint fixes the problem.