search

allo- / virtual_webcam_background

Use a virtual webcam background and overlays with body-pix and v4l2loopback
GNU General Public License v3.0
306 stars 48 forks source link

Dummy Video Device (0x0000) in mjpeg instead of YUYV? #38

Open floydfire opened 4 years ago

floydfire commented 4 years ago

Sorry to clutter here and write this issue (which i guess is more an issue of me not being really a dev) i was able to get the webcam to work (i'm on manjaro linux) and all works but in webcamoid and other apps the format for the "Dummy Video Device" is YUYV and my webcam doesn't really support too well that format, can anyone give me some guidance on how to set it to Mjpeg? (if possible) i've already set the option on True in the config.yaml but it still goes with YUYV. Thanks.

allo- commented 4 years ago

Can you check with qv4l what's supported by your camera? It can be possible, that you need the correct combination of format and resolution (though not setting anything should give you a usable default).

The program should read input frames in YUYV or MJPEG (and possibly any OpenCV supported format) and writes output forms always as YUYV to the virtual camera device. So your camera does not need to support YUYV good, but your program that should use the virtual camera must support it.

For the input OpenCV should probably fall back to the default provided by your camera when you do not enable mjpeg and otherwise try to force mjpeg. Forcing other formats like h264 (when supported) would be possible similar to forcing mjpeg, but I suppose it will only be slower.

floydfire commented 4 years ago

Under capture image formats i get MJPG (Motion-Jpeg), YUYV (YUYV 4:2:2), RGB3 (Emulated), BGR3 (Emulated),YU12 (Emulated), YV12 (Emulated) the webcam is an Aukey PC-LM1 USB Camera. The problem in question is that my webcam support 1080p w 30 fps on MJPG and it works fine without the virtual webcam running but support only 5 fps on YUYV at that resolution...

allo- commented 4 years ago

That's what the MJPEG setting is for, so you should be able to use it for faster input.

But I seriously doubt that you will reach more than 5 FPS with 1080p using this program. Your images need to be processed in every iteration by the body-pix model, what needs some time especially at high resolutions. Using CUDA will help, but 1080p is still quite large.

Try if you can get more performance using a lower resolution. Start with 640x480 and when this runs faster, start tuning what resolution can be processed fast enough. You can try to use internal_resolution for downscaling for the model and writing larger frames.

Maybe you should use 720p (what works here acceptable). I currently use 800x600 most the time as the model seems to process 4:3 images better than 16:10. But tuning the segmentation threshold is needed anyway, as it depends a bit on your background and your lighting.

floydfire commented 4 years ago

i can install CUDA (i have an nvdia gpu 1050TI) image but what of those should i install?

allo- commented 4 years ago

You'll need tensorflow-gpu and then a installed CUDA SDK (what you can possibly get through your Linux distribution) and then libcudnn and libnvinfer from the nvidia homepage (you'll need to create an account to download that).

You need to make sure that your CUDA and tensorflow are compatible. I use tensorflow 2.2 and CUDA 10.

We should write some documentation on that, but I am not sure about what can be combined in which way either, I just experimented and googled to find what versions I need.