Open V1ru5-99 opened 1 year ago
@rayburgemeestre Can you help me please?
Well, I recall that it was quite a bit of hell for me to get it to compile a while ago. :joy:
Did you run make configure
first though? It should "configure" a bunch of dependencies (it actually also compiles a lot of stuff)
Then the make compile
target (if the previous step succeeded) should be able to find those missing header files that you are seeing
I haven't tested this myself, at least not recently, but if you have docker on your system, it might be an option for you to try: make build-shell
, it should start up a build container, that someone else used to develop a feature (it was contributed)
I did build the container and pushed it to dockerhub, so you don't have to build it yourself, but I don't remember if I used it to compile the project from scratch successfully or not
@rayburgemeestre ok, so this is kinda weird, I did run "make configure" before but nothing changed, still same error, when I tried "make build-shell" i get this: docker run -it -v $PWD:$PWD -w $PWD rayburgemeestre/virtual-bg-dev-container:latest /bin/bash /bin/bash: line 1: docker: command not found make: *** [Makefile:142: build-shell] Error 127 However when I tried to install docker it tells me that I already have it installed (??):
Reading package lists... Done Building dependency tree... Done Reading state information... Done docker is already the newest version (1.5-2). The following packages were automatically installed and are no longer required: g++-11 gir1.2-javascriptcoregtk-4.0 gir1.2-soup-2.4 gir1.2-webkit2-4.0 ibverbs-providers libaccinj64-11.8 libatk-wrapper-java libatk-wrapper-java-jni libbabeltrace1 libboost-regex1.74.0 libcu++-dev libcub-dev libcublas11 libcublaslt11 libcudart11.0 libcufft10 libcufftw10 libcupti-dev libcupti-doc libcupti11.8 libcurand10 libcusolver11 libcusolvermg11 libcusparse11 libdebuginfod-common libdebuginfod1 libdmapsharing-3.0-2 libgl-dev libglx-dev libhwloc-plugins libhwloc15 libibumad3 libibverbs1 libipt2 libjs-jquery libnppc11 libnppial11 libnppicc11 libnppidei11 libnppif11 libnppig11 libnppim11 libnppist11 libnppisu11 libnppitc11 libnpps11 libnvblas11 libnvidia-ptxjitcompiler1 libnvjpeg11 libnvrtc-builtins11.8 libnvrtc11.2 libnvtoolsext1 libnvvm4 libpfm4 libpthread-stubs0-dev librdmacm1 libsoup-gnome2.4-1 libsource-highlight-common libsource-highlight4v5 libssh-4 libstdc++-11-dev libtbb-dev libtbb12 libtbbbind-2-5 libtbbmalloc2 libthrust-dev libucx0 libvdpau-dev libx11-dev libxau-dev libxcb1-dev libxdmcp-dev libxnvctrl0 nsight-compute nsight-compute-target nvidia-cuda-gdb nvidia-cuda-toolkit-doc nvidia-legacy-check nvidia-opencl-dev nvidia-openjdk-8-jre ocl-icd-opencl-dev opencl-c-headers opencl-clhpp-headers x11proto-dev xorg-sgml-doctools xtrans-dev Use 'sudo apt autoremove' to remove them. 0 upgraded, 0 newly installed, 0 to remove and 252 not upgraded.
@rayburgemeestre Meanwhile can you just upload a compiled version of the program on releases?
@rayburgemeestre ok, so this is kinda weird, I did run "make configure" before but nothing changed, still same error, when I tried "make build-shell" i get this: docker run -it -v $PWD:$PWD -w $PWD rayburgemeestre/virtual-bg-dev-container:latest /bin/bash /bin/bash: line 1: docker: command not found make: *** [Makefile:142: build-shell] Error 127 However when I tried to install docker it tells me that I already have it installed (??):
sudo apt install docker
Reading package lists... Done Building dependency tree... Done Reading state information... Done docker is already the newest version (1.5-2). The following packages were automatically installed and are no longer required: g++-11 gir1.2-javascriptcoregtk-4.0 gir1.2-soup-2.4 gir1.2-webkit2-4.0 ibverbs-providers libaccinj64-11.8 libatk-wrapper-java libatk-wrapper-java-jni libbabeltrace1 libboost-regex1.74.0 libcu++-dev libcub-dev libcublas11 libcublaslt11 libcudart11.0 libcufft10 libcufftw10 libcupti-dev libcupti-doc libcupti11.8 libcurand10 libcusolver11 libcusolvermg11 libcusparse11 libdebuginfod-common libdebuginfod1 libdmapsharing-3.0-2 libgl-dev libglx-dev libhwloc-plugins libhwloc15 libibumad3 libibverbs1 libipt2 libjs-jquery libnppc11 libnppial11 libnppicc11 libnppidei11 libnppif11 libnppig11 libnppim11 libnppist11 libnppisu11 libnppitc11 libnpps11 libnvblas11 libnvidia-ptxjitcompiler1 libnvjpeg11 libnvrtc-builtins11.8 libnvrtc11.2 libnvtoolsext1 libnvvm4 libpfm4 libpthread-stubs0-dev librdmacm1 libsoup-gnome2.4-1 libsource-highlight-common libsource-highlight4v5 libssh-4 libstdc++-11-dev libtbb-dev libtbb12 libtbbbind-2-5 libtbbmalloc2 libthrust-dev libucx0 libvdpau-dev libx11-dev libxau-dev libxcb1-dev libxdmcp-dev libxnvctrl0 nsight-compute nsight-compute-target nvidia-cuda-gdb nvidia-cuda-toolkit-doc nvidia-legacy-check nvidia-opencl-dev nvidia-openjdk-8-jre ocl-icd-opencl-dev opencl-c-headers opencl-clhpp-headers x11proto-dev xorg-sgml-doctools xtrans-dev Use 'sudo apt autoremove' to remove them. 0 upgraded, 0 newly installed, 0 to remove and 252 not upgraded.
Could you confirm whether you have Docker installed on your system? You can verify this by using the Docker CLI command docker --version in your terminal. If Docker is not installed, you can follow the instructions from the official Docker documentation at this link: https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository.
@rayburgemeestre Meanwhile can you just upload a compiled version of the program on releases?
That would indeed be a great idea, but compiling the entire program statically could potentially be quite a big task.
Hey @V1ru5-99 I wanted to reply to your message from 4 hours ago with a link, and then I discovered that actually I forgot to upload the binary. I just uploaded it here now: https://github.com/rayburgemeestre/webcam-virtual-background/releases/tag/1.1
wget https://github.com/rayburgemeestre/webcam-virtual-background/releases/download/1.1/virtual-bg-1.1.tar.gz
It was supposed to be there, sorry. I did make the tarball when I made that 1.1 release, I just forgot to upload it I guess. I tested just now if it works in debian:sid
using docker (since I use Ubuntu 20.04), and I seem to get the binary working. This is how I started docker on my system:
xhost +
docker run --privileged -it -e DISPLAY=\$DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix -v $PWD:$PWD -w $PWD debian:sid /bin/bash
Just sharing the above as a 'fyi', but in your case your system is debian:sid, so no docker needed. Inside docker, or in your case on your system, I would ensure that these dependencies are installed:
apt update
apt-get install -y v4l2loopback-dkms make sudo libreadline8 libavfilter8 libxcb-shape0 ffmpeg
Then if you download the binaries from the URL I send you (extracting the virtual-bg-1.1.tar.gz) there is a 'cam' script there which invokes 'main' after creating the two virtual devices. But you will probably get this error:
root@9d3f162f2896:/home/rayb/tmp/virtual-bg-1.1# ./cam
./main: error while loading shared libraries: libavfilter.so.7: cannot open shared object file: No such file or directory
On debian:sid the version is libavfilter.so.8, but luckily I bundled that dependency in the 'lib' dir, it just seems that I forgot to put the following line in the cam
script: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$PWD/lib
You can edit the file and put it before the ./main
line, and that should fix it. Or invoke it manually, like below:
root@9d3f162f2896:/home/rayb/tmp/virtual-bg-1.1# export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$PWD/lib
root@9d3f162f2896:/home/rayb/tmp/virtual-bg-1.1# ./cam
art by: Marcin Glinski _
/ \
/ .'_
/ __| \
`. | / (-' |
`. \_..._ : (_,-/
`-. `,' `-. /`-.__,'
`/ __ \ / /
/`/ \ :' /
_,\o\_o/ / /
(_) ___.--. / /
`-. -._.i \. :
`.\ ( |:. |
,' )`-' |:.. / \
__ ,' | `.:. `.
(_ `---: ) \:. \
,' `. .'\ \:. )
,' ,' ,' \\ o |:. /
(_,' ,7 / \`.__.':.. /,,,
(_,'(_,' _gdMbp,,dp,,,,,,dMMMMMbp,,
,dMMMMMMMMMMMMMMMMMMMMMMMMMMMb,
.dMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMb, fsc
.dMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM,
,MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
dMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM.
.dMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMb
V I R T U A L B A C K G R O U N D W E B C A M
Available commands are:
preview
stop
set-mode
set-cam
list-cams
start
exit
quit
set-model
run
help
cam>
@ChillAndImprove I didn't do a fully static binary indeed, but at least I also shipped most .so libraries it needs in the lib
directory in the .tar.gz file. :sweat_smile:
I'll re-upload the .tar.gz file tomorrow with the export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$PWD/lib
fix in the 'cam' script to prevent others from running into it... :crossed_fingers: that it works on your system
@rayburgemeestre Hi, thank you for uploading the binary, the program works just fine! How do i change the virtual camera name? I need to change it so that my browser won't detect that I'm using a virtual camera. Edit: nevermind, I figured it out myself, unfortunately the camera is a bit laggy, i get like a 1 second delay, is there anything I can do about it?
@rayburgemeestre @ChillAndImprove How do I change the codec to improve performance? I'm currently running on nvidia 340 because I'm running with an older graphics card (GT 210)
Hey @V1ru5-99 I haven't experienced that much lag personally, I can actually not notice much of the lag in my case.
I think it makes sense to figure out which step is introducing the lag. Let's say your camera is /dev/video0, you can do something like:
ffplay /dev/video0
It should be without 1 second lag.. since it's the source of the video, if there is a problem there already there isn't much we can do about it.
Then after you "start" the cam tool, you can try:
ffplay /dev/video8
If it is 1 second behind, the lag is introduced in initial resize + converting stage. Could you also share what the "start" command prints for the input? In my case this is:
Input #0, video4linux2,v4l2, from '':
Duration: N/A, start: 65904.341082, bitrate: 110592 kb/s
Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 640x480, 110592 kb/s, 30 fps, 30 tbr, 1000k tbn, 1000k tbc
If there are no issues with /dev/video8, then we need to check /dev/video9...
With: ffplay /dev/video9
If that has 1 second lag, then the slowness is in the last part where the background blurring etc happens.. That might be a little bit harder to solve, so I hope it's something with the resize step.
If the slowness is in /dev/video8, then I would suggest the following debugging..
We do: /dev/video0
-> /dev/video8
to resize and convert to YUV. The command for this something like:
/usr/bin/ffmpeg -pix_fmt mjpeg -i /dev/video0
-f v4l2 -input_format mjpeg -framerate 10 -video_size 1024x680 -vf
scale=640:480:force_original_aspect_ratio=increase,crop=640:480 -pix_fmt yuv420p -f v4l2 /dev/video8;
You could run that command manually, and then view ffplay /dev/video8
, and perhaps customize it until it becomes fast. Now that I looked this command up, I'm a little surprised about the hardcoded -video_size 1024x680
part in the command, and I'm tempted to see if I can get rid of it.
In my case I could do that and use:
/usr/bin/ffmpeg -pix_fmt mjpeg -i /dev/video0 \
-f v4l2 -input_format mjpeg -framerate 10 -vf \
scale=640:480:force_original_aspect_ratio=increase,crop=640:480 -pix_fmt yuv420p -f v4l2 /dev/video8;
I believe it would be interesting to see what happens if you scale it down even further, e.g.:
/usr/bin/ffmpeg -pix_fmt mjpeg -i /dev/video0 \
-f v4l2 -input_format mjpeg -framerate 10 -vf \
scale=480:320:force_original_aspect_ratio=increase,crop=480:320 -pix_fmt yuv420p -f v4l2 /dev/video8;
And then, all the way up to:
/usr/bin/ffmpeg -pix_fmt mjpeg -i /dev/video0 \
-f v4l2 -input_format mjpeg -framerate 10 -vf \
scale=320:240:force_original_aspect_ratio=increase,crop=320:240 -pix_fmt yuv420p -f v4l2 /dev/video8;
If the lag is reducing then it might be a CPU bound issue.. In the code I've hardcoded the 640x480 resolution, so it won't fix your issue but if it solves it, then I could prioritize making the resolution customizable
@rayburgemeestre Hi, sorry for the late response, I've thought that the lag happens in /dev/video8 because my camera is running just fine without the virtual background program however now when i tried to run ffplay /dev/video0 I've noticed that the 1 second delay is there aswell. I've also tried https://volcomix.github.io/virtual-background/ and it also worked fine using Canvas 2D + CPU. So I belive there is a chance that the lag might be caused by a specific setting used by ffmpeg. Unfortunately I'm not so familiar with configuring ffmpeg in order to see what's the issue so, I'm gonna ask for your help here.
Alright,
So ffplay /dev/video0
is already laggy..
Do the following flags help? I would try something in the following order probably...
ffplay -fflags nobuffer /dev/video0
ffplay -fflags nobuffer -flags low_delay /dev/video0
ffplay -fflags nobuffer -flags low_delay -framedrop /dev/video0
ffplay -fflags nobuffer -flags low_delay -framedrop -strict experimental /dev/video0
Source: https://stackoverflow.com/a/49273163
If the flags help maybe I can simply add them by default
@rayburgemeestre Thanks, the "nobuffer" flag alone seems to fix the issue. I haven't noticed any difference using the other flags. Let me know how do I apply the flag using your program.
Alright, So
ffplay /dev/video0
is already laggy..Do the following flags help? I would try something in the following order probably...
ffplay -fflags nobuffer /dev/video0 ffplay -fflags nobuffer -flags low_delay /dev/video0 ffplay -fflags nobuffer -flags low_delay -framedrop /dev/video0 ffplay -fflags nobuffer -flags low_delay -framedrop -strict experimental /dev/video0
Source: https://stackoverflow.com/a/49273163
If the flags help maybe I can simply add them by default
@rayburgemeestre Hi, its been a while, can you add the nobuffer flag by default? Or how do I manually run the flag with the virtual background?
Ok Ive made a release with this fix, hope it works :crossed_fingers: https://github.com/rayburgemeestre/webcam-virtual-background/releases/tag/1.1.1 :grin:
Hi @rayburgemeestre, thank you for uploading the release! The lag is now gone in /dev/video8. In /dev/video9 there is a small delay of 0.1 - 0.2 seconds maybe, I was wondering if you can look into that when you have time, otherwise the background works fine, the latency is noticeable but it can also be ignored. However I'd rather see it fixed if possible :).
make compile
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$PWD/ffmpeg/lib:$PWD/build/tensorflow/bazel-bin/tensorflow/lite \
PKG_CONFIG_PATH=$PWD/ffmpeg/lib/pkgconfig c++ -O0 -g --std=c++17 -I$PWD/ffmpeg/include -I$PWD/build/tensorflow/ -I$PWD/build/tensorflow/third_party/ \
PKG_CONFIG_PATH=$PWD/ffmpeg/lib/pkgconfig c++ -O2 --std=c++17 -I$PWD/ffmpeg/include -I$PWD/build/tensorflow/ -I$PWD/build/tensorflow/third_party/ \ -I$PWD/ffmpeg-4.4 \ -I$PWD/build/mediapipe \ -I$PWD/build/tensorflow/tensorflow/lite/tools/make/downloads/flatbuffers/include \ -I$PWD/build/cpp-readline/src \ -I$PWD/build/tiny-process-library \ -L$PWD/ffmpeg/lib \ -L$PWD/build/tensorflow/bazel-bin/tensorflow/lite \ -Wl,-rpath=lib \ src/main.cpp src/transpose_conv_bias.cc src/blur_float.cpp build/cpp-readline/src/Console.cpp src/snowflake.cpp \ -lavdevice -lavformat -lavcodec -lavutil -ltensorflowlite -lswscale -lreadline -lpthread \ $PWD/build/tiny-process-library/build/libtiny-process-library.a \ -o main src/main.cpp:7:10: fatal error: Console.hpp: No such file or directory 7 | #include "Console.hpp" | ^
~~~~ compilation terminated. src/transpose_conv_bias.cc:19:10: fatal error: mediapipe/util/tflite/operations/transpose_conv_bias.h: No such file or directory 19 | #include "mediapipe/util/tflite/operations/transpose_conv_bias.h" | ^~~~~~~~~~~~compilation terminated. cc1plus: fatal error: build/cpp-readline/src/Console.cpp: No such file or directory compilation terminated. make: *** [Makefile:66: compile] Error 1