search

RazZziel / fakecam

Open Source Virtual Background, stolen from https://elder.dev/posts/open-source-virtual-background/
8 stars 2 forks source link

Can't run the code #2

Closed tomchor closed 3 years ago

tomchor commented 3 years ago

Thanks for packaging this code nicely! However, I can't run it! After taking a look at this comment I've been focusing on the non-docker script.

What I did was:

However, the running fails at some point:

make: Entering directory '/home/tomas/repos/fakecam/fakecam' make: Entering directory '/home/tomas/repos/fakecam/bodypix' .venv/bin/python fake.py $ARGS node app.js 2021-03-08 12:19:05.508002: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory 2021-03-08 12:19:05.508029: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine. 2021-03-08 12:19:05.580577: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2021-03-08 12:19:05.610534: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2700905000 Hz 2021-03-08 12:19:05.610866: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5648a284bca0 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2021-03-08 12:19:05.610903: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version Platform node has already been set. Overwriting the platform with [object Object]. Traceback (most recent call last): File "fake.py", line 137, in interpreter = tf.lite.Interpreter(model_path=args.model) File "/home/tomas/repos/fakecam/fakecam/.venv/lib/python3.8/site-packages/tensorflow/lite/python/interpreter.py", line 197, in init _interpreter_wrapper.CreateWrapperFromFile( ValueError: Could not open 'deeplabv3_257_mv_gpu.tflite'. make: *** [Makefile:2: all] Error 1 make: Leaving directory '/home/tomas/repos/fakecam/fakecam'

Any idea as to why? Thanks!

RazZziel commented 3 years ago

Could not open 'deeplabv3_257_mv_gpu.tflite'

Looks like it's trying to use deeplabv3_257_mv_gpu.tflite, which is a file that I have unversioned on my working copy, next to fake.py, and I don't really remember how it got there :P

I looked it up and the official source for that model seems to be https://storage.googleapis.com/download.tensorflow.org/models/tflite/gpu/deeplabv3_257_mv_gpu.tflite

tomchor commented 3 years ago

Just to be clear: unversioned meaning it's simply not included in the git repo?

I downloaded it into fakecam directory and that error doesn't happen anymore. Is the solution just include that file in the repo?

However, I do face another error now. The webcam starts (I can see a blue light) but I get this:

make: Entering directory '/home/tomas/repos/fakecam/bodypix'
node app.js
make: Entering directory '/home/tomas/repos/fakecam/fakecam'
.venv/bin/python fake.py $ARGS
2021-03-08 15:26:21.512199: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory
2021-03-08 15:26:21.512222: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2021-03-08 15:26:21.512938: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2021-03-08 15:26:21.547416: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2700905000 Hz
2021-03-08 15:26:21.547881: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x556767fca660 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-03-08 15:26:21.547912: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
Platform node has already been set. Overwriting the platform with [object Object].
Opening webcam /dev/video0 ...
Writing to loopback device /dev/video20 ...
Loading background data/Big enough.mp4
/home/tomas/repos/fakecam/bodypix/node_modules/@tensorflow/tfjs/node_modules/@tensorflow/tfjs-core/dist/engine.js:542
                out = _this.tidy(function () { return forwardFunc(_this.backend, saveFunc_1); });
                                                      ^

TypeError: forwardFunc is not a function
    at /home/tomas/repos/fakecam/bodypix/node_modules/@tensorflow/tfjs/node_modules/@tensorflow/tfjs-core/dist/engine.js:542:55
    at /home/tomas/repos/fakecam/bodypix/node_modules/@tensorflow/tfjs/node_modules/@tensorflow/tfjs-core/dist/engine.js:388:22
    at Engine.scopedRun (/home/tomas/repos/fakecam/bodypix/node_modules/@tensorflow/tfjs/node_modules/@tensorflow/tfjs-core/dist/engine.js:398:23)
    at Engine.tidy (/home/tomas/repos/fakecam/bodypix/node_modules/@tensorflow/tfjs/node_modules/@tensorflow/tfjs-core/dist/engine.js:387:21)
    at kernelFunc (/home/tomas/repos/fakecam/bodypix/node_modules/@tensorflow/tfjs/node_modules/@tensorflow/tfjs-core/dist/engine.js:542:29)
    at /home/tomas/repos/fakecam/bodypix/node_modules/@tensorflow/tfjs/node_modules/@tensorflow/tfjs-core/dist/engine.js:553:27
    at Engine.scopedRun (/home/tomas/repos/fakecam/bodypix/node_modules/@tensorflow/tfjs/node_modules/@tensorflow/tfjs-core/dist/engine.js:398:23)
    at Engine.runKernelFunc (/home/tomas/repos/fakecam/bodypix/node_modules/@tensorflow/tfjs/node_modules/@tensorflow/tfjs-core/dist/engine.js:551:14)
    at Engine.runKernel (/home/tomas/repos/fakecam/bodypix/node_modules/@tensorflow/tfjs/node_modules/@tensorflow/tfjs-core/dist/engine.js:449:21)
    at pad_ (/home/tomas/repos/fakecam/bodypix/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:15185:19)
make: *** [Makefile:2: all] Error 1
make: Leaving directory '/home/tomas/repos/fakecam/bodypix'
mask request failed, retrying
mask request failed, retrying

And then the last couple of lines just print endlessly.

Any tips?

RazZziel commented 3 years ago

Just to be clear: unversioned meaning it's simply not included in the git repo?

Yep

I downloaded it into fakecam directory and that error doesn't happen anymore. Is the solution just include that file in the repo?

Either to include the file in the repo or to modify the run scripts and/or the documentation to download the file on demand. The first option seems to be more appropriate in this case.

However, I do face another error now. The webcam starts (I can see a blue light) but I get this:

(...)
/home/tomas/repos/fakecam/bodypix/node_modules/@tensorflow/tfjs/node_modules/@tensorflow/tfjs-core/dist/engine.js:542
                out = _this.tidy(function () { return forwardFunc(_this.backend, saveFunc_1); });
                                                      ^

TypeError: forwardFunc is not a function
    at /home/tomas/repos/fakecam/bodypix/node_modules/@tensorflow/tfjs/node_modules/@tensorflow/tfjs-core/dist/engine.js:542:55
    at /home/tomas/repos/fakecam/bodypix/node_modules/@tensorflow/tfjs/node_modules/@tensorflow/tfjs-core/dist/engine.js:388:22
    at Engine.scopedRun (/home/tomas/repos/fakecam/bodypix/node_modules/@tensorflow/tfjs/node_modules/@tensorflow/tfjs-core/dist/engine.js:398:23)
    at Engine.tidy (/home/tomas/repos/fakecam/bodypix/node_modules/@tensorflow/tfjs/node_modules/@tensorflow/tfjs-core/dist/engine.js:387:21)
    at kernelFunc (/home/tomas/repos/fakecam/bodypix/node_modules/@tensorflow/tfjs/node_modules/@tensorflow/tfjs-core/dist/engine.js:542:29)
    at /home/tomas/repos/fakecam/bodypix/node_modules/@tensorflow/tfjs/node_modules/@tensorflow/tfjs-core/dist/engine.js:553:27
    at Engine.scopedRun (/home/tomas/repos/fakecam/bodypix/node_modules/@tensorflow/tfjs/node_modules/@tensorflow/tfjs-core/dist/engine.js:398:23)
    at Engine.runKernelFunc (/home/tomas/repos/fakecam/bodypix/node_modules/@tensorflow/tfjs/node_modules/@tensorflow/tfjs-core/dist/engine.js:551:14)
    at Engine.runKernel (/home/tomas/repos/fakecam/bodypix/node_modules/@tensorflow/tfjs/node_modules/@tensorflow/tfjs-core/dist/engine.js:449:21)
    at pad_ (/home/tomas/repos/fakecam/bodypix/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:15185:19)
make: *** [Makefile:2: all] Error 1
make: Leaving directory '/home/tomas/repos/fakecam/bodypix'
(...)

Hmm, weird error. Looking it up, you get here: https://github.com/tensorflow/tfjs/issues/2329:

This looks like a mismatch between the version of tfjs-core and tfjs-node. Can you update to the latest @tensorflow/tfjs and @tensorflow/tfjs-node. Also what might help is to remove node_modules and rerun npm install.

It works, thx))

tomchor commented 3 years ago

Any idea on how to tackle this weird error? Otherwise unfortunately it's the end of the line for me haha

RazZziel commented 3 years ago

Can you run npm ls in the bodypix directory and see which versions are you using for each package?

For instance, these are mine, you could try using the same versions:

raziel@Bebop ~/dev/fakecam/bodypix:master $ npm ls
bodypix@0.0.1 /home/raziel/dev/fakecam/bodypix
├── @tensorflow-models/body-pix@2.0.5
└── @tensorflow/tfjs-node@1.7.2
tomchor commented 3 years ago

This is mine:

(base) tomas@np900:~/repos/fakecam/bodypix$ npm ls
bodypix@0.0.1 /home/tomas/repos/fakecam/bodypix
├── @tensorflow-models/body-pix@2.1.0
└── @tensorflow/tfjs-node@1.7.4

Also, just FYI: I have absolutely zero experience with npm and docker.

RazZziel commented 3 years ago

Then I'd recomment trying to install @tensorflow-models/body-pix@2.0.5 and @tensorflow/tfjs-node@1.7.2

You can just run npm install @tensorflow-models/body-pix@2.0.5 @tensorflow/tfjs-node@1.7.2

All this doesn't have anything to do with Docker; we're installing these packages in our machine, not in a container

tomchor commented 3 years ago

Thanks for the help! I definitely don't get that error anymore, but I get this one:

node app.js
2021-03-10 16:25:23.387262: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory
2021-03-10 16:25:23.387292: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2021-03-10 16:25:23.393326: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2021-03-10 16:25:23.426070: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2700905000 Hz
2021-03-10 16:25:23.426386: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55b1940319d0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-03-10 16:25:23.426408: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
cpu backend was already registered. Reusing existing backend factory.
Platform node has already been set. Overwriting the platform with [object Object].
Opening webcam /dev/video0 ...
Writing to loopback device /dev/video20 ...
Loading background data/Big enough.mp4
Traceback (most recent call last):
  File "fake.py", line 223, in <module>
    final_frame = blend_frame(final_frame, background, mask)
  File "fake.py", line 111, in blend_frame
    result[:,:,c] = frame[:,:,c]*mask + background[:,:,c]*inv_mask
ValueError: operands could not be broadcast together with shapes (400,640) (360,640) 
^CException ignored in: <module 'threading' from '/home/tomas/miniconda3/lib/python3.8/threading.py'>
Traceback (most recent call last):
  File "/home/tomas/miniconda3/lib/python3.8/threading.py", line 1388, in _shutdown
    lock.acquire()
KeyboardInterrupt: 
make: *** [Makefile:2: all] Interrupt
make: Leaving directory '/home/tomas/repos/fakecam/bodypix'
FATAL: exception not rethrown
make: *** [Makefile:2: all] Aborted (core dumped)

If I change the default height from 400 to 360 in fake.py it doesn't throw me an error anymore. But also the output is just a back image. It seems like software keeps loading some images but reeeally slowly. This is the output after 5 minutes:

make: Entering directory '/home/tomas/repos/fakecam/fakecam'
.venv/bin/python fake.py $ARGS
make: Entering directory '/home/tomas/repos/fakecam/bodypix'
node app.js
2021-03-10 16:27:55.436028: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2021-03-10 16:27:55.442447: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory
2021-03-10 16:27:55.442475: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2021-03-10 16:27:55.466060: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2700905000 Hz
2021-03-10 16:27:55.466441: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5645177ae0d0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-03-10 16:27:55.466479: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
cpu backend was already registered. Reusing existing backend factory.
Platform node has already been set. Overwriting the platform with [object Object].
Opening webcam /dev/video0 ...
Writing to loopback device /dev/video20 ...
Loading background data/Big enough.mp4
Loading background data/background2.jpg
Loading background data/background.jpg
Loading background data/background10.jpg
Loading background data/background3.jpg

Do you understand what's going on here?

RazZziel commented 3 years ago 
ValueError: operands could not be broadcast together with shapes (400,640) (360,640)

If I change the default height from 400 to 360 in fake.py it doesn't throw me an error anymore

That's weird, maybe there's a resizing bug over there but it just slipped because I only tested this script with my 640x400 camera, and fails if the camera doesn't support that resolution

But also the output is just a back image.

That's weird; how did you test it? Did you open a conferencing app/website and selected the correct camera? Or did you use something like mplayer tv:///dev/video20?

Please keep in mind that if your webcam usually sits on /dev/video0, that device may not be available while the program is running, because it's being used. The program creates a second "virtual" webcam on /dev/video20 where it outputs the results, and that's the webcam that should be used by any application where the modified stream needs to be displayed.

It seems like software keeps loading some images but reeeally slowly. This is the output after 5 minutes:

(...)
Loading background data/Big enough.mp4
Loading background data/background2.jpg
Loading background data/background.jpg
Loading background data/background10.jpg
Loading background data/background3.jpg

Nah that's fine, that "Loading background" is printed every time the background is changed, and it changes every minute:

fakecam/fake.py:

    while True:
        if background is None or datetime.datetime.now() - now > datetime.timedelta(minutes=1):
            change_background()
            now = datetime.datetime.now()
tomchor commented 3 years ago

That's weird; how did you test it? Did you open a conferencing app/website and selected the correct camera? Or did you use something like mplayer tv:///dev/video20?

I opened zoom, which is how I want to use the fakecam. I expected to find two webcams there: my default one and the "fake" one. But only the default one was there, except it was black!

Anyway, I just found out that you have to restart zoom for it to pick up on the second cam and now it appears to work! My camera is shitty enough that it doesn't work very well, haha but it's better than nothing.

Thanks for the help!

RazZziel commented 3 years ago

Awesome! Enjoy!