search

fangfufu / Linux-Fake-Background-Webcam

Faking your webcam background under GNU/Linux, now supports background blurring, animated background, colour map effect, hologram effect and on-demand processing.
GNU General Public License v3.0
1.61k stars 161 forks source link

Mediapipe 0.8.6 segfaults after running for a couple minutes #127

Closed darthmooguy closed 3 years ago

darthmooguy commented 3 years ago

While running, I get 20-30 fps depending on the other workload of my CPU, which is pretty good! However, like in #105, I get segfaults after 4-5 minutes.

Here is my setup: Fedora 33 Kernel 5.12.12-200.fc33.x86_64 CPU Intel i7-6700HQ (8) @ 3.500GHz RAM 32GB Latest commit of master branch of this repo (23ffc61)

To date, I have tried with v4l2loopback-dkms version 0.15.2 from this copr, and with the latest commit of the main branch of the v4l2loopback repo.

On the multithreaded branch I get better FPS (45), but it also segfaults.

Here is the output of the command during a Zoom call:

$ python3 fake.py -w /dev/video2 -v /dev/video6 --no-foreground
Real camera original values are set as: 640x480 with 30 FPS and video codec 1448695129
Real camera new values are set as: 1280x720 with 30 FPS and video codec 1196444237
Running...
Please CTRL-C to pause and reload the background / foreground images
Please CTRL-\ to exit
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
Consumers: 1
No consumers remaining, paused
Consumers: 1
No consumers remaining, paused
Consumers: 1
Real camera original values are set as: 640x480 with 30 FPS and video codec 1448695129
Real camera new values are set as: 1280x720 with 30 FPS and video codec 1196444237
No consumers remaining, paused
Consumers: 1
Real camera original values are set as: 640x480 with 30 FPS and video codec 1448695129
Real camera new values are set as: 1280x720 with 30 FPS and video codec 1196444237
No consumers remaining, paused
Consumers: 1
Consumers: 2
Consumers: 1
Consumers: 2
Consumers: 1
fish: Job 1, 'python3 fake.py -w /dev/video2…' terminated by signal SIGSEGV (Address boundary error)

It also segfaults using the --no-ondemand param.

fangfufu commented 3 years ago

How about 0b0321efae9806fb558138bbd60bf5d770f57580? That's the commit I have right before adding on-demand processing. Does it segfault?

fangfufu commented 3 years ago

@DrDynamic's reply suggests problem with mediapipe

https://github.com/fangfufu/Linux-Fake-Background-Webcam/issues/116#issuecomment-871519869

Could you both try updating Mediapipe to 0.8.6?

darthmooguy commented 3 years ago

I'm already on 0.8.6 :thinking: 

$ pip3 list | grep mediapipe
mediapipe             0.8.6
fangfufu commented 3 years ago

@darthmooguy , could you downgrade to 0.8.5 to see if it helps? You can always try compiling mediapipe from source, but that sounds extremely painful.

darthmooguy commented 3 years ago

I've tried 0b0321e also, and I still get segfaults. However, I get them pretty much everytime i'm doing something CPU intensive (opening Android Studio, compiling an Android project in Android Studio...), and with 0b0321e it crashed both the fake.py and Zoom the first time I opened Zoom!

I'll try later today with mediapipe 0.8.5.

katakombi commented 3 years ago

I am running now with mediapipe 0.8.5 which seems notably slower but no crashes so far on master...

darthmooguy commented 3 years ago

@fangfufu With mediapipe 0.8.5 it seems stable on master! I was able to be in a zoom call for 15 minutes without segfaulting, and while compiling!

However, I get 15 fps and it sometimes drops to 7 fps (seems random), which is pretty slow, altough not a deal breaker.

fangfufu commented 3 years ago

Oh! This is very interesting!

DrDynamic commented 3 years ago

@fangfufu Yesterday I tried compiling mediapipe myself but the Segmentation fault kept coming.

Now I'm running mediapipe 0.8.5 a Segmentation fault with another stacktrace:

Thread 1 "python3" received signal SIGSEGV, Segmentation fault.
0x00000000004f5a2a in ?? ()
(gdb) backtrace
#0  0x00000000004f5a2a in ?? ()
#1  0x000000000056df6e in ?? ()
#2  0x000000000060bedb in ?? ()
#3  0x000000000060bc73 in ?? ()
#4  0x000000000060b1e4 in PyPegen_ASTFromStringObject ()
#5  0x00000000006140be in PyRun_StringFlags ()
#6  0x000000000061def2 in ?? ()
#7  0x000000000052de88 in ?? ()
#8  0x000000000051635b in _PyEval_EvalFrameDefault ()
#9  0x0000000000515377 in ?? ()
#10 0x000000000052d302 in _PyFunction_Vectorcall ()
#11 0x000000000051b3ac in _PyEval_EvalFrameDefault ()
#12 0x0000000000514a75 in ?? ()
#13 0x000000000052d302 in _PyFunction_Vectorcall ()
#14 0x000000000054015f in ?? ()
#15 0x00000000005177f3 in _PyEval_EvalFrameDefault ()
#16 0x0000000000514a75 in ?? ()
#17 0x000000000052d302 in _PyFunction_Vectorcall ()
#18 0x0000000000516543 in _PyEval_EvalFrameDefault ()
#19 0x000000000052d163 in _PyFunction_Vectorcall ()
#20 0x0000000000516543 in _PyEval_EvalFrameDefault ()
#21 0x000000000052d163 in _PyFunction_Vectorcall ()
#22 0x0000000000516543 in _PyEval_EvalFrameDefault ()
#23 0x000000000052d163 in _PyFunction_Vectorcall ()
#24 0x000000000051635b in _PyEval_EvalFrameDefault ()
#25 0x0000000000514a75 in ?? ()
#26 0x000000000051480b in _PyEval_EvalCodeWithName ()
#27 0x00000000005fb257 in PyEval_EvalCode ()
#28 0x00000000006205fb in ?? ()
#29 0x000000000061b724 in ?? ()
#30 0x000000000061fb2d in ?? ()
#31 0x000000000061f63a in PyRun_SimpleFileExFlags ()
#32 0x0000000000613527 in Py_RunMain ()
#33 0x00000000005ef7fd in Py_BytesMain ()
--Type <RET> for more, q to quit, c to continue without paging--
#34 0x00007ffff7c1b565 in __libc_start_main (main=0x5ef7c0, argc=6, argv=0x7fffffffde38, init=<optimized out>, fini=<optimized out>, 
    rtld_fini=<optimized out>, stack_end=0x7fffffffde28) at ../csu/libc-start.c:332
#35 0x00000000005ef6fe in _start ()
majuwa commented 3 years ago

I have the same problem. Running it without the on-demand option helps a bit. But it still can happen. I'm using the v4l2loopback webcam and after crashing, I have to completely unload the module and load it again. Otherwise the application automatically crashes with an os error

fangfufu commented 3 years ago

@majuwa, @DrDynamic ,what OS are you on?

@majuwa, have you tried Akvcam?

@DrDynamic, could you try Python 3.7 please?

Piskvor commented 3 years ago

I'm working around occasional crashes with "if crash, restart" with Python 3.8 and mediapipe 0.8.5 - fortunately, in my case, it's only the script that segfaults:

while : ; do python ./fake.py ; if [ $? -eq 0 ] ; then break; fi ; sleep 1 ; done

(will try mp 0.8.6)

fangfufu commented 3 years ago

@Piskvor , I really dislike workarounds, if somebody can figure out why it crashes on their particular setups, it would be great.

majuwa commented 3 years ago

I'm using Ubuntu 21.04. I haven't tried Akvcam so far. I will try it

DrDynamic commented 3 years ago

I'm also on Ubuntu 21.04. And using Python 3.9.5. Any tips how to run python3.7 on my System? It isn't in the repository anymore.

katakombi commented 3 years ago

I'm also on Ubuntu 21.04. And using Python 3.9.5. Any tips how to run python3.7 on my System? It isn't in the repository anymore.

I am using Python 3.8.10 - are you getting troubles with 3.9.5?

darthmooguy commented 3 years ago

@DrDynamic It works for me on Python 3.9.5 and mediapipe 0.8.5 (on Fedora 33) :thinking:

If you want to try with python3.7, you could do so with miniconda.

fatrobin72 commented 3 years ago

Running Ubuntu 20.04.2 LTS with python 3.8.10 and mediapipe 0.8.6 and I can confirm that while running it through discord the other night that it was crashing out every 15-20 minutes (sometimes longer, sometimes shorter) though sometimes was lasting a while longer.

I was just using background replacement (no foreground) and using the no-ondemand option (haven't actually been able to get discord to pick up the camera without disabling ondemand.

Rolling back to mediapipe 0.8.5 seems at least as stable if not more (not crashed yet after about 30 mins)

Looking at mediapipes github it could be related to this https://github.com/google/mediapipe/issues/2250 but no news on investigating it yet.

fangfufu commented 3 years ago

Right I can confirm that I experience mediapipe 0.8.6 segfaulting.

cova-fe commented 3 years ago

I'm currently seeing the same behavior. I tried to put under load the machine (10+) and the issue happens much more frequently, in just under a minute you can see it. When it happens, the output is the following:

./fake.py -v /dev/video0 -w /dev/video2 -b bg.jpg -i images --no-foreground

Real camera original values are set as: 640x480 with 30 FPS and video codec 1448695129 Real camera new values are set as: 1280x720 with 30 FPS and video codec 1196444237 False Running... Please CTRL-C to pause and reload the background / foreground images Please CTRL-\ to exit INFO: Created TensorFlow Lite XNNPACK delegate for CPU. ^C Resuming, reloading background / foreground images... Real camera original values are set as: 640x480 with 30 FPS and video codec 1448695129 Real camera new values are set as: 1280x720 with 30 FPS and video codec 1196444237 fish: Job 1, './fake.py -v /dev/video0 -w /de…' terminated by signal SIGSEGV (Address boundary error) ` mediapipe is 0.8.6, self-compiled so I'm able to recompile it with different settings, hopefully :) Python 3.9.6 Linux 5.13.5 Gentoo

Let me know if more details or some specific actions can help.

fangfufu commented 3 years ago

Updated mediapipe to 0.8.6.2. It seems to work fine. Please comment if it doesn't work fine.

cova-fe commented 3 years ago

I just had this issue with 0.8.6.2. This is the output from the console (I added a debug line printing mediapipe module version):

./fake.py -v /dev/video0 -w /dev/video2 -b bg.jpg -i images --no-foreground 0.8.6.2 Real camera original values are set as: 640x480 with 30 FPS and video codec 1448695129 Real camera new values are set as: 1280x720 with 30 FPS and video codec 1196444237 False Running... Please CTRL-C to pause and reload the background / foreground images Please CTRL-\ to exit INFO: Created TensorFlow Lite XNNPACK delegate for CPU. ^C Resuming, reloading background / foreground images... Real camera original values are set as: 640x480 with 30 FPS and video codec 1448695129 Real camera new values are set as: 1280x720 with 30 FPS and video codec 1196444237 fish: Job 1, './fake.py -v /dev/video0 -w /de…' terminated by signal SIGSEGV (Address boundary error)

For me replicating the issue is quite easy: start the app and have the machine installing some gentoo packages :)

fangfufu commented 3 years ago

@cova-fe , could you try 0.8.5 to see if the problem disappears?

cova-fe commented 3 years ago

@cova-fe , could you try 0.8.5 to see if the problem disappears?

Sure, just few minutes.

cova-fe commented 3 years ago

Interesting, I was able to recreate the issue also with 0.8.5, even though it seems that the needed load was higher.

./fake.py -v /dev/video0 -w /dev/video2 -b bg.jpg -i images --no-foreground 0.8.5 Real camera original values are set as: 640x480 with 30 FPS and video codec 1448695129 Real camera new values are set as: 1280x720 with 30 FPS and video codec 1196444237 False Running... Please CTRL-C to pause and reload the background / foreground images Please CTRL-\ to exit INFO: Created TensorFlow Lite XNNPACK delegate for CPU. ^C Resuming, reloading background / foreground images... Real camera original values are set as: 640x480 with 30 FPS and video codec 1448695129 Real camera new values are set as: 1280x720 with 30 FPS and video codec 1196444237 fish: Job 1, './fake.py -v /dev/video0 -w /de…' terminated by signal SIGSEGV (Address boundary error)

I guess I can investigate more deeply on where exactly the SIGSEGV happens.

fangfufu commented 3 years ago

Yes, please do investigate. Also, it seems turning off on-demand mode helps.

cova-fe commented 3 years ago

I was able to get a bt, not sure how useful could it be: (python 3.9, mediapipe 0.8.6.2) Resuming, reloading background / foreground images... Real camera original values are set as: 640x480 with 30 FPS and video codec 1448695129 Real camera new values are set as: 1280x720 with 30 FPS and video codec 1196444237 [New Thread 0x7fffb663d640 (LWP 9412)] [New Thread 0x7fffb5e3c640 (LWP 9413)] [New Thread 0x7fffb563b640 (LWP 9414)] [New Thread 0x7fffb4e3a640 (LWP 9415)] [New Thread 0x7fffaffff640 (LWP 9416)] [New Thread 0x7fffaeffd640 (LWP 9417)] [New Thread 0x7fffae7fc640 (LWP 9418)] FPS: 17.81 Thread 1 "python" received signal SIGSEGV, Segmentation fault. 0x00007ffff7cc6cd0 in _PyTrash_begin () from /usr/lib64/libpython3.9.so.1.0 (gdb) bt

0 0x00007ffff7cc6cd0 in _PyTrash_begin () from /usr/lib64/libpython3.9.so.1.0

1 0x00007ffff7cd6b49 in ?? () from /usr/lib64/libpython3.9.so.1.0

2 0x00007ffff7c7ae7d in _PyObject_MakeTpCall () from /usr/lib64/libpython3.9.so.1.0

3 0x00007ffff7c80822 in ?? () from /usr/lib64/libpython3.9.so.1.0

4 0x00007ffff7c27c1d in _PyEval_EvalFrameDefault () from /usr/lib64/libpython3.9.so.1.0

5 0x00007ffff7d6148c in ?? () from /usr/lib64/libpython3.9.so.1.0

6 0x00007ffff7c73e81 in _PyFunction_Vectorcall () from /usr/lib64/libpython3.9.so.1.0

7 0x00007ffff7c807e8 in ?? () from /usr/lib64/libpython3.9.so.1.0

8 0x00007ffff7c27c1d in _PyEval_EvalFrameDefault () from /usr/lib64/libpython3.9.so.1.0

9 0x00007ffff7d6148c in ?? () from /usr/lib64/libpython3.9.so.1.0

10 0x00007ffff7c73e81 in _PyFunction_Vectorcall () from /usr/lib64/libpython3.9.so.1.0

11 0x00007ffff7c285da in _PyEval_EvalFrameDefault () from /usr/lib64/libpython3.9.so.1.0

12 0x00007ffff7c20551 in ?? () from /usr/lib64/libpython3.9.so.1.0

13 0x00007ffff7c285da in _PyEval_EvalFrameDefault () from /usr/lib64/libpython3.9.so.1.0

14 0x00007ffff7c20551 in ?? () from /usr/lib64/libpython3.9.so.1.0

15 0x00007ffff7c285da in _PyEval_EvalFrameDefault () from /usr/lib64/libpython3.9.so.1.0

16 0x00007ffff7c20551 in ?? () from /usr/lib64/libpython3.9.so.1.0

17 0x00007ffff7c2788e in _PyEval_EvalFrameDefault () from /usr/lib64/libpython3.9.so.1.0

18 0x00007ffff7d6148c in ?? () from /usr/lib64/libpython3.9.so.1.0

19 0x00007ffff7d6199e in _PyEval_EvalCodeWithName () from /usr/lib64/libpython3.9.so.1.0

20 0x00007ffff7d619eb in PyEval_EvalCodeEx () from /usr/lib64/libpython3.9.so.1.0

21 0x00007ffff7d5c7bb in PyEval_EvalCode () from /usr/lib64/libpython3.9.so.1.0

22 0x00007ffff7da74fe in ?? () from /usr/lib64/libpython3.9.so.1.0

23 0x00007ffff7da8af1 in PyRun_SimpleFileExFlags () from /usr/lib64/libpython3.9.so.1.0

24 0x00007ffff7dc361a in Py_RunMain () from /usr/lib64/libpython3.9.so.1.0

25 0x00007ffff7dc3c45 in Py_BytesMain () from /usr/lib64/libpython3.9.so.1.0

26 0x00007ffff7a247fd in __libc_start_main () from /lib64/libc.so.6

27 0x000055555555475a in _start ()

cova-fe commented 3 years ago

The SIGSEGV happens inside compose_frame(), when classifier is called on (frame).

def compose_frame(self, frame):
     frame.flags.writeable = False
     mask = self.classifier.process(frame).segmentation_mask`
fangfufu commented 3 years ago

Wow, how did you figure that out? Does it help if you get rid off the frame.flags.writeable lines? i.e. don't toggle the flag on and off, and make it writable the whole time.

Failing that, it might just be the problem with Mediapipe.

cova-fe commented 3 years ago

I used an horrible approach: sprinkled the code with debug print with the line number. I was able to bisect the single line in this way. It helped me that I can reproduce the crash in a minute or two. Let me try to remove the line you mentioned.

fangfufu commented 3 years ago

You might also want to remove frame.flags.writeable = true. The problem is that it doesn't crash locally for me.

cova-fe commented 3 years ago

So, last findings: removing the line frame.flags.writeable = False or setting it to True makes me unable to get other crashes, it seems you got the right suggestion. Not sure about the meaning of this flag, though, I have no knowledge about mediapipe :) Let me try to remove the frame.flags.writable to see what happens.

fangfufu commented 3 years ago

Those flags were there to improve performance, as per https://google.github.io/mediapipe/solutions/selfie_segmentation.html

fangfufu commented 3 years ago

I removed those lines for now. Hopefully it works better now.

cova-fe commented 3 years ago

So, commenting out the frame.flags.writeable = true does not seem to help, as I was able to cause a crash. Now I'm running the app with the flag set to True, let's see if I can get crashes (so far none). If it is a timing issue, it could be tricky to pin down.

cova-fe commented 3 years ago

Unfortunately it seems that it happens again, albeit less frequently. Could it be something of interest for mediapipe developers?

fangfufu commented 3 years ago

@cova-fe , there was a ticket raised in Mediapipe's repository already, have a look at https://github.com/fangfufu/Linux-Fake-Background-Webcam/issues/127#issuecomment-879384431.

fangfufu commented 3 years ago

I just upgraded to Debian Bullseye. I get crashes even at 0.8.6.2. So I pinned MediaPipe version back to 0.8.5.

fangfufu commented 3 years ago

There is a GIL fix in the latest v0.8.7.1 python binaries, which may help resolve this issue. Try pip install mediapipe==0.8.7.1

https://github.com/google/mediapipe/issues/2250#issuecomment-905986921