further ideas

[liske]

Just some further ideas to test/implement (just limited by the CPU power):

• [ ] merge the frontend images into a single .png file and use the transparent pixels as the mask
• [ ] reduce the mask calculation by skipping some frames to improve overall performance, smooth the mask to prevent mask wobble (bodypix did not know where the mask was in the last frame, but fakecam does know it and might improve accuracy)
• [ ] when more options are added it might be useful to make them configurable using parameters or a config file
• [ ] Should I add a description howto use Android or ip-based webcams? There are other howtos around but most did not work out of the box with OpenCV (since it misses YUP12 support).
• [ ] render the person(s) at the center of the screen (or make the position configurable):
  • get the dimension of the non-masked pixels from the original video stream using bodypix
  • keep it aligned at top or center to the configured position so that the head is not moving away from the speaker's desk
• [ ] try to better fit the stream mask or reduce the hallow effect (how?)
• [ ] add another (optional) root layer which can be sourced by another video source
  • video files or live stream could be easily provided as /dev/video device using ffmpeg so that we can use it from OpenCV
  • a background mask is required (or just support for .png files with alpha channels)
  • WHY? /me on the ISS in front for a window showing a live video looking at the earth from space with some spacecraft utils in the front - we just need to add a fake physics device powered by USB-C to get the zero gravity feeling ;-)

What do you think?

[fangfufu]

1. That sounds like a good idea, but the current approach works fine as a hack together project.
2. I actually thought about doing that, but it turned out that scaling the frame provided enough performance boost for my own machine. Although I am not sure how knowing the previous mask would help in terms reducing wobbling. Each frame is a new classification task. I suppose you can use it to filter out outliers, but I think bodypix performs well enough. If you know more than one previous mask, you could predict trajectory and things, but that's quite complicated, and I feel it is beyond me.
3. I agree.
4. Yes, that would be helpful.
5. That could be done with OpenCV's face tracker. I suppose. We might have to do some computation on the scaling of the face as well.
6. The halo effect might be due to blurring at the edge of the mask, it is done at fakecam/fake.py#L64. I think to fit a better mask, you need piping in more pixels to bodypix. This in term slows things down. This is why I downscale the image sent to the network, and upscale it when it comes back. One could argue that GPU would be a better performance, but I think a lot of people would agree that Nvidia GPU drivers are not the easiest thing to set up. Setting up your Nvidia GPU for playing game is actually separate to setting it up for CUDA. CUDA driver is definitely non-free.
7. At this point, have you thought about simply using obs-v4l2sink

Overall this is a big list of things. This "project" started off as tidying up someone else's blog post. I actually came across the blog post when I was reading a feature request for Webcamoid (https://github.com/webcamoid/WebcamoidIssues/issues/26). I kind of prefer this project ends up being a plugin for OBS or Webcamoid.

In terms of the things that really should be worked on - I really don't like the concept of passing images from Python to Node.js via HTTP. This creates unnecessary overhead. I did wonder why this was even a design decision, it turns out that that bodypix is only officially available a Tensorflow.js model (https://github.com/tensorflow/tfjs/issues/2043). Although someone else is working on this already (allo-/virtual_webcam_background).

I may work through the list when I have more time, I make no guarantees. I need to go back and finish my PhD thesis. I only did this whole thing because I wanted to impress my labmates in research group Zoom meetings. :)

[fangfufu]

I just want to say that be aware of https://github.com/allo-/virtual_webcam_background/issues/10 when installing Virtual Webcam Background. At the moment, its performance is similar to mine.

[fangfufu]

@liske , feel free to work on any of the suggestions you listed. I think it would be a good idea to add a description on how to use Android or ip-based webcams.

[fangfufu]

@liske , someone sent in a patch to support animated video background. It works really well.

[fangfufu]

@liske I updated the repository to use Mediapipe as the segmentation backend. It is now running so much faster.

[zeorin]

As a further idea, now that MediaPipe is being used, could we extend this to do stuff like show specific overlays when certain facial expressions or hand gestures are recognized? E.g. like SnapLensStudio does: https://twitter.com/cameronhunter/status/1305667539922550784

[fangfufu]

@zeorin , feel free to work on it yourself, and submit patches.